Create and Attach a Second Elastic Network Interface with EIP to EC2 VPC Instance (with Routing)

We’ll be using AWS CLI on a Debian Linux VM. 

Revision history (dd/mm/yy):

24/04/2014 – added “Configure Routing on Ubuntu/Debian VM” section.

28/08/2014 – added “Make Static Routes Persistent after Reboot” section.

Installation

On a Debian PC we’re going to work from, install Python pip:

# apt-get update && apt-get install python2.7 python-pip

Install awscli package:

# pip install awscli

Configure awscli if using for the first time:

$ aws configure
AWS Access Key ID [****************YAPH]: 
AWS Secret Access Key [****************+qKv]: 
Default region name [eu-west-1]: 
Default output format [text]:

The following set of IAM permissions should be sufficient:

{ "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": [
        "ec2:AllocateAddress",
        "ec2:AssociateAddress",
        "ec2:AttachNetworkInterface",
        "ec2:CreateNetworkInterface",
        "ec2:DescribeInstances",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DetachNetworkInterface"
      ],
      "Resource": ["*"]
    }]
}

Configuration

See our EC2 instance’s details below:

  1. Subnet: 10.20.0.0/24
  2. Security group name: FW-01

Note that each instance has it’s dedicated security group.

Get VPC Subnet ID

We need to know the VPC subnet’s ID:

$ aws ec2 describe-subnets | grep 10.20
SUBNETS	eu-west-1b 231 10.20.0.0/24 False False available subnet-08202861 vpc-3220285b

Get Security Group ID

We also need to know the security group’s ID:

$ aws ec2 describe-security-groups | grep -i fw-01
SECURITYGROUPS FW-01 sg-1908e47c FW-01 1*********483 vpc-3220285b

Get List of Private IPs in Use

As we want to assign a static private IP, we need to know the ones that aren’t in use. Let us get a list of private IPs which are in use:

$ aws ec2 describe-network-interfaces | egrep "PRIVATEIP.*10.20" | cut -f3-4 | sort
10.20.0.10
10.20.0.200
10.20.0.201
10.20.0.202
10.20.0.203
10.20.0.205
10.20.0.206
10.20.0.210
10.20.0.241
10.20.0.243
10.20.0.244
10.20.0.40
10.20.0.60

We should be able to use 10.20.0.11 as it is not taken yet.

Create New Network Interface

Create a new network interface with a private IP of 10.20.0.11 for the subnet subnet-08202861 and the security group sg-1908e47c:

$ aws ec2 create-network-interface --subnet-id subnet-08202861 --description "2ndEIP" --groups sg-1908e47c --private-ip-address 10.20.0.11        
NETWORKINTERFACE eu-west-1b 2ndEIP [...] eni-46c51d23 [...] 10.20.0.11 [...] subnet-08202861 vpc-3220285b
GROUPS sg-1908e47c FW-01
PRIVATEIPADDRESSES True 10.20.0.11

Allocate EIP for VPC

EIP will be associated with the newly created network interface.

$ aws ec2 allocate-address --domain vpc
eipalloc-54534336 vpc 54.72.XX.6

Get Instance ID

In our particular case, each instance has a separate security group, or in other words, its own dedicated firewall rules. Therefore one security groups is tidied with a single instance only. If we know the security group, we can find the instance ID:

$ aws ec2 describe-instances --filters "Name=instance.group-id,Values=sg-1908e47c" | \
grep -w "i-........" | cut -f8
i-846943c5

Attached Network Interface to Instance

$ aws ec2 attach-network-interface --network-interface-id eni-46c51d23 --instance-id i-846943c5 --device-index 1
eni-attach-2473bd6a

Associate EIP with Network Interface

$ aws ec2 associate-address --allocation-id eipalloc-54534336 --network-interface-id eni-46c51d23
eipassoc-fd45ba98	true

Check All EIPs Assigned to Instance

$ aws ec2 describe-instances --filters "Name=instance.group-id,Values=sg-1908e47c" | \
grep -i assoc | cut -f4 | uniq
54.72.XX.201
54.72.XX.6

Detach Network Interface

We can detached a network interface if we don’t need it:

$ aws ec2 detach-network-interface --attachment-id eni-attach-2473bd6a
true

Configure Routing on Ubuntu/Debian VM (edited: 24/04/2014)

Tested on Ubuntu 12.04/14.04 and Debian Wheezy VMs.

SSH into the VM once it’s up and running. You’ll notice that the eth1 interface has no private IP assigned yet:

# ifconfig -a | egrep 'eth|inet.*10.20'
eth0      Link encap:Ethernet  HWaddr 02:eb:10:eb:2e:69  
          inet addr:10.20.0.10  Bcast:10.20.0.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 02:48:1f:84:f3:09

The routing table should look something like this:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.20.0.1       0.0.0.0         UG    0      0        0 eth0
10.20.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0

For Ubuntu, create a config file for the eth1 interface:

# cp /etc/network/interfaces.d/eth0.cfg /etc/network/interfaces.d/eth1.cfg

Open the eth1 config file /etc/network/interfaces.d/eth1.cfg and change all occurrences of eth0 to eth1 so the file would look like:

# cat /etc/network/interfaces.d/eth1.cfg
# The secondary network interface
auto eth1
iface eth1 inet dhcp

For Debian, add configuration to /etc/network/interfaces:

# echo "auto eth1" >>/etc/network/interfaces
# echo "iface eth1 inet dhcp" >>/etc/network/interfaces

Bring the eth1 interface up:

# ifup eth1
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth1/02:48:1f:84:f3:09
Sending on   LPF/eth1/02:48:1f:84:f3:09
Sending on   Socket/fallback
DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 3 (xid=0x58501c6e)
DHCPREQUEST of 10.20.0.11 on eth1 to 255.255.255.255 port 67 (xid=0x58501c6e)
DHCPOFFER of 10.20.0.11 from 10.20.0.1
DHCPACK of 10.20.0.11 from 10.20.0.1
bound to 10.20.0.11 -- renewal in 1532 seconds.

Double-check that the private IP has been assigned:

# ifconfig -a | egrep 'eth|inet.*10.20'
eth0      Link encap:Ethernet  HWaddr 02:eb:10:eb:2e:69  
          inet addr:10.20.0.10  Bcast:10.20.0.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 02:48:1f:84:f3:09  
          inet addr:10.20.0.11  Bcast:10.20.0.255  Mask:255.255.255.0

Create a custom route table named “out”:

# echo "200 out" >>/etc/iproute2/rt_tables

Check:

# cat /etc/iproute2/rt_tables
#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep
200 out

List the kernel’s routing table to find out the default gateway:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.20.0.1       0.0.0.0         UG    0      0        0 eth0
10.20.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.20.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth1

Add a rule to route eth1 traffic via default gateway:

# ip route add default via 10.20.0.1 dev eth1 table out

Add routing rules to route all traffic from/to IP 10.20.0.11, which is assigned to the eth1 interface, via the “out” routing table:

# ip rule add from 10.20.0.11/32 table out
# ip rule add to 10.20.0.11/32 table out
# ip route flush cache

Check the “out” routing table to be sure:

# ip route show table out
default via 10.20.0.1 dev eth1

Here’s how all route entries in the kernel look like:

# ip route list
default via 10.20.0.1 dev eth0 
10.20.0.0/24 dev eth0  proto kernel  scope link  src 10.20.0.10
10.20.0.0/24 dev eth1  proto kernel  scope link  src 10.20.0.11

The routing policy rules can be seen below:

# ip rule list
0:	from all lookup local 
32764:	from all to 10.20.0.11 lookup out 
32765:	from 10.20.0.11 lookup out 
32766:	from all lookup main 
32767:	from all lookup default

At this point, you should be able to ping EIPs assigned to both eth0 and eth1.

$ ping -c2 54.72.XX.201
PING 54.72.XX.201 (54.72.XX.201) 56(84) bytes of data.
64 bytes from 54.72.XX.201: icmp_req=1 ttl=50 time=17.9 ms
64 bytes from 54.72.XX.201: icmp_req=2 ttl=50 time=17.8 ms

--- 54.72.XX.201 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 17.835/17.912/17.989/0.077 ms
$ ping -c2 54.72.XX.6
PING 54.72.XX.6 (54.72.XX.6) 56(84) bytes of data.
64 bytes from 54.72.XX.6: icmp_req=1 ttl=51 time=18.2 ms
64 bytes from 54.72.XX.6: icmp_req=2 ttl=51 time=22.8 ms

--- 54.72.XX.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 18.279/20.543/22.807/2.264 ms

Make Static Routes Persistent after Reboot (edited: 28/08/2014)

Tested on Ubuntu only.

# cat /etc/network/interfaces.d/eth0.cfg
# The primary network interface
auto eth0
iface eth0 inet dhcp
# cat /etc/network/interfaces.d/eth1.cfg
# The secondary network interface
auto eth1
iface eth1 inet static
 address 10.20.0.11
 netmask 255.255.255.0
 up ip route add default via 10.20.0.1 dev eth1 table out
 up ip rule add from 10.20.0.11/32 table out
 up ip rule add to 10.20.0.11/32 table out
 up ip route flush cache

Disclaimer

Jeff Barr wrote: “I should note that attaching two public ENIs to the same instance is not the right way to create an EC2 instance with two public IP addresses. There’s no way to ensure that packets arriving via a particular ENI will leave through it without setting up some specialized routing.”

Despite the fact that it’s not reflected here, the original idea behind this article was to bridge 2 different VPCs.

Since AWS started to support multiple private IPs assigned to a single network interface, EIPs can be associated to private IPs by using a single network card:

50 thoughts on “Create and Attach a Second Elastic Network Interface with EIP to EC2 VPC Instance (with Routing)

  1. What must my /etc/network/interfaces file on debian look like?

    Must I run any commands after for asymmetrical routing? Thank you for your help

    I have gotten this far, but I cannot ping the second EIP.

  2. After step:
    ip rule add from 10.20.0.11/32 table out
    root@ip-172-xx-13-xxx:/home/ubuntu# ip rule add from 172.xx.10.xxx/20 table out
    Write failed: Broken pipe

    EC2 instance crashed and connect impossible (ssh). Only instance termination will help.
    Is any solution for it?

    • A bit late to the party, but for anyone arriving here looking for this – try removing one of the network interfaces from the instance and force restart the instance. I rebuilt from the ground up twice before I found this article, and then once again when I missed a step above. I eventually figured (as my mental stability began to fade) that removing a NIC would force the OS to re-evaluate it’s config at startup.

  3. And for debian wheezy the same result:

    root@ip-172-xx-7-xxx:/home/admin# route -n

    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    0.0.0.0 172.31.0.1 0.0.0.0 UG 0 0 0 eth0
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
    172.31.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
    172.31.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth1

    root@ip-172-xx-7-xxx:/home/admin# ip route add default via 172.31.0.1 dev eth1 table out
    root@ip-172-xx-7-xxx:/home/admin# ip rule add from 172.31.0.0/20 table out
    Write failed: Broken pipe

  4. But after rebooting my eth0 interface is not accessible more. SSH connect i’ll take only through eth1 public ip, and i can’t ping eth0 public ip. Any ideas?

    • should i place commands “ip route && ip rule add” with static routes definition to?

      example for eth1:

      auto eth1
      iface eth1 inet static
      address 172.31.10.172
      netmask 255.255.240.0
      network 172.31.0.0
      broadcast 172.31.15.255
      up ip route add default via 172.31.0.1 dev eth1 table out #?
      up ip rule add from 172.31.10.172 table out #?
      up ip rule add to 172.31.10.172 table out #?

      up ip addr add 172.31.1.67/20 dev eth0

    • I edited the article and added my configuration from Ubuntu 14.04. You should be good to go with this:

      auto eth1
      iface eth1 inet static
       address 172.31.10.172
       netmask 255.255.240.0
       network 172.31.0.0
       broadcast 172.31.15.255
       up ip route add default via 172.31.0.1 dev eth1 table out
       up ip rule add from 172.31.10.172/32 table out
       up ip rule add to 172.31.10.172/32 table out
       up ip route flush cache
    • Do you left for eth0 interface “iface eth0 inet dhcp” or replaced it to static?

    • Left with DHCP. It’s a static DHCP lease anyway (IP doesn’t change) as I specified the primary IP for eth0 manually when created my EC2 instance.

    • Great, it’s awesome! Now stack is working. Thank you for great tutorial and quick answers ;)

  5. Why when i add this line “up ip addr add 172.31.1.67/20 dev eth1” to eth1 it not works, but from eth0 “up ip addr add 172.31.10.125/20 dev eth0” same works.

    auto eth1
    iface eth1 inet static
    address 172.31.10.172
    netmask 255.255.240.0
    network 172.31.0.0
    broadcast 172.31.15.255
    up ip route add default via 172.31.0.1 dev eth1 table out
    up ip rule add from 172.31.10.172/32 table out
    up ip rule add to 172.31.10.172/32 table out
    up ip route flush cache
    up ip addr add 172.31.1.67/20 dev eth1

    eth1 primary public ip is available, but additional public ip from 172.31.1.67/20 is not available.

    Here eth0 that’s worked fine:

    auto eth0
    iface eth0 inet dhcp
    up ip addr add 172.31.10.125/20 dev eth0

    • I don’t know what you’re trying to achieve nor I know what your environment is, but I have a feeling that it may be a bit beyond the scope of this article.

      When you say it doesn’t work do you mean that you cannot add a secondary private IP via “ip addr” command? From the post above, have you assigned the secondary private IP 172.31.1.67 to your network card on AWS? If so, is it tidied with an EIP?

  6. Yes, i try to assign secondary private ip 172.31.1.67 to eth1 interface using command “up ip addr add 172.31.1.67/20 dev eth1”. All private ips is tided with EIP.

    And ping to private ip 172.31.1.67 works, but ping to his public ip failed. In eth1 i use static routing like you described earlier.

    • Can you post the output of the following?

      # ip addr
      # curl --interface 172.31.10.172 ifconfig.me
      # curl --interface 172.31.1.67 ifconfig.me
  7. ubuntu@ip-172-31-9-253:~$ ip addr
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0a:cd:5d:ec:a3:13 brd ff:ff:ff:ff:ff:ff
    inet 172.31.9.253/20 brd 172.31.15.255 scope global eth0
    valid_lft forever preferred_lft forever
    inet 172.31.10.125/20 scope global secondary eth0
    valid_lft forever preferred_lft forever
    inet6 fe80::8cd:5dff:feec:a313/64 scope link
    valid_lft forever preferred_lft forever
    3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0a:74:e6:54:ba:c9 brd ff:ff:ff:ff:ff:ff
    inet 172.31.10.172/20 brd 172.31.15.255 scope global eth1
    valid_lft forever preferred_lft forever
    inet 172.31.1.67/20 scope global secondary eth1
    valid_lft forever preferred_lft forever
    inet6 fe80::874:e6ff:fe54:bac9/64 scope link
    valid_lft forever preferred_lft forever

    ubuntu@ip-172-31-9-253:~$ curl –interface 172.31.10.172 ifconfig.me
    54.77.XXX.178
    ubuntu@ip-172-31-9-253:~$ curl –interface 172.31.1.67 ifconfig.me
    Timeout error

    • OK, enter the following:

      # ip rule add from 172.31.1.67/32 table out
      # ip rule add to 172.31.1.67/32 table out

      And then try the curl command which timed out again. If it returns the public EIP assigned to 172.31.1.67, then you should be able to ping it.

  8. This may be old, but I’ll give it a shoot. Your example works great for all incoming traffic (to the private and public addresses, on both interfaces), but I can’t get outgoing traffic form eth1 to work. So ping -I eth1 google.com return network unreachable. Is there any other rule I should add?

    Thanks!

  9. Hi Tomas, great post. It was really useful to me.

    I was using one EIP and switching it back and forth between two interfaces. With EIP assigned to eth1, the instance lost its public DNS and it did not have inernet access, despite it was possible to ping EIP across the internet.

    Any thought on that.

    Many thanks,
    Ibrahim

    • Sorry, I don’t quite understand your problem. So an EIP was still attached to your server, but there was no public DNS?

  10. One EIP used here and I moved it back and forth between eth0 & eth1. When I attached it to eth1 there was no public DNS, So the answer is yes to your question.

    • Do you have DNS hostnames enabled for your VPC? Each instance should be given a public DNS hostname when the instance is associated with an EIP address.

  11. Yes, I do. It’s the case when associating the EIP to eth0, but not in case of eth1, the newly added interface.

    • I have on-premise VMs, which should request IPs from the AWS VPC dhcp server. The VMs could not get IPs from AWS.
      Also, in the two interfaces scenario, I configured Bridge (br0) through eth1. That may help you understand my issue here with public DNS.
      Can you help me with that?

      Ibrahim

    • I don’t think I can as I don’t use AWS On-Premises instances, and I don’t have bridged interfaces. You’re trying to do something that’s outside the scope of this article. Feel free to let us know how you get on, as it may be useful to others.

  12. I cannot do this step ‘Create a custom route table named “out”’. I have ubuntu EC2 instance.

    sudo echo “200 out” >> /etc/iproute2/rt_tables
    -bash: /etc/iproute2/rt_tables: Permission denied

    Is it blocked by AWS?

    • You can’t use sudo to affect output redirection, because redirection is done by the calling shell.

      Try as the root user, it should work.

    • yes. as a root it worked! 10x.
      For others, before executing this step switch to root “sudo su – root”.

      All the guide step by step leads to success, as of today :)

  13. Following this on AWS with Ubuntu 16.04, I’ve the following problem

    After add the first rule, my main IP (on ens3) becomes unresponsive.

    ip route add default via 172.31.48.1 dev ens4 table out
    ip rule add from 172.31.58.30/20 table out
    ip rule add to 172.31.58.30/20 table out
    ip route flush cache

    I need to restart the server (or the networking) to the main IP work, but then the secondary ip (on ens4) stops.

    Trying to figure out what’s the cause, with no success

    • Tomas wrote an addendum for restarting the server. DHCP may assign eth1 as the default route (route -n), so instead, in your .cfg, you should use ‘static’ and not ‘dhcp’. See the section where he goes over “Make Static Routes Persistent after Reboot”

  14. Arigato gozaimasu/danke/dhanyavaad

    Your explanation really helped me to add a new NIC to existing EC2 instance and make it functional.

  15. Hi

    I found your article really useful in creating a second EIP for a Ubuntu instance via the terminal. However the command

    ip route add default via 10.20.0.1 dev eth1 table out

    fails to execute when included in the cfn:init portion of a cloudformation template.

    Thanks
    Craig

Leave a Reply

Your email address will not be published. Required fields are marked *