Understanding Bridge and VLAN Management in Eucalyptus Managed Networking Mode

Follow

Eucalyptus Versions: All*

Understanding Bridge and VLAN Management in Eucalyptus Managed Networking Mode

Out of all the networking modes (until the introduction of Edge Networking mode in December 2013 [1]), Managed [2] was considered to be the most difficult to understand and/or debug.  This article will explain how a cloud administrator can use the information provided from the Cloud Controller (using the Eucalyptus Admin Tools [3] and euca2ools),  to trace information to the Cluster Controller (CC), and then finally to the Node Controller (NC) to confirm/troubleshoot network connectivity to instances, on a Eucalyptus cloud configured to use Managed networking mode.

Cloud Controller (CLC)

Once an instance is launched and reaches the 'running' state, there will be a public and private IPv4 (and public/private DNS name if Eucalyptus DNS [4] is enabled) for the instance.

[root@odc-f-13 ~]# euca-describe-instances i-6292df29
RESERVATION r-67d3158e 214091788602 default
INSTANCE i-6292df29 emi-469772d6 10.104.6.231 172.18.118.13 running euca-admin 0 m1.large 2014-05-11T00:25:57.635Z RiseOfAnEmpire monitoring-disabled 10.104.6.231 172.18.118.13 instance-store hvm sg-e69f5d88
TAG instance i-6292df29 euca:node 10.105.1.188

When executing 'euca-describe-instances' as the admin user of the 'eucalyptus' account, each instance is displayed with a custom tag (which only applies to versions of Eucalyptus 3.4 and greater) that shows which Node Controller (NC) contains the running instance.  

[root@odc-f-13 ~]# euca-describe-tags
TAG instance i-6292df29 euca:node 10.105.1.188

This information can be confirmed and cross-referenced by the information returned from 'euca_conf --list-nodes' [5]:

[root@odc-f-13 ~]# euca_conf --list-nodes
NODE RiseOfAnEmpire 10.105.1.188 ENABLED
INSTANCE i-6292df29
INSTANCE i-ea9d5544

Once the following has been confirmed:

  • euca-describe-instances displays the instance
    • in a 'running' state
    • with public IPv4 address
    • with private IPv4 address
  • euca_conf --list-nodes displays the instance has been associated with a Node Controller (NC)

the cloud administrator can move to the Cluster Controller (CC) to confirm if the networking has been set up correctly.

Cluster Controller (CC)

The Cluster Controller (CC) contains the most important information about the instance with regards to networking information.  The cloud administrator can discover the following on the Cluster Controller (CC) about an instance's networking information:

  • the public IPv4 associated with the instance
  • the private IPv4 associated with the instance
  • the VLAN assigned to the instance
  • the private MAC address assigned to the instance
  • the Eucalyptus bridge used by the instance
  • the router assigned for the instance to use
  • the number of private IPs allowed per security group
  • the iptables rules that control routing from public to private IPv4 (and vise versa) 
  • the Node Controller (NC) which communicates back to the Cluster Controller (CC) that states the instance where the instance is located
  • the current state of the instance

To gather all this information, the cloud administrator must use different resources located on the Cluster Controller (CC).  These resources are the following:

cc.log

The cc.log contains great information about the state of the instance.  The Cluster Controller (CC) log level needs to be set to DEBUG [6]. Use the 'grep' command to locate the 'refresh_instances' call, and check for the specific instance id.  For example:

[root@odc-d-31 ~]# grep i-6292df29 /var/log/eucalyptus/cc.log | grep refresh_instances 
2014-05-10 20:56:05 DEBUG 000019948 refresh_instances | describing instance i-6292df29, Extant, 1
2014-05-10 20:56:05 DEBUG 000019948 refresh_instances | storing instance state: i-6292df29/Extant/10.104.6.231/172.18.118.13
2014-05-10 20:56:05 DEBUG 000019948 print_ccInstance | refresh_instances(): instanceId=i-6292df29 reservationId=r-67d3158e state=Extant accountId=214091788602 ownerId=AIDM7CMUGPQCXBESVSMRG ts=1399767962 keyName=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZbOsrrcyTVIJfOTeQtwFvK+QxdG73ZKYz2XdiJfgE6LN7r2X5AZGHRmGGiCYCINQFjTKW6v6CTu114l7FwF6rxH0qjtV9odJouhADwvElMu37X4N8rqIIjBvnTAfD8A/NIZLIVsknnjUw8hbKAdo2txgMt4EPeCydfz+X/+8C/sjBfZSDoi4jCDl0z5zBleA9+Z1ghFJXikts3mc8DXjU3QW9oPN4Wc8apAKfsGpItqfMhPKGMEBEM0r5QOk2caypF9dUWr7g8yThoCEaqa1DMc5ukdSAq18RYdP2nbMsWHzwjjlgtS4k1uck/idX7xQ6Zt7ZWJZtFtV1wbYN4RUV 214091788602@eucalyptus.euca-admin ccnet={privateIp=172.18.118.13 publicIp=10.104.6.231 privateMac=D0:0D:62:92:DF:29 vlan=946 networkIndex=13} ccvm={cores=1 mem=1024 disk=10} ncHostIdx=0 serviceTag=http://10.105.1.188:8775/axis2/services/EucalyptusNC userData= launchIndex=0 platform=linux bundleTaskStateName=none, volumesSize=0 volumes={} groupNames={214091788602-12ce4ee5-c874-48c4-9a10-26570ccc8ba4 } migration_state=none guestStateName=poweredOn

From this information in the cc.log, the following can be noted:

  • instance's state is 'Extant' (which means 'running')
  • instance's private IPv4 address is 172.18.118.13
  • instance's public IPv4 address is 10.104.6.231
  • instance's private MAC address is D0:0D:62:92:DF:29
  • VLAN associated with the instance - 946 (i.e. vlan=946)
  • the Node Controller where the instance is located - http://10.105.1.188:8775/axis2/services/EucalyptusNC

euca-dhcp.conf

To confirm that this information is being served correctly by the dhcpd server running on the Cluster Controller (CC), which Eucalyptus controls, check the /var/run/eucalyptus/net/euca-dhcp.conf file:

 

[root@odc-d-31 ~]# cat /var/run/eucalyptus/net/euca-dhcp.conf
.....
subnet 172.18.118.0 netmask 255.255.255.224 {
option subnet-mask 255.255.255.224;
option broadcast-address 172.18.118.31;
option domain-name "eucalyptus.internal";
option domain-name-servers 8.8.8.8;
option routers 172.18.118.1;
}

 

host node-172.18.118.13 {
hardware ethernet D0:0D:62:92:DF:29;
fixed-address 172.18.118.13;
}

From this file, the following can be noted:

  • the DHCP server is associating the MAC address D0:0D:62:92:DF:29 with the private IP 172.18.118.13.
  • the router (i.e. gateway) for IPs on 172.18.118.0 will be 172.18.118.1, and the broadcast address is 172.18.118.31
  • the last four 2-digit hexadecimal groups of MAC address given node-172.18.118.13 (62:92:DF:29) match the instance ID i-6292df29

bridge command

The bridge command can be used to confirm the VLAN the Cluster Controller has assigned to the instance.  This information is dependent upon the following:

  • The maximum (cloud.network.global_max_network_tag) and minimum (cloud.network.global_min_network_tag) VLAN tags defined in the cloud properties [7]
  • The VNET_PRIVINTERFACE and VNET_PUBINTERFACE values in the /etc/eucalyptus/eucalyptus.conf file on the Cluster Controller (CC) [8]

To confirm the VLAN has been provided on the Cluster Controller (CC) and is being actively used, run the following command:

[root@odc-d-31 ~]# bridge vlan show
port vlan ids
em2.946 None

From this output, the following has been confirmed:

  • the VLAN ID 946 has been mapped to the interface device 'em2'
  • the interface device 'em2' has been defined as the VNET_PRIVINTERFACE in the /etc/eucalyptus/eucalyptus.conf on the Cluster Controller (CC)

brctl command

Since the VLAN ID has been confirmed to correctly assigned to the interface device 'em2', the brctl command will confirm that the Eucalyptus bridge has been correctly mapped to the interface.  All custom bridges created by Eucalyptus start with 'eucabr'.  To see the mapping of the VLAN ID to the bridge interface, run the following command:

[root@odc-d-31 ~]# brctl show | grep 946
bridge name bridge id STP enabledinterfaces
eucabr946 8000.b8ac6f8b7f75 no em2.946

The output of this command confirms that the interface device 'em2', with the VLAN ID '946', has been properly mapped to the Eucalyptus bridge 'eucabr946'.

ip command

The ip command is used to bring together the information discovered from the bridge and brtcl commands.  To begin, display the information for the 'em2.946' device:

[root@odc-d-31 ~]# ip addr show dev em2.946
16: em2.946@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether b8:ac:6f:8b:7f:75 brd ff:ff:ff:ff:ff:ff
inet6 fe80::baac:6fff:fe8b:7f75/64 scope link
valid_lft forever preferred_lft forever

The output confirms that the 'bridge id' displayed in the 'brctl show | grep 946' command has been correctly assigned as the hardware address on this interface. Next, display the information for the Eucalyptus bridge 'eucabr946':

[root@odc-d-31 ~]# ip addr show dev eucabr946
17: eucabr946: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether b8:ac:6f:8b:7f:75 brd ff:ff:ff:ff:ff:ff
inet 172.18.118.1/27 brd 172.18.118.31 scope global eucabr946

The output confirms the following:

  • the interface devices 'em2.946' and 'eucabr946' both have the same hardware address
  • the interface device 'eucabr946' will handle traffic for private IPs in the range of 172.18.118.1 to 172.18.118.30.
  • VNET_ADDRSPERNET in the /etc/eucalyptus/eucalyptus.conf file is 32 - which means there are 29 private IPs per security group that can be used by the user (2 of the IPs are reserved by Eucalyptus for the gateway, and broadcast address)

iptables-save command

Using the iptables-save command, the cloud administrator can observe the routing rules for public-to-private, private-to-public traffic for a given instance. Using the example above, use iptables-save with the grep command, to confirm the routing information for the instance:

[root@odc-d-31 ~]# iptables-save | grep 172.18.118.13
-A EUCA_COUNTERS_IN -d 172.18.118.13/32
-A EUCA_COUNTERS_OUT -s 172.18.118.13/32
-A PREROUTING -d 10.104.6.231/32 -j DNAT --to-destination 172.18.118.13
-A POSTROUTING -s 172.18.118.13/32 -j SNAT --to-source 10.104.6.231
-A OUTPUT -d 10.104.6.231/32 -j DNAT --to-destination 172.18.118.13

This output shows:

  • routing from the public IP 10.104.6.231 to the private IP 172.18.118.13
  • routing from the private IP 172.18.118.13 to the public IP 10.104.6.231
  • the private IP 172.18.118.13 being appended to the EUCA_COUNTERS_IN and EUCA_COUNTERS_OUT chains (this is to gather network information for Eucalyptus CloudWatch support for EC2 instances [9])

Finally, the cloud administrator can go to the Node Controller (NC) to complete the network layout for the instance.

Node Controller (NC)

As demonstrated on the Cluster Controller (CC), the cloud administrator can use the bridge, brctl, and ip commands to gather information about the network layout for a given instance.  

bridge command

Using the bridge command, the cloud administrator can confirm if the corresponding VLAN on the Cluster Controller (CC) is present on the Node Controller (NC).  Run the following command to view the VLANs on the Node Controller (NC):

[root@odc-d-37 ~]# bridge vlan show
port vlan ids
....
em2.946 None
vnet1 None

From this output, the following has been confirmed:

  • the VLAN ID 946 has been assigned to the interface device 'em2'
  • the interface device 'em2' is the value of VNET_PUBINTERFACE in the /etc/eucalyptus/eucalyptus.conf on the Node Controller (NC)

brctl show

The output from the 'brctl show' will show the cloud administrator the bridges present on the Node Controller (NC).

[root@odc-d-37 ~]# brctl show
bridge name bridge id STP enabled interfaces eucabr946 8000.b8ac6f950474 no         em2.946                                                  vnet1

This output shows:

  • The Eucalyptus bridge 'eucabr946' is associated with the interface device 'em2.946'.
  • The interface 'vnet1' is associated with the bridge 'eucabr946'

ip command

As demonstrated on the Cluster Controller (CC), the ip command connects all the information from the bridge and brctl commands.  Using the ip command, gather information for the devices 'em2.946', 'eucabr946', and 'vnet1':

[root@odc-d-37 ~]# ip addr show dev em2.946
11: em2.946@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether b8:ac:6f:95:04:74 brd ff:ff:ff:ff:ff:ff
inet6 fe80::baac:6fff:fe95:474/64 scope link
valid_lft forever preferred_lft forever
[root@odc-d-37 ~]# ip addr show dev eucabr946
12: eucabr946: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether b8:ac:6f:95:04:74 brd ff:ff:ff:ff:ff:ff
inet6 fe80::baac:6fff:fe95:474/64 scope link
valid_lft forever preferred_lft forever
[root@odc-d-37 ~]# ip addr show dev vnet1
13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
link/ether fe:0d:62:92:df:29 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc0d:62ff:fe92:df29/64 scope link
valid_lft forever preferred_lft forever

From this output, the following can be observed:

  • the 'em2.946' interface is associated with the Eucalyptus bridge 'eucabr946' based upon hardware address information
  • the interface 'vnet1' is associated with the instance ID i-6292df29.  The last four two-digit hexadecimal groups match the instance ID

The key thing to note here is that there will not be any IP information associated with these devices.  All IP information is handled at the Cluster Controller (CC). 

From stepping through all the information above, the cloud administrator can confirm the bridge and VLAN information for a given instance.

Troubleshooting Hints

From a cloud user point of view, the cloud user can do the following to troubleshoot network traffic to the instance:

  • confirm the security group associated with the running instance has the appropriate authorization rules (applied using euca-authorize)
  • use euca-get-console-output to see the boot up process of the instance

After this has been confirmed from a cloud user perspective, the cloud administrator (i.e. the individual(s) that have access to the physical machines where the Eucalyptus components are running) should use the information above to verify that the network layout is correct for the instance.  Once that has been verified (and network traffic to the instance is still not available), make sure that the VLAN tag information in the cloud properties are within the supported range allowed on the infrastructure switch hardware.  

References

[1]  Eucalyptus Videos - Benefits of Edge Networking in Eucalyptus 4.0
[2]  Eucalyptus Installation Guide - Plan Networking Modes - Managed Mode
[3]  Eucalyptus Administration Guide - Eucalyptus Administration Commands
[4]  Eucalyptus Installation Guide - Configure DNS
[5]  Eucalyptus Administration Guide - Eucalyptus Administration Commands - euca_conf
[6]  Eucalyptus Troubleshooting Guide - Log Files
[7]  Eucalyptus Installation Guide - Configure the Runtime Environment - Set Up Security Groups
[8]  Eucalyptus Installation Guide - Configure Eucalyptus - Configure Network Modes - Configure for Managed Mode
[9]  Eucalyptus User Guide - Using CloudWatch

Have more questions? Submit a request

Comments

Powered by Zendesk