Linux Etherchannel

Linux EtherChannel - Double Your Pleasure Double Your Fun

As always I'm long overdue for an update. I've been busy with other more important things, but I figured since I had to document this for others within my group, here would be the best place since it might benefit others as well.

 

For today's exercise we'll be creating an EtherChannel between a Red Hat Enterprise 5 server, and a Cisco Catalyst 3750 Switch. This is actually far simpler then it sounds, and can be completed in about ten minutes.

 

We'll begin with configuring the IEEE 802.3ad Dynamic link aggregation (AKA EtherChannel) on the Red Hat Enterprise Linux server. Begin by logging in via SSH, Telnet or directly on the console itself. I do recommend having access to the console directly, so should anything go wrong and you lose network connectivity you'll be able to easily change things back.

 

Once logged into the server, switch user to "root" if you're not already logged in as root. Change directory to "/etc" and modify the "modprobe.conf" file using your favorite text editor such as "vi". I personally like using "nano". Add the lines in bold from the example "modprobe.conf" below to your file. Then save your changes and return to the bash prompt.

 

Sample /etc/modprobe.conf

alias scsi_hostadapter megaraid_sas
alias scsi_hostadapter1 usb-storage
alias eth0 bnx2
alias eth1 bnx2
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

Next we need to create a network script for the "bond0" interface that we defined above in the "modprobe.conf" file. This will be used to configure the network properties for the virtual adapter. Once again, use your favorite text editor to create a new file called "ifcfg-bond0" in the "/etc/sysconfig/network-scripts" directory. In this file you will define the device name used above"bond0", IP address, gateway, network mask etc for the virtual adapter. Below is an example. 

 

Sample /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.25
USERCTL=no
GATEWAY=192.168.0.1
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes

 When you're done configuring the properties of the virtual adapter, save your changes and exit the editor.

 

The next step is to modify the network script for each adapter that will be added to the EtherChannel. The adapters that we'll be using in this server are eth0 and eth1. Please note your interfaces may be different, so check before continuing.

 

Start by modifying "ifcfg-<int>" using your text editor, where <int> is the interface name. In this case my file name is "ifcfg-eth0". Add the proper references to the virtual adapter created above "bond0" and remove any IP information such as IP address, gateway, netmask etc since that information will be handled by the virtual adapter. Below is an example of the "ifcfg-eth0" file. Note the bold items are required for the EtherChannel to function properly.

 

Sample /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
HWADDR=00:11:22:33:44:55
ONBOOT=yes
MASTER=bond0
SLAVE=yes
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes

 Repeat the steps above for each additional interface you add to the Etherchannel.

 

Sample /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=66:77:88:99:aa:bb
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
USERCTL=no

Now that each physical adapter has been associated to the virtual adapter, it's now time to create the EtherChannel on the Cisco switch that this server is connected to. In this example I'll be using two Cisco Catalyst 3750G switches in a stack. In addition to the added benefit of increased bandwidth that the EtherChannel provides, having multiple switches will add switch level redundancy to the EtherChannel.

 

Begin by logging into the switch, entering configuration mode, and creating the EtherChannel interface. The EtherChannel interface, also known as a "Port-channel", is a virtual interface. You can use any number from 1-48 that isn't already assigned to another EtherChannel. In this example I'll be using "28". Configure all the properties such as VLAN, switchport mode etc. that you would normally define on the physical interfaces. Please note that the properties defined in the EtherChannel will override any interface configuration you have defined on the physical interfaces this Etherchannel is assigned to.

Switch# config terminal

Switch(config)# interface Port-channel28
Switch(config-if)# description SERVER
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# spanning-tree portfast

Next we'll need to configure the physical interfaces by associating them to the EtherChannel we created above. I like to mirror the configuration of the EtherChannel on the physical interfaces, so in the event the EtherChannel on the far end of the link becomes misconfigured the server will still be accessible. The line in bold is the only required change that needs to be made on the physical interfaces.

Switch(config)# interface GigabitEthernet1/0/8
Switch(config-if)# description SERVER-1
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# channel-group 28 mode passive
Switch(config-if)# spanning-tree portfast

Repeat the process for each additional interface in the EtherChannel.

Switch(config)# interface GigabitEthernet2/0/8
Switch(config-if)# description SERVER-2
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# channel-group 28 mode passive
Switch(config-if)# spanning-tree portfast

 It's now time to restart the network service on the Linux server. Optionally you can reboot the Linux server if that's easier.

[root@SERVER network-scripts]# service network restart
Shutting down interface eth0:  /etc/sysconfig/network-scripts/ifdown-eth: [  OK  ]
Shutting down interface eth1:  /etc/sysconfig/network-scripts/ifdown-eth: [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface bond0:  [  OK  ]

Check the status of the EtherChannel on the Linux server. If the EtherChannel is operating properly you should see "Bonding Mode: IEEE 802.3ad Dynamic link aggregation" when you concatenate "/proc/net/bonding/bond0" 

[root@SERVER ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 28
        Partner Mac Address: 01:23:45:67:89:ab

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:11:22:33:44:55
Aggregator ID: 1

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 66:77:88:99:aa:bb
Aggregator ID: 1

 To check the status of the etherchannel on the Cisco switch, use the "show ether-channel" command.

Switch# show etherchannel port-channel

 

Group: 28
----------
                Port-channels in the group:
                ---------------------------

Port-channel: Po28    (Primary Aggregator)

------------

Age of the Port-channel   = 12d:18h:33m:47s
Logical slot/port   = 10/28          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Port security       = Disabled

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  0     00     Gi1/0/8  Passive            0
  0     00     Gi2/0/8  Passive            0
         
Time since last port bundled:    7d:07h:50m:28s    Gi2/0/8
Time since last port Un-bundled: 7d:07h:51m:31s    Gi1/0/8

At this point you should have a fully functional EtherChannel. If you have any questions feel free to post comments, or contact me directly.