Tags: bonding, CentOS, cisco, Debian, gentoo, ios, lacp, linux, networking, redhat
(Update: Feb 5, 2010 – I even more recently obtained a Cisco IOS switch and have included the configuration bits for IOS below.)
I recently obtained a Dell PowerConnect 5224 Gigabit switch which has the ability to combine multiple twisted-pair or fiber Ethernet links into one fault-tolerant and load balanced logical link. It also appears that its configuration syntax is very similar to that of a Cisco switch. In Linux this is called bonding, in switches its commonly referred to as a port channel. Either way, Its using the LACP (802.1ad) Protocol behind the scenes.
Configuring the switch for LACP bonding
Cisco IOS switch LACP configuration
Enabling LACP across two ports in IOS is pretty straightforward. The first thing to do is associate the ports with the channel-group. This is good to do early so that when you apply switchport parameters to the Port-channel interface it automagically applies them to the GigabigEthernet interfaces.
Here are the relevant portions of my running configuration.
interface Port-channel2 description LACP Channel for mk2 switchport trunk encapsulation dot1q switchport trunk allowed vlan 1,2 switchport mode trunk spanning-tree portfast trunk ! interface GigabitEthernet1/0/23 description mk2 eth0 switchport trunk encapsulation dot1q switchport mode trunk channel-group 2 mode active ! interface GigabitEthernet1/0/24 description mk2 eth1 switchport trunk encapsulation dot1q switchport mode trunk channel-group 2 mode active ! |
Dell PowerConnect switch LACP configuration
The Dell switch configuration is surprisingly easy. A port-channel is automatically created when the Linux host brings up it’s bond interface(s). Just figure out which ports you want to use for your bond and enable LACP on them. I used ports 1/23 and 1/24 (ports 23 & 24 on switch 1).
Vty-0#config Vty-0(config)#interface ethernet 1/23 Vty-0(config-if)#lacp Vty-0(config-if)#exit Vty-0(config)#interface ethernet 1/24 Vty-0(config-if)#lacp Vty-0(config-if)#exit |
‘show run’ now indicates that the selected ports are LACP enabled.
Vty-0#show run building running-config, please wait..... ... ! interface ethernet 1/23 switchport allowed vlan add 1 untagged switchport native vlan 1 lacp ! interface ethernet 1/24 switchport allowed vlan add 1 untagged switchport native vlan 1 lacp ! ... |
At this point your port-channel will be down. Don’t worry, it will automagically come up when the Linux host brings up the bond interface. You can verify that its down by issuing the following:
Vty-0#show interfaces status port-channel 1 % Trunk 1 does not exist. |
Note: This assumes you have no pre-existing port-channels, if you do have other port-channels configured you should iterate the port-channel number to be one more than the number of already defined port-channels.
Configuring the Linux host for LACP bonding:
There are a few places where you define the parameters of the bond. The kernel module defies the protocol, frequency and other attributes of the low-level bond channel configuration. The command ifenslave will create a bond device and allow you to manage the Ethernet devices within it (add/remove,etc.). Finally the network address configuration is handled by ifconfig, consistent with most other network interfaces in Linux. Luckily most of this is taken care of automatically by the networking init scripts.
Linux Kernel Module Configuration
LACP is referred to in linux as bonding mode 4, so we need to inform the kernel module to use this bonding mode. We’ll also pass it a few other parameters like the frequency of which to scan for changes in status.
Add the following to your module config file, in gentoo this is /etc/modules.autoload.d/kernel-2.6. This will pass the following options to the kernel module the next time it is inserted.
Red Hat and CentOS Kernel Module Configuration
#/etc/modprobe.conf alias bond0 bonding options bond0 miimon=100 mode=4 lacp_rate=1 |
RHEL 6 and CentOS 6 Kernel Module Configuration
#/etc/modprobe.d/bonding.conf alias bond0 bonding options bond0 miimon=100 mode=4 lacp_rate=1 |
Debian Kernel Module Configuration
# /etc/modules: kernel modules to load at boot time. bonding mode=4 miimon=100 lacp_rate=1 |
Ubuntu Kernel Module Configuration
# /etc/modprobe.d/bonding.conf bonding mode=4 miimon=100 lacp_rate=1 |
Gentoo Kernel Module Setup
#/etc/modules.autoload.d/kernel-2.6 bonding miimon=100 mode=4 lacp_rate=1 |
Linux Network Configuration
Red Hat and CentOS Network Setup
#/etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0ONBOOT=yesBOOTPROTO=noneUSERCTL=noMASTER=bond0SLAVE=yes |
#/etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1ONBOOT=yesBOOTPROTO=noneUSERCTL=noMASTER=bond0SLAVE=yes |
#/etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0IPADDR=10.0.0.80NETMASK=255.255.255.0BROADCAST=10.0.0.255GATEWAY=10.0.0.1ONBOOT=yesBOOTPROTO=noneUSERCTL=no |
Debian / Ubuntu Network Setup
#/etc/network/interfaces # This file describes the network interfaces available on your system# and how to activate them. For more information, see interfaces(5). auto bond0 iface bond0 inet static address 10.0.0.80 gateway 10.0.0.1 broadcast 10.0.0.255 netmask 255.255.255.0 up /sbin/ifenslave bond0 eth1 eth2 down /sbin/ifenslave -d bond0 eth0 eth1 |
*Note* This is dependant upon the ifenslave package, to install run the following:
apt-get install ifenslave |
Ubuntu 10.04 or newer support an updated interfaces(5) syntax
#/etc/network/interfaces # This file describes the network interfaces available on your system# and how to activate them. For more information, see interfaces(5). auto eth0 iface eth0 inet manual bond-master bond0 auto eth1 iface eth1 inet manual bond-master bond0 auto bond0 iface bond0 inet static address 10.0.0.80 gateway 10.0.0.1 netmask 255.255.255.0 bond-mode 802.3ad bond-miimon 100bond-lacp-rate 4bond-slaves none |
Gentoo Network Setup
#/etc/conf.d/net config_eth0=( "null" )config_eth1=( "null" ) slaves_bond0="eth0 eth1" config_bond0=( "10.0.0.80/24" ) |
We also need to create a symlink in /etc/init.d for the new bond0 interface and turn off eth0 as it is controlled by the bond now. The following will disable eth0 and enable bond0 on boot.
cd /etc/init.d ln -s net.lo net.bond0 rc-update del eth0 default rc-update add bond0 default |
Now you can bring up the bond interface.
/etc/init.d/net.bond0 start |
Checking the Status of the bonded interface
You can check the status of your bond now from within Linux by using the /proc and /sys interfaces into the Linux bond driver.
$ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 17 Partner Key: 1 Partner Mac Address: 00:77:54:71:a8:6f Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:99:97:60:9d:48 Aggregator ID: 1 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:00:85:60:9d:49 Aggregator ID: 1 |
You can check the bond from the switch.
Cisco IOS
Switch#show interfaces Port-channel 2 Port-channel2 is up, line protocol is up (connected) Hardware is EtherChannel, address is 001b.0dbf.ba17 (bia 001b.0dbf.ba17) Description: LACP Channel for mk2 MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Full-duplex, 1000Mb/s, link type is auto, media type is unknown input flow-control is off, output flow-control is unsupported Members in this channel: Gi1/0/23 Gi1/0/24 ARP type: ARPA, ARP Timeout 04:00:00 Last input 1d23h, output 00:00:01, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 5000 bits/sec, 7 packets/sec 1060041 packets input, 193406916 bytes, 0 no buffer Received 18241 broadcasts (0 multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 11873 multicast, 0 pause input 0 input packets with dribble condition detected 3181997 packets output, 2735804051 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out |
Dell PowerConnect:
Vty-0#show interfaces status port-channel 1 Information of Trunk 1 Basic information: Port type: 1000t Mac address: 00-30-F1-71-A8-82 Configuration: Name: Port admin: Up Speed-duplex: Auto Capabilities: 10half, 10full, 100half, 100full, 1000full, Flow control: Disabled Current status: Created by: Lacp Link status: Up Port operation status: Up Operation speed-duplex: 1000full Flow control type: None Member Ports: Eth1/23, Eth1/24, |
That’s is all the configuration work that I needed to perform, I hope it saves you time (I spent a while digging through dell’s site and the linux kernel docs to find the right combination of options). Please, let me know if you had troubles with these directions or if you have questions. keith (at) backdrift.org
|
|
|
|
18 Responses to “How to configure network bonding in Linux”
Join the Conversation
Name (required)
Mail (will not be published) (required)
Website
July 2nd, 2010 at 7:27 am
Hello,
there is a problem in the file modprobe.conf, you write “options bond0..” but it is “options bonding…”
thanks for this howto
[Reply]
Gavin Says:
August 20th, 2010 at 1:28 pm
Thank you. This was exactly what I was looking for. Very clear and easy to follow. Bonding/aggregating links to my storage node has solved a mysterious dying port problem on my powerconnect switch. Cheers.
[Reply]
January 29th, 2011 at 7:51 am
Before I try this – I have a question. Will it allow a single process to transfer data over both ports to a single destination? That is, if I have 2 gigabit ports and want to copy a file will the copy use both ports and possibly go at more than 1 gigabit per second? Or is the selection of a port deterministic and the transfer will use only one port?
Thanks for any guidance.
Dan Feenberg
NBER
[Reply]
February 24th, 2011 at 4:04 pm
Great how-to.
Daniel, as I understand it, you’re not going to get a 2gig session, however if someone else connects to the same endpoint as you at or near the same time, I believe that their session/connection will go over the unused 1gig line. so instead of both of you clogging up one port, you’ll be split and each able to have your own port. Any successive connections will probably be determined by the OS.
[Reply]
April 24th, 2011 at 6:35 pm
for scaling above 1gbit between two hosts:
hash by src-ip – dst-ip-port – some switches can do that.
also note that the bonding driver in centos5 / rhel5 is said to be not smp-capable, bringing in its’ own performance issues (see the presentation about mysql by facebook)
[Reply]
Linux Ethernet Bonding – WTF?! « Bill Sigmund Says:
June 13th, 2011 at 8:51 am
[...] http://backdrift.org/howtonetworkbonding [...]
February 8th, 2012 at 5:50 am
Hi,
i am using centos 5.5 as a firewall and have three ethernet ports on it as:
eth0 – 192.168.2.1 (LAN port)
eth1 – 202.xx.xx.94 (WAN1)
eth2 – 202.xx.xx.93 (WAN2)
i want to bond the two WAN ports for outgoing traffic, can i do that?
and if yes, what should i do to their assigned public IPs?… configure or not? and what IP should i assign to bond0 interface in that case?
Please help!!!
regards
[Reply]
jorge robles Reply:
May 28th, 2012 at 9:11 pm
let me know if you have a solution, i’m having the same issue as you.
[Reply]
Keith Reply:
May 28th, 2012 at 9:23 pm
As long as these are ethernet interfaces there is no distinction between LAN and WAN. You would simply assign the appropriate IP addresses to the bond interface. You can assign multiple IP addresses using interface aliases which would look something like this:
bond0:0 202.xx.xx.93
bond0:1 202.xx.xx.94
Hope that helps
[Reply]
February 28th, 2012 at 7:51 am
Thanks for the quick infos about port channel / lacp. peace.
[Reply]
March 10th, 2012 at 2:47 am
If you use LACP for bonding (mode=4) then you will in fact double link speed (for example 2x 100 Mbps = 200 Mbps or 2x 1 Gbps = 2 Gbps).
As far as I know if one port goes down your link will still be up but at 1 Gbps.
There are some redundancy modes as well which will not increase speed but just increase redundancy. LACP more or less does both (though speed is halved if one link is down), but the mode where all data is sent twice will deliver the best redundancy; it’s like RAID1 in network throughput
[Reply]
TP-Link TL-SG3210, 8-Port, Managed Switch » Philipp Klaus's Computing Blog Says:
April 13th, 2012 at 3:59 pm
[...] The linux kernel module bonding implements LACP when loaded with the parameter mode=4. More about LACP configuration on IOS switches, Dell switches and Linux machines on http://backdrift.org/howtonetworkbonding. [...]
Setting up Openfiler with 3x Dell MD1200 enclosures « IT NotepadSays:
April 27th, 2012 at 10:05 am
[...] Network refs:http://support.dell.com/support/edocs/network/m8024k/en/ucg/html/linkagg.htmhttp://backdrift.org/howtonetworkbonding [...]
May 17th, 2012 at 5:23 am
Hello all. Thank you for your post, its very interesting.
I have a question. I have a machine with 6 ethernet nics. So, I would like to set up a bond interface using mode=4 for improve band with speed and fault tolerance. But, I would like to connect three of those nics to one chasis and the rest to other chasis, Its possible to configure one lacp port in one switch and other one in the other chasis?
I dont know if this question its so stupid…
Thank you.
[Reply]
Keith Reply:
May 17th, 2012 at 10:38 am
Hey @Antonio, typically you would accomplish that by stacking your two switches together and creating a lacp group with the ports split between the two switches. If your switches aren’t stackable, you could accomplish a similar level of redundancy and performance by utilizing a different bonding mode.
Hope that helps
[Reply]
June 13th, 2012 at 3:49 pm
Just saw a bit of a typo that I wanted to help clear up. With the modprobe.conf file, if you have a more current version of redhat/centos this is not the file you want to use. Try instead /etc/modprobe.d/bonding.conf . You’ll have to create the file. Write the same information
studyhat.blogspot.com/2011/11/redhat-centos-6-bonding.html
[Reply]
June 23rd, 2012 at 9:17 am
@Antonio: your switches needs a feature that is called Multi-Chassis Link Aggregation to accomplish that (other switch vendors may have a different name for that)
cheers
[Reply]
How to configure network bonding in Linux | Backdrift « My BlogSays:
September 10th, 2012 at 11:57 pm
[...] How to configure network bonding in Linux | Backdrift. Share this:TwitterFacebookLike this:LikeBe the first to like this. [...]