高并发、大流量网卡调优

周五晚上LVS集群的一台LB由于CPU单核耗尽挂掉了，最后查到原因是网卡调优没有生效，今天查了一下网卡调优的资料，感谢同事higkoo给我讲解了一下这方面的东西，下面是关于我们公司网卡调优的相关知识，欢迎大家共同探讨，让我们的机器跑的更high。

1、Broadcom的网卡建议关闭GRO功能

ethtool  -K eth0 gro off
ethtool  -K eth1 gro off
ethtool  -K eth2 gro off
ethtool  -K eth3 gro off

2、关闭irqbalance服务并手动分配网卡中断

service irqbalance stop
chkconfig irqbalance off
# 查看网卡中断号
grep  ethx  /proc/interrupts
# 分配到每颗颗CPU核上
cat  /proc/irq/ {84,85,86,87,88,89,90,91,92,93} /smp_affinity
echo  1 >  /proc/irq/84/smp_affinity
echo  2 >  /proc/irq/85/smp_affinity
echo  4 >  /proc/irq/86/smp_affinity
echo  8 >  /proc/irq/87/smp_affinity
echo  10 >  /proc/irq/88/smp_affinity
echo  20 >  /proc/irq/89/smp_affinity
echo  40 >  /proc/irq/90/smp_affinity
echo  80 >  /proc/irq/91/smp_affinity
echo  100 >  /proc/irq/92/smp_affinity
echo  200 >  /proc/irq/93/smp_affinity
PS:smp_affinity的值可以用下面脚本算哈，此脚本来自：http: //rfyiamcool .blog.51cto.com /1030776/1335700
#!/bin/bash
#
echo  "统计cpu的16进制"
[ $ # -ne 1 ] && echo ‘$1 is Cpu core number’ && exit 1
CCN=$1
echo  “Print eth0 affinity”
for ((i=0; i<${CCN}; i++))
do
echo  ==============================
echo  "Cpu Core $i is affinity"
((affinity=(1<
echo  "obase=16;${affinity}"  |  bc
done
使用方法：sh 脚本名字 空格 cpu核数

 

3、开启网卡的RPS功能 （Linux内核2.6.38或以上版本支持）

# Enable RPS (Receive Packet Steering)
rfc=4096
cc=$( grep  -c processor  /proc/cpuinfo )
rsfe=$( echo  $cc*$rfc |  bc )
sysctl -w net.core.rps_sock_flow_entries=$rsfe
for  fileRps  in  $( ls  /sys/class/net/eth * /queues/rx- * /rps_cpus )
do
echo  fff > $fileRps
done
for  fileRfc  in  $( ls  /sys/class/net/eth * /queues/rx- * /rps_flow_cnt )
do
echo  $rfc > $fileRfc
done
tail  /sys/class/net/eth * /queues/rx- */{rps_cpus,rps_flow_cnt}

献上一个完整的脚本：

vi  /opt/sbin/change_irq .sh
#!/bin/bash
ethtool  -K eth0 gro off
ethtool  -K eth1 gro off
ethtool  -K eth2 gro off
ethtool  -K eth3 gro off
service irqbalance stop
chkconfig irqbalance off
cat  /proc/irq/ {84,85,86,87,88,89,90,91,92,93} /smp_affinity
echo  1 >  /proc/irq/84/smp_affinity
echo  2 >  /proc/irq/85/smp_affinity
echo  4 >  /proc/irq/86/smp_affinity
echo  8 >  /proc/irq/87/smp_affinity
echo  10 >  /proc/irq/88/smp_affinity
echo  20 >  /proc/irq/89/smp_affinity
echo  40 >  /proc/irq/90/smp_affinity
echo  80 >  /proc/irq/91/smp_affinity
echo  100 >  /proc/irq/92/smp_affinity
echo  200 >  /proc/irq/93/smp_affinity
# Enable RPS (Receive Packet Steering)
rfc=4096
cc=$( grep  -c processor  /proc/cpuinfo )
rsfe=$( echo  $cc*$rfc |  bc )
sysctl -w net.core.rps_sock_flow_entries=$rsfe
for  fileRps  in  $( ls  /sys/class/net/eth * /queues/rx- * /rps_cpus )
do
echo  fff > $fileRps
done
for  fileRfc  in  $( ls  /sys/class/net/eth * /queues/rx- * /rps_flow_cnt )
do
echo  $rfc > $fileRfc
done
tail  /sys/class/net/eth * /queues/rx- */{rps_cpus,rps_flow_cnt}
chmod  +x  /opt/sbin/change_irq .sh
echo  "/opt/sbin/change_irq.sh"  >>  /etc/rc . local
PS：记得修改网卡中断号，别直接拿来用哦