近我这里docker单机平台正式上线使用，使用中有很多问题都一样解决，在给一个游戏项目做测试的时候，此项目由于8080端口对公网全部开放，并且安全策略没有做好（默认的tomcat模板没有删除），导致被人进行webshell，跑了很多的流量，为了解决此问题我针对tc与openvswitch本身的qos做了深入研究，最后选择openvswitch的qos作为容器的网络资源限制方法。
docker本身仅能对容器的cpu、内存做限制，而且必须是在容器运行前做，运行过程中未发现如何动态修改，并且不提供网络资源限制，所以只能使用其他软件做了。
我的docker网络没有使用默认bridge，使用none，然后绑定openvswitch的bridge，并使用pipework提供网络，所以我可以根据容器对于openvswitch的port来进行基于port的网络资源限制，好处是可以动态的修改，坏处是容器一重启还得重新做，但也可以通过其他方法来解决。
一、下面是我做测试的结果：
对于限速100m以下，其实硬盘的类型与读写速度没什么影响，但如何限速150m以上，或者无限速，那么硬盘肯定是ssd>sas>sata，所以建议进来使用sas的磁盘作为docker的挂载分区。
openvswitch默认官方文档提供的限速方法“rate limiting vm traffic using qos policing"，地址是
http://openvswitch.org/support/config-cookbooks/qos-rate-limiting/
，此限速仅能对上传做限速，下载没有办法，所以还得通过其他的qos来做限制。
二、下面是具体限速脚本
#!/bin/bash
#filename:modify_docker_container_network_limit.sh
#author:Deng Lei
#email:[email protected]
op=$1
container=$2
limit=$3 # Mbits/s
if [ -z $1 ] || [ -z $2 ]; then
echo "Usage: operation container_name limit(default:5m)"
echo "Example1: I want limit 5m in the container:test"
echo "The command is: bash `basename $0` limit test 5"
echo "Example2: I want delete network limit in the container:test"
echo "The command is: bash `basename $0` ulimit test"
exit 1
fi
if [ -z $3 ];then
limit='5m'
fi
if [ `docker inspect --format "``.`State`.`Pid`" $container &>>/dev/null && echo 0 || echo 1` -eq 1 ];then
echo "no this container:$container"
exit 1
fi
ovs_prefix='veth1pl'
container_id=`docker inspect --format "``.`State`.`Pid`" $container`
device_name=`echo ${ovs_prefix}${container_id}`
if [ $op == 'limit' ];then
for v in $device_name; do
ovs-vsctl set interface $v ingress_policing_rate=$((limit*1000))
ovs-vsctl set interface $v ingress_policing_burst=$((limit*100))
ovs-vsctl set port $v qos=@newqos -- --id=@newqos create qos type=linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- --id=@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &>>/dev/null && echo 'modify success!' || echo 'modify fail!'
done
elif [ $op == 'ulimit' ];then
for v in $device_name; do
ovs-vsctl set interface $v ingress_policing_rate=0
ovs-vsctl set interface $v ingress_policing_burst=0
ovs-vsctl clear Port $v qos &>>/dev/null && echo 'modify success!' || echo 'modify fail!'
done
fi
复制代码
此脚本使用的话，限速可以直接针对下载与上传，并且限制是统一生效的，比如我限制一个容器带宽为5m，那么下载与上传的限速都是5m，单位是bit不是byte。
三、下面是使用方法：
[root@docker-test3 tmp]# sh modify_docker_container_network_limit.sh
Usage: operation container_name limit(default:5m)
Example1: I want limit 5m in the container:test
The command is: bash modify_docker_container_network_limit.sh limit test 5
Example2: I want delete network limit in the container:test
The command is: bash modify_docker_container_network_limit.sh ulimit test
复制代码
四、下面是测试过程：
测试的方法是：
找另外一个主机172.16.1.126，然后dd生成个100m的文件/tmp/test_client.iso，在本机下载这个文件来测试下载速度，在本机dd生成100m的文件/tmp/test_server.iso把此文件上传到172.16.1.126里测试上传速度。
sata 7.5k
1、没有限制的情况
下载速度
14:12:18 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 56.82MB/s 0:00:01 (xfer#1, to-check=0/1)
sent 30 bytes received104892888 bytes 1226817.75 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:14:27 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 51.08MB/s 0:00:01 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 29969404.00 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
2、限速5m的
下载速度
14:15:27 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 580.46kB/s 0:02:56 (xfer#1, to-check=0/1)
sent 30 bytes received104892888 bytes 590946.02 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:22:10 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 616.19kB/s 0:02:46 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 571623.51 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
3、限速为10m
下载速度
14:28:55 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 1.13MB/s 0:01:28 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888bytes 1109977.97 bytes/sec
total size is 104857600 speedup is 1.00
root@fdc81b0d2508:/tmp
复制代码
上传速度
14:30:33 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 1.21MB/s 0:01:22 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 1133977.45 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
4、限速为20m
下载速度
14:32:57 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 2.27MB/s 0:00:44 (xfer#1, to-check=0/1)
sent 30 bytes received104892888 bytes 2305338.86 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:33:59 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 2.45MB/s 0:00:40 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 2305338.77 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
5、限速为50m
下载速度
14:35:20 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 5.67MB/s 0:00:17 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888bytes 5379124.00 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:35:54 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 6.33MB/s 0:00:15 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 5116727.51 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
6、限速100m
下载速度
14:37:18 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 11.35MB/s 0:00:08 (xfer#1, to-check=0/1)
sent 30 bytes received104892888 bytes 8391433.44 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:37:39 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 13.19MB/s 0:00:07 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 11041359.37 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
7、限速150m
下载速度
14:38:39 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 11.35MB/s 0:00:08 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888bytes 11041359.79 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
14:38:56 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 13.14MB/s 0:00:07 (xfer#1, to-check=0/1)
sent 104892883 bytes received31 bytes 11041359.37 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
下面是使用sas 7.5k硬盘的测试结果
8、无限速的
下载速度
17:11:07 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
2147483648 100% 106.42MB/s 0:00:19 (xfer#1, to-check=0/1)
sent 30 bytes received 2148204633 bytes 95475762.80 bytes/sec
total size is 2147483648 speedup is 1.00
复制代码
上传速度
17:20:06 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
2147483648 100% 112.51MB/s 0:00:18 (xfer#1, to-check=0/1)
sent 2148204628 bytes received 31 bytes 95475762.62 bytes/sec
total size is 2147483648 speedup is 1.00
复制代码
9、5m限速
下载速度
17:30:30 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 591.44kB/s 0:02:53 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 587635.39 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:38:57 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 590.35kB/s 0:02:53 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 574755.69 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
10、限速10m
下载速度
17:42:54 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 1.15MB/s 0:01:26 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 1146370.69 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:44:31 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 1.16MB/s 0:01:26 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 1146370.64 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
11、限速20m
下载速度
17:47:02 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 2.32MB/s 0:00:43 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 2162740.58 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:48:06 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 2.38MB/s 0:00:42 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 2255761.59 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
12、限速50m
下载速度
17:52:52 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 5.84MB/s 0:00:17 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 5116727.71 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:53:15 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 6.10MB/s 0:00:16 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 5993880.80 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
13、限速为100m
下载速度
17:55:16 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 11.75MB/s 0:00:08 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 9989801.71 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:55:39 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 12.72MB/s 0:00:07 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 11041359.37 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
14、限速150m
下载速度
17:56:58 # rsync -avz --progress 172.16.1.53:/tmp/test_client.iso/tmp/
[email protected]'s password:
receiving incremental file list
test_client.iso
104857600 100% 17.69MB/s 0:00:05 (xfer#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 13985722.40 bytes/sec
total size is 104857600 speedup is 1.00
复制代码
上传速度
17:57:18 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
[email protected]'s password:
sending incremental file list
test_server.iso
104857600 100% 20.44MB/s 0:00:04 (xfer#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 16137371.38 bytes/sec
total size is 104857600 speedup is 1.00
复制代码