Lvs下使用Docker扩容实验
Lvs下使用Docker扩容实验
拓扑
搭环境
LVS
1. 安装组件：keepalived和lvs
注意，keepalived apt-get安装的版本有bug，在自动测试rs时，一直报出错，不能自动探测，下载1.2.19版本的源码编译安装后觖决。
apt-getupdate
apt-getinstall ipvsadm
apt-getinstall keepalived
2. 配置
主要是/etc/keepalived/keepalived.conf文件。
root@ubuntu:~/dev/keepalived-1.2.19#cat /etc/keepalived/keepalived.conf
#GlobalConfiguration
global_defs{
lvs_id lvs_m
}
#VRRPConfiguration
vrrp_instanceZGYE_LVS {
state MASTER #LVS_S上修发成BACKUP
interface eth0
virtual_router_id 60
priority 100 #LVS_S上修发成50,小于LVS_M，以确定哪个为主
advert_int 5
authentication {
auth_type PASS
auth_pass zgye_test
}
virtual_ipaddress {
172.18.2.100
}
#Virtual Server Configu - for WWW service
virtual_server 172.18.2.100 80 {
delay_loop 1
lb_algo rr
lb_kind DR
persistence_timerout 60
protocol TCP
#Real Server 1 Config - web1
real_server 172.18.2.83 80 {
weight 1
TCP_CHECK {
connection_timeout10
nb_get_retry 3
delay_before_retry3
connect_port 80
}
}
real_server 172.18.2.85 80 {
weight 1
TCP_CHECK {
connection_timeout10
nb_get_retry 3
delay_before_retry3
connect_port 80
}
}
real_server 172.18.2.125 80 {
weight 1
TCP_CHECK {
connection_timeout10
nb_get_retry 3
delay_before_retry3
connect_port 80
}
}
}
}
RS
VM
1. 安装组件: nginx
apt-get install nginx
2. 配置nginx
主要是修改 /usr/share/nginx/html/index.html，以区分哪个rs做了应答。
3. 增加vip
ifconfig eth0:0 172.18.2.100 broadcast172.18.2.100 netmask 255.255.255.255 up
route add -host 172.18.2.100 deveth0:0
4. 修改arp过滤
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
DOCKER
1. 安装docker
apt-key adv --keyserverhkp://keyserver.ubuntu.com:80 --recv-keys36A1D7869245C8950F966E92D8576A8BA88D21E9
sh -c "echo debhttps://get.docker.com/ubuntu docker main >/etc/apt/sources.list.d/docker.list"
apt-get update
apt-get install -y lxc-docker gitmake
source/etc/bash_completion.d/docker
2. 下载nginx镜像
docker pull nginx
3. 运行nginx容器
docker run -it --net=none--name=nginx2 nginx /bin/bash
4. 配置固定ip
使用脚本配置：
./docker_static_ip.sh a2ad572f9ca2172.18.2.125 255.255.255.128 172.18.2.1 nginx_2
其中脚本为：
root@OA-72:~# catdocker_static_ip.sh
#/bin/bash
BR_DEV=br0
if [ -z $1 ] || [ -z $2 ] || [ -z$3 ] || [ -z $4 ] || [ -z $5 ];
then
echo "*****Input the necessaryparameters: CONTAINERID IP MASK GATEWAY ETHNAME"
echo "*****Call the script like:sh manual_con_static_ip.sh b0e18b6a4432192.168.5.123 24 192.168.5.1 deth0"
exit
fi
CONTAINERID=$1
SETIP=$2
SETMASK=$3
GATEWAY=$4
ETHNAME=$5
ifconfig $ETHNAME > /dev/null2>&1
if [ $? -eq 0 ]; then
read -p "$ETHNAME exist,do you want delelte it? y/n " del
if [[ $del == 'y' ]]; then
ip link del $ETHNAME
else
exit
fi
fi
#
pid=`docker inspect -f'{{.State.Pid}}' $CONTAINERID`
mkdir -p /var/run/netns
find -L /var/run/netns -type l-delete
if [ -f /var/run/netns/$pid ]; then
rm -f /var/run/netns/$pid
fi
ln -s /proc/$pid/ns/net/var/run/netns/$pid
#
ip link add $ETHNAME type veth peername B
brctl addif $BR_DEV $ETHNAME
ip link set $ETHNAME up
ip link set B netns $pid
#delete eth0 in docker
ip netns exec $pid ip link del eth0> /dev/null 2>&1
#set eth0 in docker
ip netns exec $pid ip link set devB name eth0
ip netns exec $pid ip link set eth0up
ip netns exec $pid ip addr add$SETIP/$SETMASK dev eth0
ip netns exec $pid ip route adddefault via $GATEWAY
5. 配置vip
参考脚本中的配置ip命令：
ip netns exec 7397 ip addr add 172.18.2.100/25dev eth0
6. 启动nginx服务
docker exec -d nginx2 service nginxstart
数据报文
Client:发请求到vip上。
LVS:只接收请求，并修改请求发到rs，由rs做应答。
RS：收到第一个syn的smac为lvs的，但回的syn\ack的dmac变成了client的mac。
测试
在cilent上通过
root@ubuntu:~# for ((i=0; i<100; i++)); do curlhttp://172.18.2.100:80; done
访问来测试均衡情况
在113上可以看到：
root@ubuntu:~# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
->RemoteAddress:Port ForwardWeight ActiveConn InActConn
TCP 172.18.2.100:80 rr
->172.18.2.83:80 Route 1 0 33
->172.18.2.85:80 Route 1 0 33
->172.18.2.125:80 Route 1 0 34