LVS ，Nginx测试
LVS Nginx负载均衡方案测试
方案目标
为单点nginx分发节点提供高可用，负载均衡保证
测试方案及配置
Firewall:测试环境采用Vmware station搭建。内部网络采用由vmware基于NAT方式建立的虚拟网络，内部网络地址采用192.168.118.x/24，宿主主机作为虚拟网络的网关，外网地址为10.48.192.26,内网地址为192.168.118.2.
DirectorServer:采用一台虚拟主机（内存1G，硬盘40G，所有虚拟主机配置相同）作为Director Server，增加地址192.168.118.50作为提供服务的服务地址.
LVS配置
-A -t s1.test.domain:http -s rr -a -t s1.test.domain:http -r r1.test.domain:http -g-w 1 -a -t s1.test.domain:http -r r2.test.domain:http -g-w 2
ProxyServer 1/Proxy Server 2:除本地ip外，增加虚拟的服务器IP地址192.168.118.50,同时关闭arp的响应，避免和DirectorServer发生冲突,启动，关闭脚本如下:
[root@r1 sbin]# cat ~/real_server.sh
#!/bin/bash
#description : start realserver
VIP=192.168.118.50
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
case "$1" in
start)
echo " start LVS of REALServer"
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask255.255.255.255 up
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
/sbin/ifconfig lo:0 down
echo "close LVS Directorserver"
echo "0">/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0">/proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
Proxy Server通过tengine 1.5.2配置，通过sessionstickly模块配置集群，保证session的有效性。后端节点集群配置如下:
[root@r1 sbin]# cat upstream.conf
upstream backend
{
servera1.test.domain:8080;
servera1.test.domain:8180;
servera2.test.domain:8080;
servera2.test.domain:8180;
session_sticky;
}
为了便于了解当前输出的ProxyServer是哪一台，通过footer模块输出一些注释到输出的页面当中，具体虚拟主机配置如下:
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
location /demo {
footer "<!-- $hostname, $year/$month/$day $hour:$minute:$second,$request -->";
proxy_passhttp://backend;
}
}
为了测试方案是否可以达到粘性分发的效果，通过jsp代码测试当前session的效果，主要代码如下：
<%if(request.getSession().getAttribute("test") == null){
Stringserver = request.getLocalAddr();
SimpleDateFormatsdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Stringtime = sdf.format(new java.util.Date());
request.getSession().setAttribute("test",server);
request.getSession().setAttribute("time",time);
%>NEW<%}%>
<br> Session:<%=session.getId() %>
<br>Host:<%=request.getSession().getAttribute("test") %>
<br>Time:<%=request.getSession().getAttribute("time") %>
<br>UserName:<%=System.getProperty("user.name") %>
正常情况下，用户第一次出现"NEW"生成新的jsessionid,后续应该不会出现，以此作为判断session是否失效的依据。
测试场景
1.通过不同的Proxy Server时，用户的session不会发生变化
测试方法：访问http://10.48.192.26/demo/Hello.jsp,刷新页面看获取的页面代码，结果如下
<?xml version="1.0"encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML1.0 Transitional//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<htmlxmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type"content="text/html; charset=ISO-8859-1" />
<title>Insert title here</title>
</head>
<body>
<br>Session:0B0D27823EA7EBFC1C67EF007B24085E
<br>Host:192.168.118.31
<br>Time:2013-08-30 22:27:18
<br>UserName:jboss
</body>
</html><!-- r1.test.domain, 2013/08/30 07:51:32, GET/demo/Hello.jsp HTTP/1.1,192.168.118.31:8180 -->
<?xml version="1.0"encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML1.0 Transitional//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<htmlxmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type"content="text/html; charset=ISO-8859-1" />
<title>Insert title here</title>
</head>
<body>
<br>Session:0B0D27823EA7EBFC1C67EF007B24085E
<br>Host:192.168.118.31
<br>Time:2013-08-30 22:27:18
<br>UserName:jboss
</body>
</html><!-- r2.test.domain, 2013/08/30 07:53:43, GET/demo/Hello.jsp HTTP/1.1,192.168.118.31:8180 -->
通过不同前端分发时，连接后端应用服务器的节点始终是同一节点,该项测试通过
2.单台Proxy Server 异常而不能正常提供服务的情况下，用户的反馈和session不会发生变化
测试方法：访问页面，然后关闭一台Proxy Server,测试结果如下:
<?xml version="1.0"encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML1.0 Transitional//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<htmlxmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type"content="text/html; charset=ISO-8859-1" />
<title>Insert title here</title>
</head>
<body>
<br>Session:0B0D27823EA7EBFC1C67EF007B24085E
<br>Host:192.168.118.31
<br>Time:2013-08-30 22:27:18
<br>UserName:jboss
</body>
</html><!-- r1.test.domain, 2013/08/3007:57:06, GET /demo/Hello.jsp HTTP/1.1,192.168.118.31:8180 -->
关闭Proxy Server 1
[root@r1 ~]# ./real_server.sh stop close LVS Directorserver
再次刷新页面
<?xml version="1.0"encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML1.0 Transitional//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<htmlxmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type"content="text/html; charset=ISO-8859-1" />
<title>Insert title here</title>
</head>
<body>
<br>Session:0B0D27823EA7EBFC1C67EF007B24085E
<br>Host:192.168.118.31
<br>Time:2013-08-30 22:27:18
<br>UserName:jboss
</body>
</html><!-- r2.test.domain, 2013/08/3007:58:21, GET /demo/Hello.jsp HTTP/1.1,192.168.118.31:8180 -->
页面始终是通过Proxy Server 2进行分发，服务未发生任何异常，测试通过
3.在关闭单台jboss实例的情况下，不会影响其他服务器的运行
测试方法：关闭一台jboss实例，测试结果如下
关闭当前实例192.168.118.31:8180
[jboss@a1 jboss]$ ps aux|grep java jboss 2751 2.1 39.7 1725280 537291pxs/0 Sl 17:59 6:23 java -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m-Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000-Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true-Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jar org.jboss.Main-b 0.0.0.0 -c app01 jboss 2788 2.1 39.2 1659744 530213pxs/0 Sl 17:59 6:20 java -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m-Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000-Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true-Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jarorg.jboss.Main -b 0.0.0.0 -c app02 jboss 9485 0.0 0.0 103244 824 pts/0 S+ 23:00 0:00 grep java [jboss@a1 jboss]$ kill -9 2751 [jboss@a1 jboss]$ ps aux|grep java jboss 2788 2.1 39.2 1659744 530219pxs/0 Sl 17:59 6:20 java -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m-Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000-Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true-Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jarorg.jboss.Main -b 0.0.0.0 -c app02 jboss 9499 0.0 0.0 103244 828 pts/0 S+ 23:01 0:00 grep java [jboss@a1 jboss]$ kill -9 2788 [jboss@a1 jboss]$ ps aux|grep java jboss 9502 0.0 0.0 103244 828 pts/0 S+ 23:02 0:00 grep java
刷新页面
<?xml version="1.0"encoding="ISO-8859-1" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML1.0 Transitional//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<htmlxmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type"content="text/html; charset=ISO-8859-1" />
<title>Insert title here</title>
</head>
<body>
NEW
<br>Session:0C6ED6B8B7C40C1A74B7648128715A60
<br>Host:192.168.118.32
<br>Time:2013-08-30 23:02:31
<br>UserName:jboss
</body>
</html><!-- r1.test.domain, 2013/08/3008:02:32, GET /demo/Hello.jsp HTTP/1.1,192.168.118.31:8180,192.168.118.31:8080, 192.168.118.32:8080 -->
jboss实例发生了切换，同时生成了新的jsessionid，测试通过
测试结论
通过lvs实现对nginx负载均衡的分发，再通过nginx对内容的分发，能够给nginx提供高可用性，在分发节点异常时，可以保证服务的不中断，在发生分发节点的切换时，对客户来说是透明的，不影响客户段的响应。同时因为lvs良好的可扩展性，为整套方案提供了非常好的可扩展性，可以通过增加的新的分发节点来扩展分发层的负载能力，同时能够保持session的一致。