linux nginx 负载均衡 图片缓存 tomcat集群 session 共享

原文：http://www.weiruoyu.cn/?p=504

1 安装包下载

nginx-0.8.51下载
tomcat-6.0.20下载
pcre-8.12下载

2 环境描述

• window7下运行的centos5.4的Vmware7虚拟机

• nginx-0.8.51

• tomcat-6.0.20

• pcre-8.12

• jdk-6u7-linux-i586

3 安装步骤

3.1 安装nginx所需的pcre

tar xvzf pcre-8.12.tar.gz 
cd pcre-8.12
./configure
make && make install

也可以采用yum方式安装

yum -y install pcre-devel

3.2 安装nginx

tar xvzf nginx-0.8.51.tar.gz 
cd nginx-0.8.51
./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
make && make install

这里为了简单没有加入user和group信息,如果加入命令如下:

./configure --user=www --group=www --prefix=/usr/local/webserver/nginx --with-http_stub_status_module --with-http_ssl_module

如果不安装pcre则会提示

./configure: error: the HTTP rewrite module requires the PCRE library.

3.3 配置nginx

user nobody;
worker_processes 1;

error_log logs/error.log info;

#pid    logs/nginx.pid;


events {
use epoll;
  worker_connections 1024;
}


http {
  include    mime.types;
  default_type application/octet-stream;

  log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

  sendfile    on;

  keepalive_timeout 65;

#设置Web缓存区名称为cache_one，内存缓存空间大小为100MB，1天没有被访问的内容自动清除，硬盘缓存空间大小为1GB
  proxy_cache_path /usr/local/nginx/cache_data levels=1:2 keys_zone=cache_one:100m inactive=1d max_size=1g;

  upstream www.test.com {
#ip_hash策略将同一IP的所有请求都转发到同一应用服务器
   ip_hash;
   server localhost:8080;
   server localhost:8081;
}

  server {
    listen    80;
    server_name localhost;
    charset utf-8;
    location /{
      root  html;
      index index.html index.htm;
      proxy_pass http://www.test.com;
      proxy_set_header X-Real-IP $remote_addr;
      client_max_body_size 100m;
}

    error_page  500502503504/50x.html;

    location ~*\.(gif|jpg|jpeg|png|bmp)$ {
      proxy_cache cache_one;
      proxy_cache_valid 2003023041h;
      proxy_cache_key $host$uri$is_args$args;
      proxy_pass http://www.test.com;
      add_header Last-Modified $date_gmt;
      add_header Via $server_addr;
      expires 30d;
}
}
}

3.4 安装jdk1.6

不详细说明了，如不清楚，请google

3.5 安装两个tomcat6

tar xvzf apache-tomcat-6.0.20.tar.gz
mv apache-tomcat-6.0.20/usr/local/tomcat6
cd /usr/local
cp -r tomcat6/ tomcat6_1

• 修改tomcat6_1中的端口配置避免两个tomcat启动时端口冲突

执行命令

vi /usr/local/tomcat6_1/conf/server.xml

修改

...
<!--8005修改8006-->
<Server port="8006" shutdown="SHUTDOWN">
...
<!--8080修改为8081-->
<Connector port="8081" protocol="HTTP/1.1"
        connectionTimeout="20000"
        redirectPort="8443"/>
...
<!--8009修改为8010-->
<Connector port="8010" protocol="AJP/1.3" redirectPort="8443"/>
...

============完成上面既可以实现负载均衡======

3.6 配置tomcat集群实现session共享

• 修改server.xml配置tomcat集群

修改Engine部分配置

<!--加入jvmRoute="jvm1" 两个tomcat的值不能相同-->
<Enginename="Catalina"defaultHost="localhost"jvmRoute="jvm1">

修改Cluster部分配置

<ClusterclassName="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">

<ManagerclassName="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>

<ChannelclassName="org.apache.catalina.tribes.group.GroupChannel">
<MembershipclassName="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="3000"/>
<ReceiverclassName="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="localhost"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>

<SenderclassName="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<TransportclassName="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<InterceptorclassName="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<InterceptorclassName="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>

<ValveclassName="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<ValveclassName="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

<DeployerclassName="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>

<ClusterListenerclassName="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListenerclassName="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

• windows默认情况下是开通组播服务的，但是linux默认情况下并没有开通，可以通过指令打开

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

发布testWeb测试工程，启动两个tomcat

/usr/local/tomcat6/bin/startup.sh
/usr/local/tomcat6_1/bin/startup.sh

如果tomcat日志中出现

 WARNING:Manager[localhost#/testWeb], requesting session state from 
org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 92, -127}:4000,
{-64,-88,92,-127},4000, alive=24591,id={-7829-87-33-65-11270-58-10275
-1041110668-3-67}, payload={}, command={}, domain={},].This operation will
 timeout ifno session state has been received within 60 seconds.

表示session共享正常

• 启动nginx测试testWeb工程当分配请求到A tomcat就shutdown.sh此tomcat，观察是否正确切换。

4 遇到问题

4.1 session共享不成功

日志提示

INFO:Manager[localhost#/testWeb]: skipping state transfer. No members active in cluster group.

• 可能是组播没有开启 见3.5

• centos5.4自带的jdk使用组播总有问题，后来重新安装jdk1.6后，可以成功共享。

5 参考文献

• http://tgyd2006.javaeye.com/blog/314492

• http://www.duduwolf.com/wiki/2007/287.html

------------------------------------------------------------------

测试脚本：

测试：

1.在tomcat的webapps创建目录 test，并建立文件 test.jsp，

内容为：

<%
System.out.println("===========================");
%>

2个tomcat都要加上，然后都重启，在地址栏输入

192.168.138.132/test/test.jsp

不停刷新，你会看到上面的图片就是两边的负载均衡已经成功。

测试2

我就参考了喝了点酒的那个哥们的测试脚本：直接复制过来

集群测试
1. tomcat7_a和tomcat7_b的cluster（cluster工程是直接复制webapps下的examples改名就可以了）工程中分别添加测试文件:testCluster.jsp

<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>
<html><head><title>Cluster Test</title></head>
<body>
<%
//HttpSession session = request.getSession(true);
 System.out.println(session.getId());
 out.println("<br> SESSION ID:" + session.getId()+"<br>");
// 如果有新的请求，则添加session属性
 String name = request.getParameter("name");
if (name != null && name.length() > 0) {
 String value = request.getParameter("value");
 session.setAttribute(name, value);
 }
 out.print("<b>Session List:</b>");
 Enumeration<String> names = session.getAttributeNames();
while (names.hasMoreElements()) {
 String sname = names.nextElement();
 String value = session.getAttribute(sname).toString();
 out.println( sname + " = " + value+"<br>");
 System.out.println( sname + " = " + value);
 }
%>
 <form action="testCluster.jsp" method="post">
 名称:<input type=text size=20 name="name">
 <br>
 值:<input type=text size=20 name="value">
 <br>
 <input type=submit value="提交">
 </form>
</body>
</html>

<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>
<html><head><title>Cluster Test</title></head>
<body>
<%
//HttpSession session = request.getSession(true);
 System.out.println(session.getId());
 out.println("<br> SESSION ID:" + session.getId()+"<br>");
// 如果有新的请求，则添加session属性
 String name = request.getParameter("name");
if (name != null && name.length() > 0) {
 String value = request.getParameter("value");
 session.setAttribute(name, value);
 }
 out.print("<b>Session List:</b>");
 Enumeration<String> names = session.getAttributeNames();
while (names.hasMoreElements()) {
 String sname = names.nextElement();
 String value = session.getAttribute(sname).toString();
 out.println( sname + " = " + value+"<br>");
 System.out.println( sname + " = " + value);
 }
%>
 <form action="testCluster.jsp" method="post">
 名称:<input type=text size=20 name="name">
 <br>
 值:<input type=text size=20 name="value">
 <br>
 <input type=submit value="提交">
 </form>
</body>
</html>

2. 启动tomcat7_a,启动完毕后，启动tomcat7_b

3. 进入http://localhost:8081/cluster/testCluster.jsp 对应tomcat7_a(8081)，登录几次，可看到

参考：http://www.weiruoyu.cn/?p=504