nginx+tomcat+memcached实现负载均衡和session共享

安装包：

apache-tomcat-7.0.37.tar.gz  
jdk-7u79-linux-x64.tar.gz  

架包：（共9个）

asm-3.2.jar
kryo-1.04.jar
kryo-serializers-0.10.jar
memcached-session-manager-1.6.3.jar
memcached-session-manager-tc7-1.6.3.jar
minlog-1.2.jar
msm-kryo-serializer-1.6.3.jar
reflectasm-1.01.jar
spymemcached-2.7.3.jar

实验环境：

两个节点   
server5（n1）  
server4（n2）

搭建实验环境：

在server5中：
[root@server5 ~]# nginx   #打开nginx
[root@server5 ~]# /etc/init.d/php-fpm start  
Starting php-fpm  done
## -C 表示解压路径
[root@server5 ~]# tar zxf jdk-7u79-linux-x64.tar.gz -C /usr/local/
[root@server5 ~]# cd /usr/local/
[root@server5 local]# ln -s jdk1.7.0_79/  java
##修改全局变量
[root@server5 local]# vim /etc/profile
############
添加：
export JAVA_HOME=/usr/local/java
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin

[root@server5 local]# source /etc/profile
##查看$PATH的路径
[root@server5 local]# echo $PATH 
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/java/bin:/root/bin:/usr/local/lnmp/mysql/bin:/usr/local/lnmp/nginx/sbin:/usr/local/lnmp/php/bin:/usr/local/java/bin
[root@server5 local]# echo $CLASSPATH
.:/usr/local/java/lib:/usr/local/java/jre/lib
[root@server5 local]# echo $JAVA_HOME
/usr/local/java
[root@server5 local]# which java
/usr/local/java/bin/java
[root@server5 local]# cd
##测试jdk环境
[root@server5 ~]# vim test.java

[root@server5 ~]# javac test.java   #检测语法是否正确
##查看文件类型是一个java库文件
[root@server5 ~]# file test.class    
test.class: compiled Java class data, version 51.0
[root@server5 ~]# java test        #执行脚本
Hello World!
[root@server5 ~]# tar zxf apache-tomcat-7.0.37.tar.gz -C /usr/local/

[root@server5 ~]# cd /usr/local/
[root@server5 local]# ln -s apache-tomcat-7.0.37/  tomcat
[root@server5 local]# cd tomcat/
##开启tomcat
[root@server5 tomcat]# bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@server5 tomcat]# cd logs/
[root@server5 logs]# cat catalina.out    #看有没有成功启动的日志

##查看tomcat默认端口8080是否开启
[root@server5 logs]# netstat -antlp|grep :8080
tcp        0      0 :::8080                     :::*                        LISTEN      1087/java     

测试：
输入172.25.52.5:8080

输入：172.25.52.5/index.php

为了使界面更好看，可以进行以下操作

[root@server5 ROOT]# cd /usr/local/lnmp/nginx/
[root@server5 nginx]# vim conf/nginx.conf

再次输入：172.25.52.5/index.php

[root@server5 tomcat]# cd webapps/ROOT/
[root@server5 ROOT]# vim test.jsp
###################
写入：
The time is: <%=new java.util.Date() %>

测试：
输入：172.25.52.5：8080/test.jsp

负载均衡:

在server4上做相同的操作：
[root@server4 ~]# tar zxf jdk-7u79-linux-x64.tar.gz  -C /usr/local/
[root@server4 ~]# tar zxf apache-tomcat-7.0.90.tar.gz  -C /usr/local/
[root@server4 ~]# vim /etc/profile
#################
写入：
export JAVA_HOME=/usr/local/java
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin

[root@server4 ~]# source /etc/profile
##制作软链接
[root@server4 ~]# cd /usr/local/
[root@server4 local]# ln -s jdk1.7.0_79/ java
[root@server4 local]# ln -s apache-tomcat-7.0.90/ tomcat
[root@server4 local]# cd tomcat/
##开启tomcat
[root@server4 tomcat]# bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@server4 tomcat]# cd webapps/ROOT/
[root@server4 ROOT]# vim test.jsp
####################
写入：
Server4  The time is: <%=new java.util.Date() %>

[root@server5 tomcat]# cd webapps/ROOT/
[root@server5 ROOT]# vim test.jsp
####################
修改：
Server5  The time is: <%=new java.util.Date() %>

#编写配置文件
[root@server5 ROOT]# cd /usr/local/lnmp/nginx/
[root@server5 nginx]# vim conf/nginx.conf

[root@server5 nginx]# nginx -s reload

测试：

Session共享

##重新编写server4与server5测试文件：
##测试文件是相同的
[root@server5 ROOT]# pwd
/usr/local/tomcat/webapps/ROOT
[root@server5 ROOT]# vim test.jsp
####################
<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>
<html><head><title>Cluster App Testtitle>head>
<body>
Server Info:
<%
out.println(request.getLocalAddr() + " : " + request.getLocalPort()+"
");%>
<%
out.println("
 ID " + session.getId()+"
");
String dataName = request.getParameter("dataName");
if (dataName != null && dataName.length() > 0) {
String dataValue = request.getParameter("dataValue");
session.setAttribute(dataName, dataValue);
}
out.print("Session list");
Enumeration e = session.getAttributeNames();
while (e.hasMoreElements()) {
String name = (String)e.nextElement();
String value = session.getAttribute(name).toString();
out.println( name + " = " + value+"
");
System.out.println( name + " = " + value);
}
%>
<form action="test.jsp" method="POST">
name:<input type=text size=20 name="dataName">
<br>
key:<input type=text size=20 name="dataValue">
<br>
<input type=submit>
form>
body>
html>

[root@server4 ROOT]# vim test.jsp
####################
<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>
<html><head><title>Cluster App Testtitle>head>
<body>
Server Info:
<%
out.println(request.getLocalAddr() + " : " + request.getLocalPort()+"
");%>
<%
out.println("
 ID " + session.getId()+"
");
String dataName = request.getParameter("dataName");
if (dataName != null && dataName.length() > 0) {
String dataValue = request.getParameter("dataValue");
session.setAttribute(dataName, dataValue);
}
out.print("Session list");
Enumeration e = session.getAttributeNames();
while (e.hasMoreElements()) {
String name = (String)e.nextElement();
String value = session.getAttribute(name).toString();
out.println( name + " = " + value+"
");
System.out.println( name + " = " + value);
}
%>
<form action="test.jsp" method="POST">
name:<input type=text size=20 name="dataName">
<br>
key:<input type=text size=20 name="dataValue">
<br>
<input type=submit>
form>
body>
html>

测试：

查看到server info一直在交替变更

为了不让一直交替变换，执行以下操作

[root@server5 nginx]# vim conf/nginx.conf
####################
17 http {
 18         upstream tomcat{
 ##也可以添加sticky模块（需要重新编译）
 19          ip_hash;      #哈希
 20         server 172.25.52.4:8080;
 21         server 172.25.52.5:8080;
 22         }
[root@server5 nginx]# nginx -s reload

测试：
虽然server infio没有再变更，但是一旦某个节点出现故障， 那么保存在该节点上的数据便会丢失

解决方案：
数据交叉存储

在server5与server4中做以下相同的操作：
##下载jar包并将其放到/usr/local/tomcat/lib 目录中
[root@server5 lib]# ls
asm-3.2.jar
kryo-1.04.jar
kryo-serializers-0.10.jar
memcached-session-manager-1.6.3.jar
memcached-session-manager-tc7-1.6.3.jar
memcached-session-manager-tc6-1.6.3.jar
minlog-1.2.jar
msm-kryo-serializer-1.6.3.jar
reflectasm-1.01.jar
spymemcached-2.7.3.jar
[root@server5 lib]# rm -fr memcached-session-manager-tc6-1.6.3.jar 
[root@server5 lib]# cd /usr/local/tomcat/conf
##编写配置文件
[root@server5 conf]# vim context.xml
################### 
添加：
"de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:172.25.52.5:11211,n2:172.25.52.4.1:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>
[root@server5 tomcat]# cd bin/
[root@server5 bin]# ./startup.sh
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@server5 bin]# cd ..
[root@server5 tomcat]# cd logs/
[root@server5 logs]#cat catalina.out  

[root@server5 conf]# yum install memcached  -y
[root@server5 tomcat]# /etc/init.d/memcached start
Starting memcached:                                        [  OK  ]

##下载jar包并将其放到/usr/local/tomcat/lib 目录中
[root@server4 lib]# ls
asm-3.2.jar
kryo-1.04.jar
kryo-serializers-0.10.jar
memcached-session-manager-1.6.3.jar
memcached-session-manager-tc7-1.6.3.jar
memcached-session-manager-tc6-1.6.3.jar
minlog-1.2.jar
msm-kryo-serializer-1.6.3.jar
reflectasm-1.01.jar
spymemcached-2.7.3.jar
[root@server4 lib]# rm -fr memcached-session-manager-tc6-1.6.3.jar 
[root@server4 lib]# cd /usr/local/tomcat/conf
##编写配置文件
[root@server4 conf]# vim context.xml 
添加：
"de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:172.25.52.5:11211,n2:172.25.52.4：11211"
failoverNodes="n2"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>
[root@server4 tomcat]# cd bin/
[root@server4 bin]# ./startup.sh
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@server4 bin]# cd ..
[root@server4 tomcat]# cd logs/
[root@server4 logs]#cat catalina.out   

[root@server4 conf]# yum install memcached  -y
[root@server4 tomcat]# /etc/init.d/memcached start
Starting memcached:                                        [  OK  ]

测试：
Server Info 在server5上 即节点server5工作

Server5的数据存储在server4上：

模拟破坏节点server5：

[root@server5 tomcat]# bin/shutdown.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

此时Server Info会切换到server4上 即节点server5出现故障 节点server4顶替工作

由于是交叉存储,即使节点server5被破坏,但原始数据仍然保存完好在server4上

排错：

1.没有sticky模块 怎么办？

[root@server5 nginx-1.10.1]# nginx
nginx: [emerg] unknown directive "sticky" in /usr/local/lnmp/nginx/conf/nginx.conf:20
[root@server5 conf]# killall -9 nginx 
[root@server5 conf]#cd
[root@server5 ~]# ls
nginx-sticky-module-ng.tar.gz 
[root@server5 ~]#tar zxf nginx-sticky-module-ng.tar.gz 
[root@server5 ~]# cd nginx-1.10.1
[root@server5 nginx-1.10.1]# make clean
rm -rf Makefile objs
[root@server5 nginx-1.10.1]# ./configure --help
[root@server5 nginx-1.10.1]# ./configure --prefix=/usr/local/lnmp/nginx/ --with-http_ssl_module --with-http_stub_status_module --with-threads --with-file-aio --add-module=/root/nginx-sticky-module-ng
[root@server5 nginx-1.10.1]# make && make install
[root@server5 nginx-1.10.1]# nginx

1.如果在网页上输入172.25.52.5：8080
                172.25.52.4：8080
刷不出来 该怎么办？


（1）.查看是否有8080端口 
Netstat -antlp
如果没有 就执行 cd /usr/local/tomcat
                bin/startup.sh 

（2）.查看进程 将多余的进程杀死
[root@server4 conf]# ps
  PID TTY          TIME CMD
 1089 pts/0    00:00:00 bash
 1857 pts/0    00:00:02 java
 1899 pts/0    00:00:00 ps
[root@server4 conf]# kill -9 1857
[root@server4 conf]# cd ..
[root@server4 tomcat]# bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar


2.如果 输入http://172.25.52.5/test.jsp 刷不出来 该怎么办？
(1).在两个节点server4和server5上：
[root@server4 tomcat]# bin/shutdown.sh   #关闭
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
[root@server4 tomcat]# bin/startup.sh     #打开
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/java
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

(2)查看两个节点上/usr/local/tomcat/lib中的架包：（总共9个）
asm-3.2.jar
kryo-1.04.jar
kryo-serializers-0.10.jar
memcached-session-manager-1.6.3.jar
memcached-session-manager-tc6-1.6.3.jar  ##删除
memcached-session-manager-tc7-1.6.3.jar
minlog-1.2.jar
msm-kryo-serializer-1.6.3.jar
reflectasm-1.01.jar
spymemcached-2.7.3.jar

（3）查看两个节点上的文件是否编写正确：
Vim /usr/local/tomcat/webapps/ROOT/test.jsp
#####################
<%@ page contentType="text/html; charset=GBK" %>
<%@ page import="java.util.*" %>


Server Info:
<%
out.println(request.getLocalAddr() + " : " + request.getLocalPort()+"
");%>
<%
out.println("
 ID " + session.getId()+"
");
String dataName = request.getParameter("dataName");
if (dataName != null && dataName.length() > 0) {
String dataValue = request.getParameter("dataValue");
session.setAttribute(dataName, dataValue);
}
out.print("Session list");
Enumeration e = session.getAttributeNames();
while (e.hasMoreElements()) {
String name = (String)e.nextElement();
String value = session.getAttribute(name).toString();
out.println( name + " = " + value+"
");
System.out.println( name + " = " + value);
}
%>
"test.jsp" method="POST">
name:20 name="dataName">


key:20 name="dataValue">







vim /usr/local/tomcat/conf/context.xml    
##################
   "org.apache.catalina.valves.CometConnectionManagerValve" />
-->

"de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:172.25.52.5:11211,n2:172.25.52.4:11211"
failoverNodes="n2"  ##在server5上写n1
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>