tomcat7+apache(mod_jk/mod_proxy)负载均衡

操作目的
前端的web服务器接收请求，若是jsp页面，就通过mod_jk/mod_proxy连接器转发至后端的tomcat服务器，并实现tomcat的负载均衡，session会话的保持

简单拓扑图
 

准备工作：
两台服务器
172.16.220.11  安装httpd tomcat  把web服务器和tomcat做在一台服务器上 主机名为tomcat1
172.16.220.12   安装tomcat     做为tomcat服务器   主机名为tomcat2
同步两台服务器的时间
hwclock -s

一、准备两台服务器

tomcat1
 编译安装httpd
准备的包apr-1.4.6.tar.bz2 apr-util-1.4.1.tar.bz2 httpd-2.4.2.tar.bz2

安装apr
#tar xf apr-1.4.6.tar.bz2
#cd apr-1.4.6
#./configure --prefix=/usr/local/apr
#make
#make install

#tar xf apr-util-1.4.1.tar.bz2
#cd apr-util-1.4.1
#./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr
#make
#make install

编译安装apache
(注意：编译的时候要启用--enable-so  指定apr的位置  启用--enable-proxy --enable-proxy-http --enable-proxy=ajp模块）
#yum -y install pcre-devel
#tar xf httpd-2.4.2.tar.bz2
#cd httpd-2.4.2
#./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-so --enable-sll --enable-zlib --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util --enable-proxy --enable-proxy-http --enable-proxy-ajp
#make
#make install

 

添加PidFile文件
 

#vim /etc/httpd/httpd.conf
添加
PidFile "/var/run/httpd.pid"

 

提供服务脚本编辑/etc/rc.d/init.d/httpd

#!/bin/bash
#
# httpd        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server.  It is used to serve \
#        HTML files and CGI.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
# config: /etc/sysconfig/httpd
# pidfile: /var/run/httpd.pid

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server
# with the thread-based "worker" MPM; BE WARNED that some modules may not
# work correctly with a thread-based MPM; notably PHP will refuse to start.

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/local/apache/bin/apachectl
httpd=${HTTPD-/usr/local/apache/bin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd.pid}
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

stop() {
 echo -n $"Stopping $prog: "
 killproc -p ${pidfile} -d 10 $httpd
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=$?
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
    fi
    echo
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  status)
        status -p ${pidfile} $httpd
 RETVAL=$?
 ;;
  restart)
 stop
 start
 ;;
  condrestart)
 if [ -f ${pidfile} ] ; then
  stop
  start
 fi
 ;;
  reload)
        reload
 ;;
  graceful|help|configtest|fullstatus)
 $apachectl $@
 RETVAL=$?
 ;;
  *)
 echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
 exit 1
esac

exit $RETVAL

#chmod +x /etc/rc.d/init.d/httpd
#chkconfig --add httpd
#service httpd start

添加apache的路径
#vim /etc/profile
PATH:$PATH:/usr/local/apache/bin
而后重新载入/etc/profile
#. /etc/profile

# httpd -D DUMP_MODULES  | grep proxy              //查看加载的模块
 proxy_http_module (shared)
 proxy_fcgi_module (shared)
 proxy_scgi_module (shared)
 proxy_ajp_module (shared)


tomcat1 tomcat2
安装tomcat
# rpm -ivh jdk-7u5-linux-i586.rpm

为JVM指定JDK路径
#vim /etc/profile
JAVA_HOME=/usr/java/jdk1.7.0_05
PATH=$PATH:$JAVA_HOME/bin:$PATH
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME
#. /etc/profile
# printenv
有如下内容
JAVA_HOME=/usr/java/jdk1.7.0_05
# java -version              //可以测试jdk是否生效
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) Client VM (build 23.1-b03, mixed mode, sharing)


#tar xf  apache-tomcat-7.0.29.tar.gz -C /usr/local
# cd /usr/local
# ln -sv apache-tomcat-7.0.29 tomcat
# /usr/local/tomcat/bin/catalina.sh start 启动

至此，两台服务器准备完毕

tomcat1和tomcat2上统一做如下配置 

tomcat1
#vim /usr/local/tomcat/conf/server.xml
修改
 <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatA">
演示效果，在TomcatA上某context中提供特定页面
#cd /usr/local/tomcat/webapps
#mkdir -pv sess/WEB-INF/{lib,classes}
#cd sess
#vim index.jsp
添加如下内容
<%@ page language="java" %>
<html>
  <head><title>TomcatA</title></head>
  <body>
    <h1><font color="red">TomcatA </font></h1>
    <table align="centre" border="1">
      <tr>
        <td>Session ID</td>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>
#/usr/local/tomcat/conf/catalina.sh stop
#/usr/local/tomcat/conf/catalina.sh stop


tomcat2
#vim /usr/local/tomcat/conf/server.xml
修改
 <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatB">

演示效果，在TomcatB上某context中，提供特定页面
#cd /usr/local/tomcat/webapps
#mkdir -pv sess/WEB-INF/{lib,classes}
#cd sess
#vim index.jsp
添加如下
<%@ page language="java" %>
<html>
  <head><title>TomcatB</title></head>
  <body>
    <h1><font color="blue">TomcatB </font></h1>
    <table align="centre" border="1">
      <tr>
        <td>Session ID</td>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>

# /usr/local/tomcat/bin/catalina.sh stop
# /usr/local/tomcat/bin/catalina.sh start

 

二、整合apache和tomcat

2.1 基于mod_jk 实现负载均衡

tomcat1:
修改/etc/httpd/extra/httpd-jk.conf为如下内容：
# Load the mod_jk
LoadModule  jk_module  modules/mod_jk.so
JkWorkersFile  /etc/httpd/extra/workers.properties
JkLogFile  logs/mod_jk.log
JkLogLevel  debug
JkMount  /*  lbcluster1
JkMount  /status  stat1
保存退出

（lbcluster1  这个名字是虚的，专门是为了实现负载均衡的)


编辑/etc/httpd/extra/workers.properties，添加如下内容：

worker.list=lbcluster1,stat1

worker.TomcatA.port=8009
worker.TomcatA.host=172.16.220.11
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1

worker.TomcatB.port=8009
worker.TomcatB.host=172.16.220.12
worker.TomcatB.type=ajp13
worker.TomcatB.lbfactor=1

worker.lbcluster1.type=lb
worker.lbcluster1.balance_workers=TomcatA,TomcatB
worker.stat1.type=status
worker.lbcluster1.sticky_session = 0

#service httpd restart

 

测试：172.16.220.11/sess 刷新访问，可以看到两台tomcat主机轮询访问，实现了负载均衡

 

 

 

问题依然存在，就是会话无法保持。接下来会解决这个问题
（把/etc/httpd/extra/workers.properties中的worker.lbcluster1.sticky_session 的值改为1，发现请求总是定向到一台tomcat服务器上）

2.2 基于mod_proxy实现负载均衡

tomcat1:

#vim /etc/httpd/httpd.conf
Include /etc/httpd/extra/httpd-vhosts.conf
#Include /etc/httpd/extra/httpd-jk.conf

#vim /etc/httpd/extra/httpd-vhosts.conf
内容改为：
ProxyRequests Off
ProxyPass / balancer://lbcluster1/ stickysession=jsessionid
ProxyPassReverse / balancer://lbcluster1/
<proxy balancer://lbcluster1>
        BalancerMember ajp://172.16.220.11:8009 loadfactor=10 route=TomcatA
        BalancerMember ajp://172.16.220.12:8009 loadfactor=10 route=TomcatB
</proxy>

<VirtualHost *:80>
        ServerAdmin [email protected]
        ServerName tomcat.magedu.com
</VirtualHost>

#service httpd restart

测试tomcat.magedu.com/sess 实现了负载均衡

 

 

 

但是会话依然无法保持（若配置worker.lbcluster1.sticky_session = 1 这里的sticksession=jsessionid 访问可以实现请求定向到一台tomcat上）

四、tomcat基于session复制的集群

（DeltaManager 和BackupManager都可以实现）
在tomcat1主机上：
#vim /usr/local/tomcat/conf/server.xml
<Engine...>行下面添加如下内容：

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                   channelSendOptions="8">
     <Manager className="org.apache.catalina.ha.session.DeltaManager"
                    expireSessionsOnShutdown="false"
                    notifyListenersOnReplication="true"/>
       <Channel className="org.apache.catalina.tribes.group.GroupChannel">
          <Membership className="org.apache.catalina.tribes.membership.McastService"
                   address="228.50.10.1"   bind="172.16.220.11"   port="45564"
                   frequency="500"  dropTime="3000"/>
             <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                                address="172.16.220.11"   port="4000"  autoBind="100"
                                selectorTimeout="5000"   maxThreads="6"/>
             <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
             </Sender>
       <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
       <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
     </Channel>
     <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                        filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"/>
     <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
     <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                     tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"  watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
      <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

#cp /usr/local/tomcat/conf/web.xml /usr/local/tomcat/webapps/sess/WEB-INF
#vim /usr/local/tomcat/webapps/sess/WEB-INF/web.xml
在
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
                      http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0"> 后面添加如下
<distributable/>

#/usr/local/tomcat/bin/catalina.sh stop
#/usr/local/tomcat/bin/catalina.sh start
#service httpd restart

tomcat2上的配置和tocmat1上的配置一样，只不过是把tomcat1的地址改为tomcat2的地址为172.16.220.12，然后复制web.xml，重启服务

访问 tomcat.magedu.com/sess 效果是负载均衡，并且会话是保持的

 

此时前端也可以是mod_jk（把apache的配置文件中的虚拟主机取消,mod_jk启用即可）

 

操作过程中遇到了不少问题，这里分享一下：

1、第五步中启动httpd时，启动成功了，但是端口并没有监听，什么原因？
# tail /usr/local/apache/logs/error_log
[Wed Aug 29 12:25:43.918479 2012] [proxy_balancer:emerg] [pid 25008:tid 3086427904] AH01177: Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded?
编译配置文件启用这个模块即可
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

2、 使用mod_jk总是实现不了复制均衡，什么问题？页面访问的一直是TomcatA，郁闷啊
原因1：worker.lbcluster1.balance_workers = TomcatA,TomcatB  中 “TomcatA和TomcatB之间多了一个空格，导致TomcatA出不来
 2：没有加worker.lbcluster1.sticky_session = 0 ，导致无法负载均衡，添加上之后就OK了

3、做mod_proxy时遇到了错误
重启httpd 时错误
# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]
# tail /usr/local/apache/logs/error_log
[proxy_balancer:emerg] [pid 4753:tid 3086935808] (22)Invalid argument: AH01186: worker slotmem_grab failed
[Wed Aug 29 20:15:18.609721 2012] [:emerg] [pid 4753:tid 3086935808] AH00020: Configuration Failed, exiting

解决：在虚拟主机的配置文件中，第一次是把
ProxyPass / balancer://lbcluster1/ stickysession=jsessionid
ProxyPassReverse / balancer://lbcluster1/ 这两项内容定义在了虚拟主机内部，出了错误

定义在全局位置就可以了

4、在实现session复制时，定义<distributable/>时，放在其他地方就不行，即便是放在
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
                      http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0"> 的前面就不行，一定要放在后面才可以，可以是下面的任何一定地方，这是文件读取时加载的顺序