Apache+Tomcat构建Tomcat负载均衡集群

一、环境介绍

二、安装后端服务器

三、安装前端Apache服务

四、配置Apache使用mod_jk模块实现代理及负载均衡

五、配置Apache基于mod_proxy模块实现代理及负载均衡

六、论坛安装

七、安装并配置Keepalived实现前端高可用

一、环境介绍

系统版本：CentOS 6.4_x86_64

Mysql版本：mysql-5.1.66-2.el6_3.x86_64

Apache版本：httpd-2.2.25 

Keepalived版本：keepalived-1.2.7 

Tomcat版本：apache-tomcat-7.0.33 

Jdk版本： jdk-7u40-linux-x64 

Tomcat-commectors版本：tomcat-connectors-1.2.37 

论坛版本：JspRun!_6.0.0_GBK 

环境介绍：

当用户通过 "www.allen.com" 域名访问时，首先是由前端两台Apache代理服务器响应并转发到后端Tomcat服务器上，而在Apache上做的是反向代理负载均衡到后端Tomcat服务器上，前端服务器使用Keepalived做的高可用集群，【Apache1默认为Master，Apache2为Backup】虚拟IP地址为"172.16.14.10"模拟为公网IP；当然这里数据库只有一台主机会成为点故障，这里不要介绍Mysql的高可用，如果有兴趣可以看前面写的博客

提示：这里使用Keepalived做高可用时，前面所有的服务都已完成后，最后实现的对前端服务器做高可用

二、安装后端Tomcat与Mysql服务器

1、在Tomcat1与Tomcat2服务器上安装Tomcat，安装方法相同，这里只介绍一次

安装JDK

# rpm -ivh jdk-7u40-linux-x64.rpm

# vim /etc/profile.d/java.sh

export JAVA_HOME=/usr/java/latest

export PATH=$JAVA_HOME/bin:$PATH

# . /etc/profile.d/java.sh

# java -version

java version "1.7.0_40"

Java(TM) SE Runtime Environment (build 1.7.0_40-b43)

Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

------------------------------------------------------------------------

安装Tomcat

# tar xf apache-tomcat-7.0.33.tar.gz -C /usr/local/

# cd /usr/local/

# ln -s apache-tomcat-7.0.33 tomcat

# vim /etc/profile.d/tomcat.sh

export CATALINA_HOME=/usr/local/tomcat

export PATH=$CATALINA_HOME/bin:$PATH

# . /etc/profile.d/tomcat.sh

# catalina.sh version

Using CATALINA_BASE:   /usr/local/tomcat

Using CATALINA_HOME:   /usr/local/tomcat

Using CATALINA_TMPDIR: /usr/local/tomcat/temp

Using JRE_HOME:        /usr/java/latest

Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

Server version: Apache Tomcat/7.0.33

Server built:   Nov 18 2012 04:15:21

Server number:  7.0.33.0

OS Name:        Linux

OS Version:     2.6.32-358.el6.x86_64

Architecture:   amd64

JVM Version:    1.7.0_40-b43

JVM Vendor:     Oracle Corporation

2、启动Tomcat服务并访问测试，默认访问端口为"8080";如下

# catalina.sh start

Using CATALINA_BASE:   /usr/local/tomcat

Using CATALINA_HOME:   /usr/local/tomcat

Using CATALINA_TMPDIR: /usr/local/tomcat/temp

Using JRE_HOME:        /usr/java/latest

Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

# jps

2283 Jps

2272 Bootstrap

# ss -tanlp | grep 8080

LISTEN     0      100       :::8080        :::*      users:(("java",2272,40))

3、开启Tomcat管理及状态页面

# vim /usr/local/tomcat/conf/tomcat-users.xml

  <role rolename="manager-gui"/>

  <role rolename="admin-gui"/>

  <user username="allen" password="admin" roles="manager-gui,admin-gui"/>

# catalina.sh stop

# catalina.sh start

4、访问Tomcat后台管理

5、安装Mysql服务器

######安装Mysql并创建数据库，授权用户访问新创建的数据库有所有权，方便后面使用

[root@Mysql ~]# yum -y install mysql-server

[root@Mysql ~]# service mysqld start

[root@Mysql ~]# mysql

mysql> create database bbs;

mysql> grant all on bbs.* to 'bbsuser'@'172.16.%.%' identified by 'bbspass';

mysql> flush privileges;

----------------------------------------------------------------------

远程连接数据库测试授权用户是否能连接成功

# mysql -ubbsuser -pbbspass -h 172.16.14.5 -e 'show databases';

+--------------------+

| Database           |

+--------------------+

| information_schema |

| bbs                |

| test               |

+--------------------+

三、安装前端Apache服务

1、在Apache1与Apache2服务器上安装Httpd软件，安装方法相同，这里只介绍一次

# tar xf httpd-2.2.25.tar.bz2

# cd httpd-2.2.25

# yum -y install gcc gcc-c++ openssl-devel pcre-devel

# ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-modules=most --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --enable-mpms-shared=all --with-mpm=prefork --enable-proxy --enable-proxy-http --enable-proxy-ajp --enable-proxy-balancer --enable-cache --enable-cache-disk --enable-mem-cache --enable-file-cache --enable-proxy-connect

# make && make install

# cp build/rpm/httpd.init /etc/init.d/httpd

------修改如下内容

# vim /etc/init.d/httpd

httpd=${HTTPD-/usr/local/apache/bin/httpd}

pidfile=${PIDFILE-/usr/local/apache/logs/${prog}.pid}

lockfile=${LOCKFILE-/var/lock/subsys/${prog}}

RETVAL=0

# check for 1.3 configuration

check13 () {

        CONFFILE=/etc/httpd/httpd.conf

----------------------------------------------------------------------

修改环境变量并把Httpd加入到系统服务

# echo "PATH=/usr/local/apache/bin:$PATH" >> /etc/profile.d/http.sh

# . /etc/profile.d/http.sh

# ln -s /usr/local/apache/include/ /usr/include/httpd

# chkconfig --add httpd

----------------------------------------------------------------------

创建测试页

# echo "<h1>APACHE</h1>" > /usr/local/apache/htdocs/index.html

2、在Apache1与Apache2服务器上安装"tomcat-connectors"

# tar xf tomcat-connectors-1.2.37-src.tar.gz

# cd tomcat-connectors-1.2.37-src/native/

# ./configure --with-apxs=/usr/local/apache/bin/apxs

# make && make install

------------------------------------------------------------------------

安装完成后会生成如下文件

# ls /usr/local/apache/modules/

  mod_jk.so

------------------------------------------------------------------------

启动Httpd服务

#service httpd start

# ss -tanl |grep 80

LISTEN     0      128              :::80               :::*

3、访问测试Apache是否正常工作

四、配置Apache使用mod_jk模块实现代理及负载均衡

1、修改Apache主配置文件，包含一个文件并创建该文件；实现代理功能

将下面指令添加到文件末尾即可

# vim /etc/httpd/httpd.conf

Include /etc/httpd/extra/mod_jk.conf

------------------------------------------------------------------------

创建文件添加如下内容

# vim /etc/httpd/extra/mod_jk.conf

LoadModule  jk_module  modules/mod_jk.so

JkWorkersFile  /etc/httpd/extra/workers.properties

JkLogFile  logs/mod_jk.log

JkLogLevel  debug

JkMount  /*  TomcatA

JkMount  /status/  stat1

------------------------------------------------------------------------

# vim /etc/httpd/extra/workers.properties

worker.list=TomcatA,stat1

worker.TomcatA.port=8009

worker.TomcatA.host=172.16.14.3

worker.TomcatA.type=ajp13

worker.TomcatA.lbfactor=1

worker.stat1.type = status

----------------------------------------------------------------------

# service httpd restart

2、修改后端Tomcat1服务器，添加一个虚拟主机并提供测试页面

[root@Tomcat1 ~]# cd /usr/local/tomcat/conf/

[root@Tomcat1 conf]# cp server.xml server.xml.bak

[root@Tomcat1 conf]# vim server.xml

######修改如下内容

<Engine name="Catalina" defaultHost="www.allen.com" jvmRoute="TomcatA">

######在"Engine"中添加如下内容

<Host name="www.allen.com"  appBase="/web/allen"

            unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

               prefix="allen_access_log." suffix=".txt"

               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

        <Context path="" docBase="/web/allen" />

      </Host>

----------------------------------------------------------------------

######创建网站存放目录并创建测试页

[root@Tomcat1 ~]# mkdir -p /web/allen

[root@Tomcat1 ~]# vim /web/allen/index.jsp

<%@ page language="java" %>

<html>

  <head><title>TomcatA</title></head>

  <body>

    <h1><font color="red">TomcatA </font></h1>

    <table align="centre" border="1">

      <tr>

        <td>Session ID</td>

    <% session.setAttribute("abc","abc"); %>

        <td><%= session.getId() %></td>

      </tr>

      <tr>

        <td>Created on</td>

        <td><%= session.getCreationTime() %></td>

     </tr>

    </table>

  </body>

</html>

----------------------------------------------------------------------

######重启Tomcat服务并查看是否启动成功

[root@Tomcat1 ~]# catalina.sh stop

[root@Tomcat1 ~]# catalina.sh configtest

[root@Tomcat1 ~]# catalina.sh start

[root@Tomcat1 ~]# ss -tanlp | grep java

LISTEN     0      100                      :::8080                    :::*      users:(("java",6323,40))

LISTEN     0      1          ::ffff:127.0.0.1:8005                    :::*      users:(("java",6323,52))

LISTEN     0      100                      :::8009                    :::*      users:(("java",6323,41))

3、访问Apache1主机，验证是否代理成功

4、修改Apache配置文件(mod_jk.conf、workers.properties)；实现负载均衡

# vim /etc/httpd/extra/mod_jk.conf

LoadModule  jk_module  modules/mod_jk.so

JkWorkersFile  /etc/httpd/extra/workers.properties

JkLogFile  logs/mod_jk.log

JkLogLevel  debug

JkMount  /*  lbcluster

JkMount  /status/  stat1

------------------------------------------------------------------------

# vim /etc/httpd/extra/workers.properties

worker.list=lbcluster,stat1

worker.TomcatA.port=8009

worker.TomcatA.host=172.16.14.3

worker.TomcatA.type=ajp13

worker.TomcatA.lbfactor=1

worker.TomcatB.port = 8009

worker.TomcatB.host=172.16.14.4

worker.TomcatB.type = ajp13

worker.TomcatB.lbfactor = 1

worker.lbcluster.type = lb

worker.lbcluster.sticky_session = 0  #取值为{1|0}1将用户session与后端服务器绑定，0为不绑定，如果支持session复制或session共享可以设置为0

worker.lbcluster.balance_workers = TomcatA, TomcatB

worker.stat1.type = status

----------------------------------------------------------------------

# service httpd reload

5、修改后端Tomcat2服务器，添加一个虚拟主机并提供测试页面

[root@Tomcat2 ~]# cd /usr/local/tomcat/conf/

[root@Tomcat2 conf]# cp server.xml server.xml.bak

[root@Tomcat2 conf]# vim server.xml

######修改如下内容

<Engine name="Catalina" defaultHost="www.allen.com" jvmRoute="TomcatB">

######在"Engine"中添加如下内容

<Host name="www.allen.com"  appBase="/web/allen"

            unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

               prefix="allen_access_log." suffix=".txt"

               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

        <Context path="" docBase="/web/allen" />

      </Host>

----------------------------------------------------------------------

######创建网站存放目录并创建测试页

[root@Tomcat2 ~]# mkdir -p /web/allen

[root@Tomcat2 ~]# vim /web/allen/index.jsp

<%@ page language="java" %>

<html>

  <head><title>TomcatB</title></head>

  <body>

    <h1><font color="blue">TomcatB </font></h1>

    <table align="centre" border="1">

      <tr>

        <td>Session ID</td>

    <% session.setAttribute("abc","abc"); %>

        <td><%= session.getId() %></td>

      </tr>

      <tr>

        <td>Created on</td>

        <td><%= session.getCreationTime() %></td>

     </tr>

    </table>

  </body>

</html>

----------------------------------------------------------------------

######重启Tomcat服务并查看是否启动成功

[root@Tomcat2 ~]# catalina.sh stop

[root@Tomcat2 ~]# catalina.sh configtest

[root@Tomcat2 ~]# catalina.sh start

[root@Tomcat2 ~]# ss -tanlp | grep java

LISTEN     0      100                      :::8080                    :::*      users:(("java",6323,40))

LISTEN     0      1          ::ffff:127.0.0.1:8005                    :::*      users:(("java",6323,52))

LISTEN     0      100                      :::8009                    :::*      users:(("java",6323,41))

6、访问Apache服务器，测试是否实现负载均衡

由上可见，基于Apache的mod_jk模块已成功实现负载均衡功能

五、配置Apache基于mod_proxy模块实现代理及负载均衡

1、修改Apache的主配置文件，包含一个文件并创建该文件；实现代理功能

# vim /etc/httpd/httpd.conf

#Include /etc/httpd/extra/mod_jk.conf    #注释此行

Include /etc/httpd/extra/mod_proxy.conf

------------------------------------------------------------------------

# vim /etc/httpd/extra/mod_proxy.conf

ProxyVia On

ProxyRequests Off

ProxyPreserveHost Off

<Proxy *>

  Order allow,deny

  Allow from all

</Proxy>

  ProxyPass  /  ajp://172.16.14.3:8009/

  ProxyPassReverse  /  ajp://172.16.14.3:8009/

<Location  / >

  Order allow,deny

  Allow from all

</Location>

----------------------------------------------------------------------

# service httpd reload

======================================================================

注释：

  ProxyPass  /  ajp://172.16.14.3:8009/    #使用的是ajp协议

  ProxyPassReverse  /  ajp://172.16.14.3:8009/

可以更改为使用http协议；如下

  ProxyPass  /  http://172.16.14.3:8080/

  ProxyPassReverse  /  http://172.16.14.3:8080/

######这里只演示使用ajp协议,如果是Apache与Tomcat结合建议使用ajp协议

2、访问Apache1服务器，测试是否代理成功

3、修改Apache配置文件(mod_proxy.conf)；实现负载均衡

# vim /etc/httpd/extra/mod_proxy.conf

ProxyVia Off

ProxyRequests Off

ProxyPreserveHost Off

<Proxy balancer://allen>

  BalancerMember ajp://172.16.14.3:8009 loadfactor=1

  BalancerMember ajp://172.16.14.4:8009 loadfactor=1

  ProxySet lbmethod=bytraffic

</Proxy>

<Location /allen>    #设置状态及管理页面

  SetHandler balancer-manager

  Proxypass !

  Order allow,deny

  Allow from all

</Location>

<Proxy *>

  Order allow,deny

  Allow from all

</Proxy>

  ProxyPass  /  balancer://allen/    stickysession=JSESSIONID

  ProxyPassReverse  /  balancer://allen/

<Location  / >

  Order allow,deny

  Allow from all

</Location>

----------------------------------------------------------------------

注释：

  BalancerMember ajp://172.16.14.3:8009 loadfactor=1  #使用ajp协议

  BalancerMember ajp://172.16.14.4:8009 loadfactor=1

可更改为使用http协议

  BalancerMember http://172.16.14.3:8080 loadfactor=1

  BalancerMember http://172.16.14.4:8080 loadfactor=1

######这里只介绍使用ajp协议，如果有兴趣可以更改一下；loadfactor：权重

----------------------------------------------------------------------

ProxySet lbmethod=bytraffic    #设置调度算法

byrequests：即基于权重将统计请求个数进行调度(默认)

bytraffic：则执行基于权重的流量计数调度

bybusyness：通过考量每个后端服务器的当前负载进行调度

----------------------------------------------------------------------

# service httpd reload

4、访问Apache服务器，测试基于mod_proxy模块是否实现负载均衡

5、访问状态页面

六、论坛安装

1、在Tomcat1服务器上安装

# unzip JspRun\!_6.0.0_GBK.zip

# mv /web/allen/index.jsp /web/allen/test.jsp

# cp -r upload/* /web/allen/

------------------------------------------------------------------------

修改论坛数据库连接文件

# vim /web/allen/config.properties

dbhost = 172.16.14.5

dbport=3306

dbuser = bbsuser

dbpw = bbspass

dbname = bbs

------------------------------------------------------------------------

# catalina.sh stop

# catalina.sh start

2、安装论坛程序

3、点击下一步-->我同意-->下一步，进入数据库配置页面

4、点下一步-->(填写管理员密码)下一步-->进入创建数据表阶段-->然后点-->进入首页

5、将论坛程序拷贝到Tomcat2服务器上一份并访问测试

# mv /web/allen/index.jsp /web/allen/test.jsp    #把原来的测试页重命名

------------------------------------------------------------------------

# scp -r 172.16.14.3:/web/allen/* /web/allen/

# catalina.sh stop

# catalina.sh start

6、访问Apache服务器，测试是否能正常访问论坛程序，这里就不在做访问测试

七、安装并配置Keepalived

1、将Apache1服务器的Httpd配置文件复制到Apache2服务器

[root@Apache1 ~]# cd /etc/httpd/

[root@Apache1 httpd]# scp httpd.conf 172.16.14.2:/etc/httpd/

[root@Apache1 httpd]# cd extra/

[root@Apache1 extra]# scp mod_jk.conf mod_proxy.conf workers.properties 172.16.14.2:/etc/httpd/extra/

2、在Apache1与Apache2服务器上安装Keepalived；这里使用rpm包安装，光盘映像中有

[root@Apache1 ~]# yum -y install keepalived

--------------------------------------------

[root@Apache2 ~]# yum -y install keepalived

3、配置Apache1服务器上的Keepalived

[root@Apache1 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

   root@localhost

   }

   notification_email_from [email protected]

   smtp_server 172.16.0.0

   smtp_connect_timeout 30

   router_id LVS_ALLEN

}

vrrp_script chk_httpd {

    script "killall -0 httpd"

    interval 1

    weight -2

}

vrrp_instance httpd_1 {

    state MASTER

    interface eth0

    virtual_router_id 58

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1058

    }

    virtual_ipaddress {

    172.16.14.10

    }

    track_script {

    chk_httpd

    }

}

[root@Apache1 ~]# service keepalived start

[root@Apache1 ~]# chkconfig keepalived on

[root@Apache1 ~]# chkconfig --list keepalived

keepalived      0:off   1:off   2:on    3:on    4:on    5:on    6:off

[root@Apache1 ~]# ip addr show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:2c:1a:24 brd ff:ff:ff:ff:ff:ff

    inet 172.16.14.1/16 brd 172.16.255.255 scope global eth0

    inet 172.16.14.10/32 scope global eth0

    inet6 fe80::20c:29ff:fe2c:1a24/64 scope link

       valid_lft forever preferred_lft forever

4、配置Apache2服务器上的Keepalived

[root@Apache2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

   root@localhost

   }

   notification_email_from [email protected]

   smtp_server 172.16.0.0

   smtp_connect_timeout 30

   router_id LVS_ALLEN

}

vrrp_script chk_httpd {

    script "killall -0 httpd"

    interval 1

    weight -2

}

vrrp_instance httpd_1 {

    state BACKUP

    interface eth0

    virtual_router_id 58

    priority 99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1058

    }

    virtual_ipaddress {

    172.16.14.10

    }

    track_script {

    chk_httpd

    }

}

[root@Apache2 ~]# service keepalived start

[root@Apache2 ~]# chkconfig --list keepalived

keepalived      0:off   1:off   2:on    3:on    4:on    5:on    6:off

5、使用"www.allen.com" 域名访问，验证是否能访问到论坛；由于没有DNS服务器解析，修改了"hosts"文件；如：

######添加如下内容解析

C:\Windows\System32\drivers\etc\hosts

172.16.14.10    www.allen.com

6、模拟前端一台服务器出现故障；查看虚拟IP是否转移

######停止Apache1服务器上的Httpd服务来模拟故障；并查看IP

[root@Apache1 ~]# service httpd stop

[root@Apache1 ~]# ip addr show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:2c:1a:24 brd ff:ff:ff:ff:ff:ff

    inet 172.16.14.1/16 brd 172.16.255.255 scope global eth0

    inet6 fe80::20c:29ff:fe2c:1a24/64 scope link

       valid_lft forever preferred_lft forever

========================================================================

######查看Apache2服务器上的IP

[root@Apache2 ~]# ip addr show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:ec:f6:3f brd ff:ff:ff:ff:ff:ff

    inet 172.16.14.2/16 brd 172.16.255.255 scope global eth0

    inet 172.16.14.10/32 scope global eth0

    inet6 fe80::20c:29ff:feec:f63f/64 scope link

       valid_lft forever preferred_lft forever

7、从上一步可以看出，虚拟IP地址已成功转移，说明还能正常提供服务，可以再次访问测试；如果服务器修复好重新上线，虚拟IP还会转移回去；这里就不再测试

到此，基于Apache做反向代理实现了Tomcat的负载均衡；而Keepalived实现了前端服务器的高可用；关于session保存的问题，方案有很多，比如可以保存到"memcached"、"redis"等，使用哪种方案具体还要根据自己的需求而定...