搭建nginx+tomcat+keepalived高可用集群

拓扑图

1实现目标:

使用tomcat+nginx+keepalived搭建高可用集群

2环境准备:

2.1软件下载地址及版本

Keepalived-2.2.7.tar.gz

下载地址:https://www.keepalived.org/download.html

Apache-Tomcat8.5.70.tar.gz

下载地址:https://tomcat.apache.org/download-80.cgi

Nginx-1.22.0.tar.gz

下载地址:http://nginx.org/download

jdk-8u291-linux-x64.tar.gz

下载地址:

https://www.oracle.com/java/technologies/javse/javase8u211-later-archive-downloads.html

2.2编译环境

yum groupinstall "Development Tools"

yum-y install openssl-devel libnl libnl-devel

2.3 配置要求

四台centos7.9虚拟机(两台安装nginx+keepalived，两台安装tomcat)

2.4网段规划

nginx+keepalived(主) 192.168.205.150

nginx+keepalived(备) 192.168.205.151

Tomcat(1) 192.168.205.152

Tomcat(2) 192.168.205.153

2.5 模块安装

Nginxweb ui模块nginx-module-vts

下载地址:https://github.com/vozlt/nginx-module-vts/releases/tag/v0.2.1

负载均衡模块nginx-goodies-nginx-sticky

下载地址:

https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/downloads/

3.操作步骤:

注意:3.1-3.3.8步骤在152和153同步执行

3.1配置jdk

3.1.1创建jdk目录

mkdir -p /usr/java/jvm

3.1.2解压jdk压缩包到/usr/java/jvm

3.1.3配置并更新环境变量

vim /etc/profile

exportJAVA_HOME=/usr/java/jvm/jdk1.8.0_291

exportJRE_HOME=${JAVA_HOME}/jre

exportCLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

exportPATH=${JAVA_HOME}/bin:$PATH

更新:source/etc/profile

3.1.4 查看是否安装成功

javac -version

3.2配置tomcat

3.2.1创建tomcat目录

mkdir -p /usr/java/tomcat

3.2.2解压tomcat压缩包到/usr/java/tomcat

tar -zxf apache-tomcat-8.5.70.tar.gz -C/usr/java/tomcat

注意:以下操作都在/usr/ava/tomcat/apache-tomcat-8.5.70里执行

3.2.3创建模板目录

mkdir web-template

3.2.4将conf、logs、temp、webapps、work移动到web-template目录下

3.2.5创建实例(两个实例)

cp-rp web-template web-8080(配置默认)

cp-rp web-template web-8081(配置里面修改端口号)

（8081配置

原本的8005改为9005

原本的8080改为8081

3.2.6 创建访问界面

cd /usr/java/tomcat/apache-tomcat-8.5.70/web-8080/webapps

mkdir testweb

cd testweb

vim /indes.jsp

<%@page language="java" contentType="text/html; charset=utf-8"

pageEncoding="utf-8"%>

" target="_blank">http://www.w3.org/TR/html4/loose.dtd">

192.168.205.152

注意:

3.2.7创建实例脚本并执行(有几个实例就写几个脚本)

vim/etc/tomcat_8080.sh

RETVAL=$?

#tomcat instance directory

export CATALINA_HOME="/usr/java/tomcat/apache-tomcat-8.5.70"

#tomcat installation directory, change to your own

export CATALINA_BASE="$CATALINA_HOME/web-8080"

#Optional

exportJVM_OPTIONS="-Xms128m -Xmx1024m -XX:PermSize=128m-XX:MaxPermSize=512m"

case"$1" in

start)

if[ -f $CATALINA_HOME/bin/startup.sh ];then

echo$"Start Tomcat"

$CATALINA_HOME/bin/startup.sh

fi

;;

stop)

if[ -f $CATALINA_HOME/bin/shutdown.sh ];then

echo$"Stop Tomcat"

$CATALINA_HOME/bin/shutdown.sh

fi

;;

*)

echo$"Usage:$0 {start|stop}"

exit1

;;

esac

exit$RETVAL

vim /etc/tomcat_8081.sh

RETVAL=$?

#tomcat instance directory

export CATALINA_HOME="/usr/java/tomcat/apache-tomcat-8.5.70"

#tomcat installation directory, change to your own

export CATALINA_BASE="$CATALINA_HOME/web-8081"

#Optional

exportJVM_OPTIONS="-Xms128m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m"

case"$1" in

start)

if[ -f $CATALINA_HOME/bin/startup.sh ];then

echo$"Start Tomcat"

$CATALINA_HOME/bin/startup.sh

fi

;;

stop)

if[ -f $CATALINA_HOME/bin/shutdown.sh ];then

echo$"Stop Tomcat"

$CATALINA_HOME/bin/shutdown.sh

fi

;;

*)

echo$"Usage:$0 {start|stop}"

exit1

;;

esac

exit$RETVAL

赋予权限

chmod +x tomcat_8080.sh

chmod +x tomcat_8081.sh

执行

./tomcat_8080.sh start

./tomcat_8081.sh start

如要停止请执行：./tomcat_8080.shstop

3.2.8测试

192.168.205.152:8080

192.168.205.153:8080

静态页面

192.168.205.152:8080/testweb/index.jsp

192.168.205.153:8080/testweb/index.jsp

注意:3.3-3.3.8步骤在150和151同步执行

3.3 配置nginx

3.3.1安装编译环境

yum groupinstall "Development Tools"（安装之前先打下面两条命令不然安不上）

yum groups mark install "DevelopmentTools"

yum groups mark convert "DevelopmentTools"

yum-y install openssl-devel libnl libnl-devel

3.3.2创建nginx用户和组(不允许登录)

groupaddnginx && useradd -s /sbin/nologin -g nginx nginx

3.3.3解压nginx压缩包，进入并编译安装

tar-zxf nginx-1.22.0.tar.gz && cd nginx-1.22.0

Nginxweb ui模块nginx-module-vts

负载均衡模块nginx-goodies-nginx-sticky

这两个模块放在nginx父目录(环境准备里有下载地址)

3.3.4预编译

./configure\

--prefix=/usr/local/nginx\

--user=nginx--group=nginx \

--with-http_stub_status_module\

--with-http_ssl_module\

--with-http_realip_module\

--add-module=../nginx-module-vts\

--add-module=../nginx-sticky-module-ng

执行完

3.3.5编译并安装

make -j8 && make install

3.3.6修改nginx配置文件

vim /usr/local/nginx/conf/nginx.conf

添加: include/usr/local/nginx/conf/conf.d/*.conf;

3.3.7添加server模块

vim /usr/local/nginx/conf/conf.d/tomcat.conf

set_real_ip_from 192.168.205.150/32;

set_real_ip_from 192.168.205.151/32;

real_ip_header X-Forwarded-For;

real_ip_recursiveon;

vhost_traffic_status_zone;

upstreamtomcat{

sticky;

server 192.168.205.152:8080;

server 192.168.205.153:8081;

server 192.168.205.152:8080;

server 192.168.205.153:8081;

}

server{

listen 8090;

server_name localhost;

location / {

proxy_pass http://tomcat;

root html;

index index.html index.htm;

}

location /status {

vhost_traffic_status_display;

vhost_traffic_status_display_formathtml;

}

error_log /var/nginx/logs/error.log error;

access_log /var/nginx/logs/access.log main;

}

3.3.8 启动，停止，重启，检测命令

/usr/local/nginx/sbin/nginx -t //检测配置语法

/usr/local/nginx/sbin/nginx -s reload //重启nginx

/usr/local/nginx/sbin/nginx -s stop //停止nginx

/usr/local/nginx/sbin/nginx //启动nginx

3.4部署keepalived

3.4.1解压keepalived压缩包并进入

3.4.2预编译(以下操作在keepalived 2.2.7目录中运行)

./configure--prefix=/usr/local/keepalived

3.4.3编译与安装

make-j8 && make install

3.4.4创建配置文件目录并复制二进制启动文件，配置文件，keepalived系统配置文件，keepalived指令

mkdir -p /etc/keepalived

二进制启动文件: cp keepalived/etc/init.d/keepalived /etc/init.d/

配置文件: cp /usr/local/keepalived/etc/keepalived/keepalived.conf/etc/keepalived/

keepalived系统配置文件:cpkeepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

keepalived指令: cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

3.4.5创建nginx检测脚本并执行

创建脚本目录

mkdir -p /opt/nginx_check

cat>> /opt/nginx_check/nginx_check.sh << EOF

#!/bin/bash

A=`ps-C nginx --no-header |wc -l`

if[ $A -eq 0 ];then

/usr/local/nginx/sbin/nginx

sleep 2

if [ `ps -C nginx --no-header |wc -l` -eq 0];then

killall keepalived

fi

fi

EOF

执行权限

chmod755 /opt/nginx_check/nginx_check.sh

3.4.6配置keepalived

192.168.205.150(主) vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

[email protected]

[email protected]

[email protected]

}

notification_email_from [email protected]

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL1

vrrp_skip_check_adv_addr

!vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

state MASTER

interface ens33

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.205.159

}

}

192.168.205.151(备) vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

[email protected]

[email protected]

[email protected]

}

notification_email_from [email protected]

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL2

vrrp_skip_check_adv_addr

!vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

state BACKUP

interface ens33

virtual_router_id 51

priority 90

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.205.159

}

}

3.4.7启动keepalived服务并让其开机自启动

systemctl start keepalived && systemctl enable keepalived

3.4.8查看配置结果

把192.168.205.150 关闭

可以看到已经飘到192.168.205.151

4.访问测试

192.168.205.150测试

192.168.205.159:8090

访问静态页面

http://192.168.205.159:8090/testweb/index.jsp

把192.168.205.150宕机

可以看到已经漂移到192.168.205.151

测试前把浏览器缓存清理一下

Shift+ctrl+delete 清理缓存

192.168.205.151测试

访问静态网页

http://192.168.205.159:8090/testweb/index.jsp