nginx实现请求的负载均衡 + keepalived实现nginx的高可用的shell脚本使用教程
nginx实现请求的负载均衡 + keepalived实现nginx的高可用的shell脚本使用教程
前言
- 使用集群是网站解决高并发、海量数据问题的常用手段。当一台服务器的处理能力、存储空间不足时，不要企图去换更强大的服务器，对大型网站而言，不管多么强大的服务器，都满足不了网站持续增长的业务需求。这种情况下，更恰当的做法是增加一台服务器分担原有服务器的访问及存储压力。通过负载均衡调度服务器，将来自浏览器的访问请求分发到应用服务器集群中的任何一台服务器上，如果有更多的用户，就在集群中加入更多的应用服务器，使应用服务器的负载压力不再成为整个网站的瓶颈。
一、准备工作
1、环境准备
master：192.168.25.71
backup：192.168.25.72
eurake服务1：192.168.25.51：1111
eurake服务2：192.168.25.52：1111
虚拟ip(VIP):192.168.25.47，对外提供服务的ip，也可称作浮动ip
2、安装4个CentOS7
配置服务器以上的ip地址
// 修改network文件，不同机器的文件名不一样，但都是ifcfg-开头
vi /etc/sysconfig/network-scripts/ifcfg-xxxxx
// 修改一下参数
BOOTPROTO=static
IPADDR=xxxxxxx // 如master IPADDR="192.168.25.71"
GATEWAY=xxxxxx // GATEWAY="192.168.25.2"
NETMASK="255.255.255.0"
DNS1="114.114.114.114"
ZONE=public
ONBOOT=yes
保存退出
重启网络
systemctl restart network
2、Linux启动两个eurake服务
- 浏览器访问成功
二、keepalived_master.sh+keepalived_backup.sh脚本实现负载均衡高可用集群
1、keepalived_master.sh
#!/bin/bash
#linux基本软件和nginx+keepalived的master高可用集群安装脚本
echo "开始安装vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安装完毕"
echo "安装telnet和xinetd和设置开机启动"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum install telnet-server.x86_64
yum install telnet.x86_64
yum list |grep xinetd
yum install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和设置开机启动安装完毕"
echo "开启telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "开启telnet和xinetd的service完毕"
echo "开启防火墙端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "开启防火墙端口完毕"
echo "安装net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安装net工具包完毕"
echo "安装wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安装wget成功"
echo "安装ntp时间同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安装ntp时间同步完毕"
echo "设开机启动ntp时间同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "设开机启动ntp时间同步完毕"
#nginx安装脚本
echo "Nginx安装"
echo "/usr/local/software文件夹"
mkdir /usr/local/software
echo "进入目录：cd /usr/local/software"
cd /usr/local/software
echo "Nginx安装"
wget http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
echo "安装依赖"
rpm -ivh nginx-release-centos-7-0.el7.ngx.noarch.rpm
echo "安装nginx"
yum install -y nginx
echo "安装nginx成功"
echo "开启80端口"
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload
echo "开启80端口完毕"
echo "开机启动nginx"
systemctl enable nginx
echo "删除/etc/nginx/nginx.conf"
rm -f /etc/nginx/nginx.conf
echo "创建/etc/nginx/nginx.conf"
echo "开始编辑/etc/nginx/nginx.conf"
echo "
user nginx; #运行用户
worker_processes 1; #启动进程,通常设置成和cpu的数量相等
#全局错误日志及PID文件
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# 工作模式及连接数上线
events {
use epoll; #epoll是多路复用IO(I/O Multiplexing)中的一种方式,但是仅用于linux2.6以上内核,可以大大提高nginx的性能
worker_connections 1024; #单个后台worker process进程的最大并发链接数
}
#设定http服务器,利用它的反向代理功能提供负载均衡支持
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
# sendfile 指令指定 nginx 是否调用 sendfile 函数（zero copy 方式）来输出文件，对于普通应用，
# 必须设为 on,如果用来进行下载等应用磁盘IO重负载应用，可设置为 off，以平衡磁盘与网络I/O处理速度，降低系统的uptime.
sendfile on;
tcp_nopush on;
tcp_nodelay on;
#连接超时时间
keepalive_timeout 65;
#添加eureka列表，真实应用服务器都放在这
upstream eureka {
#server eureka地址:端口号 weight表示权值，权值越大，被分配的几率越大;
server $1; #192.168.25.51:1111 weight=4 max_fails=2 fail_timeout=30s;
server $2; #192.168.25.52:1111 weight=4 max_fails=2 fail_timeout=30s;
}
include /etc/nginx/conf.d/*.conf;
}
" > /etc/nginx/nginx.conf
echo "编辑/etc/nginx/nginx.conf完毕"
echo "删除/etc/nginx/conf.d/default.conf"
rm -f /etc/nginx/conf.d/default.conf
echo "创建/etc/nginx/conf.d/default.conf"
echo "开始编辑/etc/nginx/conf.d/default.conf"
echo "server {
listen 80; #监听端口
server_name localhost;
#图片请求设置
location /images{
root /home/nginx/;
autoindex on;
}
#proxy_pass的http://eureka和/etc/nginx/nginx.conf的eureka列表upstream eureka一致
location / {
proxy_pass http://eureka;
}
#定义错误提示页面
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}" > /etc/nginx/conf.d/default.conf
echo "编辑/etc/nginx/conf.d/default.conf完毕"
echo "测试是否能正常启动"
nginx -t
nginx -c /etc/nginx/nginx.conf
echo "启动nginx"
nginx -s reload
echo "启动nginx成功"
#keepalived安装脚本
echo "keepalived安装"
yum install -y keepalived
echo "keepalived安装完毕"
echo "进入 cd /etc/keepalived"
cd /etc/keepalived
echo "删除keepalived.conf"
rm -f /etc/keepalived/keepalived.conf
echo "创建/etc/keepalived/keepalived.conf，第一行为global_defs {"
echo "开始编辑keepalived.conf"
echo "global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.hysec.com
smtp_connection_timeout 30
router_id nginx_master # 设置nginx_master的id，在一个网络应该是唯一的
}
vrrp_script chk_http_port {
script "/usr/local/src/check_nginx_pid.sh" #最后手动执行下此脚本，以确保此脚本能够正常执行
interval 2 #（检测脚本执行的间隔，单位是秒）
weight 2
}
vrrp_instance VI_1 {
state MASTER # 指定keepalived的角色，MASTER为主，BACKUP为备
interface ens33 # 当前进行vrrp通讯的网络接口卡(当前centos的网卡)
virtual_router_id 66 # 虚拟路由编号，主从要一直
priority 100 # 优先级，数值越大，获取处理请求的优先级越高
advert_int 1 # 检查间隔，默认为1s(vrrp组播周期秒数)
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_http_port #（调用检测脚本）
}
virtual_ipaddress {
$3 # 定义虚拟ip(VIP)，可多设，每行一个
}
}" > /etc/keepalived/keepalived.conf
echo "编辑keepalived.conf完毕"
echo "创建/usr/local/src/check_nginx_pid.sh"
echo "开始编辑check_nginx_pid.sh"
echo "#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
/usr/local/nginx/sbin/nginx #重启nginx
if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then #nginx重启失败
exit 1
else
exit 0
fi
else
exit 0
fi" > /usr/local/src/check_nginx_pid.sh
echo "编辑check_nginx_pid.sh完毕"
echo "启动keepalived"
/bin/systemctl start keepalived.service
echo "启动keepalived完毕"
echo "防火墙添加vrrp,防止脑裂"
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload
echo "防火墙加载完毕"
2、keepalived_backup.sh
#!/bin/bash
#linux基本软件和nginx+keepalived的master高可用集群安装脚本
echo "开始安装vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安装完毕"
echo "安装telnet和xinetd和设置开机启动"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum install telnet-server.x86_64
yum install telnet.x86_64
yum list |grep xinetd
yum install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和设置开机启动安装完毕"
echo "开启telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "开启telnet和xinetd的service完毕"
echo "开启防火墙端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "开启防火墙端口完毕"
echo "安装net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安装net工具包完毕"
echo "安装wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安装wget成功"
echo "安装ntp时间同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安装ntp时间同步完毕"
echo "设开机启动ntp时间同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "设开机启动ntp时间同步完毕"
#nginx安装脚本
echo "Nginx安装"
echo "/usr/local/software文件夹"
mkdir /usr/local/software
echo "进入目录：cd /usr/local/software"
cd /usr/local/software
echo "Nginx安装"
wget http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
echo "安装依赖"
rpm -ivh nginx-release-centos-7-0.el7.ngx.noarch.rpm
echo "安装nginx"
yum install -y nginx
echo "安装nginx成功"
echo "开启80端口"
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload
echo "开启80端口完毕"
echo "开机启动nginx"
systemctl enable nginx
echo "删除/etc/nginx/nginx.conf"
rm -f /etc/nginx/nginx.conf
echo "创建/etc/nginx/nginx.conf"
echo "开始编辑/etc/nginx/nginx.conf"
echo "
user nginx; #运行用户
worker_processes 1; #启动进程,通常设置成和cpu的数量相等
#全局错误日志及PID文件
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# 工作模式及连接数上线
events {
use epoll; #epoll是多路复用IO(I/O Multiplexing)中的一种方式,但是仅用于linux2.6以上内核,可以大大提高nginx的性能
worker_connections 1024; #单个后台worker process进程的最大并发链接数
}
#设定http服务器,利用它的反向代理功能提供负载均衡支持
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
# sendfile 指令指定 nginx 是否调用 sendfile 函数（zero copy 方式）来输出文件，对于普通应用，
# 必须设为 on,如果用来进行下载等应用磁盘IO重负载应用，可设置为 off，以平衡磁盘与网络I/O处理速度，降低系统的uptime.
sendfile on;
tcp_nopush on;
tcp_nodelay on;
#连接超时时间
keepalive_timeout 65;
#添加eureka列表，真实应用服务器都放在这
upstream eureka {
#server eureka地址:端口号 weight表示权值，权值越大，被分配的几率越大;
server $1; #192.168.25.51:1111 weight=4 max_fails=2 fail_timeout=30s;
server $2; #192.168.25.52:1111 weight=4 max_fails=2 fail_timeout=30s;
}
include /etc/nginx/conf.d/*.conf;
}
" > /etc/nginx/nginx.conf
echo "编辑/etc/nginx/nginx.conf完毕"
echo "删除/etc/nginx/conf.d/default.conf"
rm -f /etc/nginx/conf.d/default.conf
echo "创建/etc/nginx/conf.d/default.conf"
echo "开始编辑/etc/nginx/conf.d/default.conf"
echo "server {
listen 80; #监听端口
server_name localhost;
#图片请求设置
location /images{
root /home/nginx/;
autoindex on;
}
#proxy_pass的http://eureka和/etc/nginx/nginx.conf的eureka列表upstream eureka一致
location / {
proxy_pass http://eureka;
}
#定义错误提示页面
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}" > /etc/nginx/conf.d/default.conf
echo "编辑/etc/nginx/conf.d/default.conf完毕"
echo "测试是否能正常启动"
nginx -t
nginx -c /etc/nginx/nginx.conf
echo "启动nginx"
nginx -s reload
echo "启动nginx成功"
#keepalived安装脚本
echo "keepalived安装"
yum install -y keepalived
echo "keepalived安装完毕"
echo "进入 cd /etc/keepalived"
cd /etc/keepalived
echo "删除keepalived.conf"
rm -f /etc/keepalived/keepalived.conf
echo "创建/etc/keepalived/keepalived.conf，第一行为global_defs {"
echo "开始编辑keepalived.conf"
echo "global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.hysec.com
smtp_connection_timeout 30
router_id nginx_backup # 设置nginx_backup的id，在一个网络应该是唯一的
}
vrrp_script chk_http_port {
script "/usr/local/src/check_nginx_pid.sh" #最后手动执行下此脚本，以确保此脚本能够正常执行
interval 2 #（检测脚本执行的间隔，单位是秒）
weight 2
}
vrrp_instance VI_1 {
state BACKUP # 指定keepalived的角色，MASTER为主，BACKUP为备
interface ens33 # 当前进行vrrp通讯的网络接口卡(当前centos的网卡)
virtual_router_id 66 # 虚拟路由编号，主从要一直
priority 99 # 优先级，数值越大，获取处理请求的优先级越高
advert_int 1 # 检查间隔，默认为1s(vrrp组播周期秒数)
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_http_port #（调用检测脚本）
}
virtual_ipaddress {
$3 # 定义虚拟ip(VIP)，可多设，每行一个
}
}" > /etc/keepalived/keepalived.conf
echo "编辑keepalived.conf完毕"
echo "创建/usr/local/src/check_nginx_pid.sh"
echo "开始编辑check_nginx_pid.sh"
echo "#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
/usr/local/nginx/sbin/nginx #重启nginx
if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then #nginx重启失败
exit 1
else
exit 0
fi
else
exit 0
fi" > /usr/local/src/check_nginx_pid.sh
echo "编辑check_nginx_pid.sh完毕"
echo "启动keepalived"
/bin/systemctl start keepalived.service
echo "启动keepalived完毕"
echo "防火墙添加vrrp,防止脑裂"
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload
echo "防火墙加载完毕"
三、运行脚本
1、运行master脚本
-
我把keepalived_master.sh上传到/home目录下，用以下命令执行脚本权限
- chmod 777 ./*.sh
-
执行脚本命令，keepalived_master.sh脚本里面$1代表第一个ip地址，$2,$3代表第二第三个
- ./keepalived_master.sh 192.168.25.51:1111 192.168.25.52:1111 192.168.25.47
安装软件都输入y
2、运行backup脚本
-
和master脚本一样,执行命令为
- ./keepalived_backup.sh 192.168.25.51:1111 192.168.25.52:1111 192.168.25.47
运行完毕
四、查看ip地址是否能运行
访问主master的ip地址
刷新ip地址，看到会切换到eurake服务2
访问备backup的ip地址也一样
访问VIP虚拟地址 192.168.25.47
刷新VIP虚拟地址
master虚拟机输入ip addr查看,VIP虚拟地址在master这里，如果不在就是配置文件出了错误
backup虚拟机输入ip addr查看，虚拟地址现在在master哪里，如果backup也出现了VIP，说明备节点和主节点争用IP资源，这个现象叫“脑裂”。CentOS7我遇到过这个问题,关闭防火墙就不会”脑裂“。生产环境都是要开启防火墙的，所以要配置脚本最后那个防火墙设置,添加vrrp。
四、高可用的主备服务器切换测试
-
停掉主节点的keepalived服务，查看备节点会不会生成VIP：192.168.25.47
- /bin/systemctl stop keepalived.service
查看主，备节点，VIP飘到备节点了
启动主节点的keepalived服务，然后查看主节点和备节点的VIP，主节点应该会抢夺回来VIP
可以输入 tail -f /var/log/messages 查看日志
五、总结
-
访问虚拟IP(VIP)，keepalived将请求映射到本地nginx，nginx将请求转发至eurake服务，例如：http://192.168.25.47，被映射成http://192.168.25.71，端口是80，而71上nginx的端口正好是80；映射到nginx上后，nginx再进行请求的转发。
-
VIP总会在keepalived服务器中的某一台上，也只会在其中的某一台上；VIP绑定的服务器上的nginx就是master，当VIP所在的服务器宕机了，keepalived会将VIP转移到backup上，并将backup提升为master。
-
VIP也称浮动ip，是公网ip，与域名进行映射，对外提供服务； 其他ip一般而言都是内网ip， 外部是直接访问不了的。
-
keepalived.conf，nginx.conf的配置文件可简单可复杂，看自己需要怎么配。