项目需求
当我们访问某个网站的时候可以在浏览器中输入IP或者域名链接到Web Server进行访问,如果这个Web Server挂了,那么整个系统都无法使用,用户也就不能进行正常的访问,这种情况将对公司产生一定的影响。这就是我们常说的系统中的单点故障。这部分的单点故障可以通过引入负载均衡器和至少另一个Web Server来缓解。同时由于有多台服务器同时提供服务,也加大了系统的负载能力提高了性能。
因此我们采用LVS的负载均衡技术,将前端请求按照设定规则调度到后端服务器,并与keepalived相结合实现高可用负载均衡。
项目拓扑
项目环境
主机名 | 主机IP | 主机角色 |
---|---|---|
K1 | 192.168.36.110 | Keepalived-Master |
K2 | 192.168.36.111 | Keepalived-Backup |
WEB1 | 192.168.36.112 | Nginx、PHP |
WEB2 | 192.168.36.113 | Nginx、PHP |
NFS | 192.168.36.114 | NFS |
Mariadb-M | 192.168.36.115 | Mariadb-Master |
Mariadb-S | 192.168.36.116 | Mariadb-Slave |
开始部署:Keepalived服务器配置
安装Keepalived
[root@k1 ~]#yum install -y keepalived
修改Keepalived配置文件,开启邮件通知功能
[root@k1 ~]#vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs { # 全局配置
notification_email {
[email protected] # keepalived 发生故障切换时邮件发送的对象,可以按行区分写多个
}
notification_email_from [email protected] # 设置邮件发送地址
smtp_server 127.0.0.1 # smtp服务器地址
smtp_connect_timeout 30 # 指定smtp连接超时时间
router_id K1.mylinuxops.com # 运行keepalived服务器标识,发送邮件时显示在邮件标题中的信息
vrrp_skip_check_adv_addr # 所有报文都检查比较消耗性能,此配置为如果收到的报文和上一个报文是同一个路由器则跳过检查报文中的源地址
#vrrp_strict # 严格遵守VRRP协议,不允许状况:1,没有VIP地址,2.单播邻居,3.在VRRP版本2中有IPv6地址
vrrp_iptables # 严格遵守VRRP防火墙规则
vrrp_garp_interval 0 # ARP保温发送延迟
vrrp_gna_interval 0 # 消息发送延迟
}
vrrp_instance VI_1 { # vrrp实例定义
state MASTER # 在此虚拟路由器上节点的初始状态:其中所有服务器里只有一个可以是MASTER节点,其余的是BACKUP节点
interface ens33 # 指定HA检测网络的接口,即网卡名称
virtual_router_id 27 # 当前虚拟路由器的惟一标识,范围是0-255
priority 100 # 当前主机在此虚拟路径器中的优先级;范围1-254。主服务器一定要高于备用服务器,且两者之间的数值差越小越好
advert_int 1 # vrrp通告间隔
authentication { # 存储的验证类型和密码以进行验证
auth_type PASS # 进行验证类型:类型仅可以设置成PASS和AH两种
auth_pass 1111 # 进行验证的密码:在同一个vrrp_instance中,使用相同的密码才能进行正确的通信
}
unicast_src_ip 192.168.36.110 # 单播配置
unicast_peer {
192.168.36.111 # 目标主机IP
}
virtual_ipaddress { # 虚拟IP的网络地址,即VIP地址
192.168.36.100 dev ens33 label ens33:0
192.168.36.200 dev ens33 label ens33:1
}
# 定义邮件通知脚本
notify_master "/etc/keepalived/notify.sh master" # 当前节点成为主节点时触发的脚本
notify_backup "/etc/keepalived/notify.sh backup" # 当前节点转为备节点时触发的脚本
notify_fault "/etc/keepalived/notify.sh fault" # 当前节点转为“失败”状态时触发的脚本
}}
# 编写邮件通知脚本
[root@k1 ~]#vim /etc/keepalived/notify.sh
#!/bin/bash
contact='[email protected]' # 通知的邮箱(首先需要确保能连通外网,否则通知不过去)
notify() {
mailsubject="$(hostname) to be $1, vip转移" # 主题
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" # 邮件内容
echo "$mailbody" | mail -s "$mailsubject" $contact # 邮件发送的内容
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
# 添加执行权限
[root@k1 ~]#chmod a+x /etc/keepalived/notify.sh
# 邮箱配置
[root@k1 ~]#yum install -y mailx
[root@k1 ~]#vim /etc/mail.rc
set bsdcompat
set [email protected] # 接收邮件的邮箱
set smtp=smtp.qq.com
set [email protected]
set smtp-auth-password=kosulaxbbhxrgaci # 邮箱授权码(我的邮箱-->设置-->账户-->开启POP3/SMTP等服务,生成授权码)
[root@k1 ~]#chmod a+x /etc/mail.rc # 文件添加执行权限
# 重启keepalived服务
[root@k1 ~]#systemctl restart keepalived
# 生成VIP地址(注:Master存活时,VIP在Master上,Slave上不会存在VIP。当Master宕机,VIP将调到Slave上)
[root@k1 ~]#ifconfig
ens33:0: flags=4163 mtu 1500
inet 192.168.36.100 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
ens33:1: flags=4163 mtu 1500
inet 192.168.36.200 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
# 将keepalived配置文件 scp 到BACKUP服务器中
[root@k1 ~]#scp /etc/keepalived/keepalived.conf 192.168.36.104:/etc/keepalived/keepalived.conf
[email protected]'s password:
keepalived.conf 100% 1374 1.4MB/s 00:00
# BACKUP服务器配置,其余配置与MASTER相同
[root@k1 ~]#vim /etc/keepalived/keepalived.conf
....
state BACKUP # 修改为BACKUP节点
interface ens33
virtual_router_id 37 # 修改标识为37
priority 90 # 修改优先级,需要比 MASTER 节点低
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 192.168.36.111
unicast_peer {
192.168.36.110
}
....
# 编写邮件通知脚本
[root@k1 ~]#vim /etc/keepalived/notify.sh
#!/bin/bash
contact='[email protected]' # 通知的邮箱(首先需要确保能连通外网,否则通知不过去)
notify() {
mailsubject="$(hostname) to be $1, vip转移" # 主题
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" # 邮件内容
echo "$mailbody" | mail -s "$mailsubject" $contact # 邮件发送的内容
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
# 添加执行权限
[root@k1 ~]#chmod a+x /etc/keepalived/notify.sh
# 邮箱配置
[root@k1 ~]#yum install -y mailx
[root@k1 ~]#vim /etc/mail.rc
set bsdcompat
set [email protected] # 接收邮件的邮箱
set smtp=smtp.qq.com
set [email protected]
set smtp-auth-password=kosulaxbbhxrgaci # 邮箱授权码(我的邮箱-->设置-->账户-->开启POP3/SMTP等服务,生成授权码)
[root@k1 ~]#chmod a+x /etc/mail.rc # 文件添加执行权限
# 重启keepalived服务,并进行宕机测验,查看VIP跳转到K2服务器上
[root@k2 ~]#ifconfig
ens33:0: flags=4163 mtu 1500
inet 192.168.36.100 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
ens33:1: flags=4163 mtu 1500
inet 192.168.36.200 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
VIP跳转邮箱接收邮件
搭建WEB站点(两个WEB站点执行相同操作)
# 编写Nginx编译安装脚本
[root@WEB1 ~]#vim nginx.sh
#!/bin/bash
yum install -y vim lrzsz tree screen psmisc lsof tcpdump wget ntpdate gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools iotop bc zip unzip zlib-devel bash-completion nfs-utils automake libxml2 libxml2-devel libxslt libxslt-devel perl perl-ExtUtils-Embed &>/dev/null
wget https://nginx.org/download/nginx-1.14.2.tar.gz &>/dev/null
cd nginx-1.14.2/
./configure --prefix=/apps/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-stream_ssl_module --with-stream_realip_module &>/dev/null
make && make install
useradd nginx -s /sbin/nologin -u 2000
chown nginx.nginx -R /apps/nginx/
echo "\n=======================版本==================================\n"
/apps/nginx/sbin/nginx -V
# 给脚本添加执行权限
[root@WEB1 ~]#chmod a+x nginx.sh
# 启动安装脚本
[root@WEB1 ~]#./nginx.sh
# 成功安装Nginx,做nginx命令软链接
[root@WEB1 ~]#ln -sv /apps/nginx/sbin/nginx /usr/sbin/
‘/usr/sbin/nginx’ -> ‘/apps/nginx/sbin/nginx’
# 启动Nginx
[root@WEB1 ~]#nginx
# 查看80端口
[root@WEB1 ~]#ss -ntl
# 修改Nginx配置文件,使其开启php页面访问功能
[root@WEB1 ~]#vim /apps/nginx/conf/nginx.conf
2 user nginx nginx;
9 pid logs/nginx.pid;
39 charset utf-8;
42 location / {
43 root html;
44 index index.php index.html index.htm;
45 }
64 location ~ \.php$ {
65 root /apps/nginx/html;
66 fastcgi_pass 127.0.0.1:9000;
67 fastcgi_index index.php;
68 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
69 include fastcgi_params;
70 }
# 编写php状态页面
[root@WEB1 ~]#vim /apps/nginx/html/index.php
# Nginx启动前对配置文件进行检查
[root@WEB1 ~]#nginx -t
# 重新加载Nginx配置文件
[root@WEB1 ~]#nginx -s reload
nginx: the configuration file /apps/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /apps/nginx/conf/nginx.conf test is successful
# 安装php-fpm模块
[root@WEB1 ~]#yum install -y php-fpm php-mysql
# 编写php-fpm模块配置文件
[root@WEB1 ~]#vim /etc/php-fpm.d/www.conf
12 listen = 127.0.0.1:9000
33 listen.mode = 0666
39 user = nginx
41 group = nginx
# 启动php-fpm
[root@WEB1 ~]#systemctl restart php-fpm
# 9000端口查看
[root@WEB1 ~]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:9000 *:*
两个keepalived服务器中添加 web 的虚拟服务器
[root@k1 ~]#vim /etc/keepalived/keepalived.conf
....
virtual_server 192.168.36.100 80 { # 虚拟服务器的虚拟IP地址和服务的端口号
delay_loop 6 # 系统执行健康检查的时间间隔
lb_algo wrr # lvs调度的算法:wrr轮询算法
lb_kind DR # LVS的DR直接路由机制
protocol TCP # 指定转发协议,TCP/UDP
real_server 192.168.36.112 80 { # 实际服务器IP地址和端口号
weight 1 # 权重值
TCP_CHECK { # 通过tcpcheck判断RealServer的健康状态
connect_port 80 # 检测连接端口
connect_timeout 5 # 连接超时时间
nb_get_retry 3 # 重连次数
delay_before_retry 3 # 重连时间间隔
}
}
real_server 192.168.36.113 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
}
# 重新启动keepalived服务
[root@k1 ~]#systemctl restart keepalived
# 查看生成的ipvsadm规则
[root@k1 ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.36.100:80 wrr
-> 192.168.36.112:80 Route 1 0 0
-> 192.168.36.113:80 Route 1 0 0
两个WEB服务器创建lvs检测脚本(步骤相同)
[root@WEB1 ~]#vim lvs_dr_rs.sh
#!/bin/bash
vip=192.168.36.100
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
# 添加执行权限并运行脚本
[root@WEB1 ~]#chmod a+x ./lvs_dr_rs.sh
[root@WEB1 ~]#./lvs_dr_rs.sh start
The RS Server is Ready!
# 生成检测的虚拟IP
[root@WEB1 ~]#ifconfig lo:1
lo:1: flags=73 mtu 65536
inet 192.168.36.100 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
状态页面查看
搭建数据库主从复制服务器
Master服务器
# yum安装mariadb服务
[root@Mariadb-M ~]#yum install -y mariadb-server
# 启动数据库服务
[root@Mariadb-M ~]#systemctl restart mariadb
# 修改mariadb配置文件
[root@Mariadb-M ~]#vim /etc/my.cnf
[mysqld]
server_id=1 # ID号
binlog_format=row # 基于行复制
log-bin=/data/bin/mysql-bin # 生成二进制文件的目录与格式
# 创建二进制文件存放的目录
[root@Mariadb-M ~]#mkdir /data/bin
# 授予目录所属关系
[root@Mariadb-M ~]#chown mysql.mysql /data/bin/ -R
# 重新启动mariadb服务
[root@Mariadb-M ~]#systemctl restart mariadb
# 执行安全脚本
[root@Mariadb-M ~]#mysql_secure_installation
# 进入数据库
[root@Mariadb-M ~]#mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 19
Server version: 5.5.60-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant replication slave on *.* to repluser@'192.168.36.%' identified by 'centos'; # 添加slave复制权限
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> flush privileges; # 刷新权限
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> reset master; # 重置master二进制文件大小
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show master logs; # 查看并记录
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 245 |
+------------------+-----------+
1 row in set (0.00 sec)
Slave服务器
# yum安装mariadb服务
[root@Mariadb-M ~]#yum install -y mariadb-server
# 启动数据库服务
[root@Mariadb-M ~]#systemctl restart mariadb
# 修改mariadb配置文件
[root@Mariadb-M ~]#vim /etc/my.cnf
[mysqld]
server_id=2 # ID号
read_only # 只读
# 重新启动mariadb服务
[root@Mariadb-M ~]#systemctl restart mariadb
# 执行安全脚本
[root@Mariadb-M ~]#mysql_secure_installation
# 进入数据库
[root@Mariadb-S ~]#mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 5.5.60-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CHANGE MASTER TO # Slave节点添加同步Master数据库的语句
-> MASTER_HOST='192.168.36.115',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='centos',
-> MASTER_PORT=3306,
-> MASTER_LOG_FILE='mysql-bin.000001',
-> MASTER_LOG_POS=245;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> slave start; # 启用从节点
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show slave status\G; # 状态查看
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.36.115
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 245
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 529
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes # IO、SQL线程已经启动,数据同步
Slave_SQL_Running: Yes
Master、Slave数据同步测试
MariaDB [(none)]> create database darius; # Master节点创建一个darius数据库
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| darius |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> show databases; # Slave查看,如同步成功,则主从复制创建完成。
+--------------------+
| Database |
+--------------------+
| information_schema |
| darius |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
两个keepalived服务器添加mariad虚拟服务器
[root@k1 ~]#vim /etc/keepalived/keepalived.conf
....
virtual_server 192.168.36.200 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.36.115 3306 {
weight 1
TCP_CHECK {
connect_port 3306
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.36.116 3306 {
weight 1
TCP_CHECK {
connect_port 3306
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
}
....
# 生成ipvsadm规则
[root@k1 ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.36.100:80 wrr
-> 192.168.36.112:80 Route 1 0 0
-> 192.168.36.113:80 Route 1 0 0
TCP 192.168.36.200:3306 wrr
-> 192.168.36.115:3306 Route 1 0 0
-> 192.168.36.116:3306 Route 1 0 0
两个mariadb服务器创建lvs检测脚本(步骤相同)
[root@Mariadb-S ~]#vim lvs_dr_rs.sh
#!/bin/bash
vip=192.168.36.200
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
# 给脚本添加执行权限
[root@Mariadb-S ~]#chmod a+x lvs_dr_rs.sh
# 运行脚本
[root@Mariadb-S ~]#./lvs_dr_rs.sh start
The RS Server is Ready!
# 生成检测的虚拟IP
[root@Mariadb-S ~]#ifconfig lo:1
lo:1: flags=73 mtu 65536
inet 192.168.36.200 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
搭建NFS服务器
# 修改NFS配置文件,设置将要共享的目录
[root@NFS ~]#vim /etc/exports
/data *(rw,no_root_squash)
# 重新启动NFS服务,并设置开机启动
[root@NFS ~]#systemctl restart nfs
[root@NFS ~]#systemctl enable nfs
# 查看NFS共享目录
[root@NFS ~]#exportfs -v
/data (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
# 解压wordpress包
[root@NFS ~]#unzip wordpress-5.0-zh_CN.zip
# 将wordpress包内文件移动到共享目录中,进行共享
[root@NFS ~]#mv wordpress/* /data/
[root@NFS ~]#cd /data/
# 生成wordpress配置文件
[root@NFS data]#mv wp-config-sample.php wp-config.php
# 修改wordpress配置文件
[root@NFS data]#vim wp-config.php
...
/** WordPress数据库的名称 */
define('DB_NAME', 'wordpress');
/** MySQL数据库用户名 */
define('DB_USER', 'wpuser');
/** MySQL数据库密码 */
define('DB_PASSWORD', 'centos');
/** MySQL主机 */
define('DB_HOST', '192.168.36.200');
/** 创建数据表时默认的文字编码 */
define('DB_CHARSET', 'utf8');
...
将NFS共享的文件挂载到两个WEB服务器中
# 写入fstab文件中,开机自动挂载
[root@WEB1 ~]#vim /etc/fstab
192.168.36.114:/data /apps/nginx/html nfs _netdev,defaults 0 0
# 查看挂载情况
[root@WEB2 ~]#df -h
Filesystem Size Used Avail Use% Mounted on
192.168.36.114:/data 95G 3.9G 92G 5% /apps/nginx/html