1、Nginx+Keepalived实现站点高可用
-
网络拓扑
- 环境说明
- Nginx代理服务器:10.192.1.163、10.192.1.164
- Keepalived 高可用程序提供对外服务IP地址VIP:10.192.1.99
- Web服务:10.192.1.66、10.192.1.67、10.192.1.68提供站点服务
- 搭建
- 10.192.1.163/64安装、配置Nginx和Keepalived
安装程序包 yum -y install nginx keepalived 配置nginx代理 vim /etc/nginx/nginx.conf http { ... upstream web { #用upstream模拟四层代理,对后端web站点进行代理 server 10.192.1.66:80; server 10.192.1.67:80; server 10.192.1.68:80; } server { ... location / { proxy_pass http://web; } ... } ... } 配置keepalived (MASTER) vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.1.1.11 } vrrp_script chk_down { script "/etc/keepalived/file.sh" #文件检测脚本 weight -5 interval 1 fall 1 rise 1 } vrrp_script chk_nginx { script "killall -0 nginx && exit 0 || exit 1" #nginx进程检测脚本 weight -5 interval 2 fall 3 rise 3 } vrrp_instance VI_1 { #虚拟路由配置 state MASTER interface ens32 virtual_router_id 25 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 11112222 } virtual_ipaddress { 10.192.1.99/24 dev ens32 label ens32:0 } track_script { #监控脚本 chk_down chk_nginx } notify_master "/etc/keepalived/notify.sh master" #转换通知 notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } 配置keepalived (BACKUP) router_id node2 state BACKUP priority 98 notify.sh脚本, #!/bin/bash # contact='root@localhost' notify() { local mailsubject="$(hostname) to be $1,vip floating" local mailbody="$(date +'%F %T'):vrrp transition,$(hostname) change to $1" echo "$mailbody" |mail -s "$mailsubject" $contact } case $1 in master) systemctl start nginx #转变为master时开启nginx notify master ;; backup) systemctl stop nginx #转变为bakcup时停止nginx notify backup ;; fault) systemctl stop nginx #转变为失败时停止nginx notify fault ;; *) echo "Usage:$(basename $0) {master|backup|fault}" exit 1 ;; esac file.sh 手动切换脚本 存在down文件就发生切换 #!/bin/bash # [[ -f /etc/keepalived/down ]] && exit 1 || exit 0
- 10.192.1.66/67/68安装、配置web服务,这里也安装Nginx
10.192.1.66/67/68 提供同一个web站点 为演示实现效果这里分别用
web 66
web 67
web 66
来表示不同的机器
- 测试可用性
- 模拟后台66宕机
在66主机上执行iptables命令 [root@localhost conf.d]# iptables -I INPUT -d 10.192.1.66 -j REJECT 客户机每3秒访问一次站点 [root@ceph_deploy ping]# for i in {1..10} ;do sleep 3 ;curl 10.192.1.99 ;done
web 66
web 68
web 67
web 67
web 68
web 67
web 68
web 67
web 68
web 68
[root@ceph_deploy ping]#
- 模拟Nginx代理服务器故障和恢复
在163主机上touch一个down文件,实现vip地址漂移 [root@node1 keepalived]# touch down 通过抓包监控主备切换 20:03:20.487986 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:03:21.489032 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 95, authtype simple, intvl 1s, length 20 20:03:21.517232 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:03:22.517422 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:03:23.519136 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:03:24.520448 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 在163主机上将down文件删除,实现vip地址漂移回来 20:07:12.616295 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:07:13.617912 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:07:13.619004 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:07:14.618981 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:07:15.620928 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:07:16.624430 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20
2、实现keepalived主主模型
-
网络拓扑
- 环境说明
- Nginx代理服务器:10.192.1.163、10.192.1.164
- Keepalived 高可用程序提供对外服务IP地址VIP:10.192.1.98和10.192.1.99
- Web服务:10.192.1.66、10.192.1.67、10.192.1.68提供站点服务
- 10.192.1.98和10.192.1.99可以代理同一个web站点,也可以是不同站点,本实验为同一个站点
- VIP:10.192.1.99以10.192.1.163为主,VIP:10.192.1.99以10.192.1.164为主
- 搭建
在上面配置的基础上只需要对keepalived配置进行响应修改即可
在163服务器上配置如下:
[root@node1 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.1.1.11
}
vrrp_script chk_down {
script "/etc/keepalived/file.sh"
weight -5
interval 1
fall 1
rise 1
}
vrrp_script chk_nginx {
script "killall -0 nginx && exit 0 || exit 1"
weight -5
interval 2
fall 3
rise 3
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 25
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
10.192.1.99/24 dev ens32 label ens32:0
}
track_script {
chk_down
chk_nginx
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 { #添加第二个虚拟路由器配置
state BACKUP
interface ens32
virtual_router_id 24
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
10.192.1.98/24 dev ens32 label ens32:1
}
track_script {
chk_down
chk_nginx
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
在164服务器上配置
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
#vrrp_skip_check_adv_addr
#vrrp_strict
#vrrp_garp_interval 0
#vrrp_gna_interval 0
vrrp_mcast_group4 224.1.1.11
}
vrrp_script chk_down {
script "/etc/keepalived/file.sh"
weight -5
interval 1
fall 1
rise 1
}
vrrp_script chk_nginx {
script "killall -0 nginx && exit 0 || exit 1"
weight -5
interval 2
fall 3
rise 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 25
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
10.192.1.99/24 dev ens32 label ens32:0
}
track_script {
chk_down
chk_nginx
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 { #第二个虚拟路由器配置
state MASTER
interface ens32
virtual_router_id 24
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
10.192.1.98/24 dev ens32 label ens32:1
}
track_script {
chk_down
chk_nginx
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
- 测试
- 正常情况下
20:36:06.616278 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:36:06.618772 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:36:07.616765 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:36:07.618843 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:36:08.617185 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:36:08.619023 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:36:09.617493 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:36:09.620231 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:36:10.617977 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:36:10.619289 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 [root@ceph_deploy ping]# for i in {1..10} ;do curl 10.192.1.99 && curl 10.192.1.98 ;echo "--------";done
web 67
web 67
--------
web 67
web 68
--------
web 68
web 66
--------
web 68
web 66
--------
web 66
web 68
--------
web 66
web 67
--------
web 67
web 67
--------
web 67
web 68
--------
web 68
web 66
--------
web 68
web 66
-------- [root@ceph_deploy ping]#
- 模拟163主机宕机
163 停止nginx服务 [root@node1 keepalived]# systemctl stop nginx 客户端访问站点情况 [root@ceph_deploy ping]# for i in {1..10} ;do sleep 3; curl 10.192.1.99 ; curl 10.192.1.98 ;echo "--------";done
web 66
web 67
--------
web 66
web 66
-------- curl: (7) Failed connect to 10.192.1.99:80; Connection refused
web 68
-------- curl: (7) Failed connect to 10.192.1.99:80; No route to host
web 68
--------
web 67
web 66
--------
web 66
web 67
--------
web 68
web 67
--------
web 66
web 68
--------
web 67
web 66
--------
web 68
web 66
-------- VIP飘移过程 20:45:55.961315 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:45:55.974024 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:45:56.961483 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 100, authtype simple, intvl 1s, length 20 20:45:56.974199 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:45:57.961596 IP 10.192.1.163 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 95, authtype simple, intvl 1s, length 20 20:45:57.962084 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:45:57.974398 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:45:58.962709 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:45:58.974582 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:45:59.964495 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:45:59.974764 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:46:00.965066 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:46:00.974976 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:46:01.966199 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:46:01.975185 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20 20:46:02.966805 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 25, prio 98, authtype simple, intvl 1s, length 20 20:46:02.975735 IP 10.192.1.164 > 224.1.1.11: VRRPv2, Advertisement, vrid 24, prio 100, authtype simple, intvl 1s, length 20
3、采用varnish为nginx实现缓存加速
- 环境准备
varnish服务器:10.192.1.167
nginx服务器:10.192.1.99 - varnish安装配置
[root@localhost ~]# yum -y install varnish
[root@localhost ~]# cd /etc/varnish/
[root@localhost varnish]# vim varnish.params
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_ADDRESS=10.192.1.167
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_STORAGE="file,/data/varnish/cache,256M"
VARNISH_USER=varnish
VARNISH_GROUP=varnish
[root@localhost varnish]# vim default.vcl
backend default {
.host = "10.192.1.99";
.port = "80";
}
sub vcl_recv {
}
sub vcl_backend_response {
}
sub vcl_deliver {
if (obj.hits>0) {
set resp.http.X-Cache="HIT via " + server.ip;
} else {
set resp.http.X-Cache="MISS via " + server.ip;
}
}
[root@localhost varnish]# systemctl enable varnish
[root@localhost varnish]# systemctl start varnish
- 测试
[root@ceph_deploy ~]# curl -I http://10.192.1.167
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 29 Apr 2019 12:08:58 GMT
Content-Type: text/html
Content-Length: 17
Last-Modified: Mon, 29 Apr 2019 06:20:10 GMT
ETag: "5cc6979a-11"
X-Varnish: 33065
Age: 0
Via: 1.1 varnish-v4
X-Cache: MISS via 10.192.1.167 第一次访问没有命中缓存
Connection: keep-alive
[root@ceph_deploy ~]# curl -I http://10.192.1.167
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 29 Apr 2019 12:08:58 GMT
Content-Type: text/html
Content-Length: 17
Last-Modified: Mon, 29 Apr 2019 06:20:10 GMT
ETag: "5cc6979a-11"
X-Varnish: 290 33066
Age: 33
Via: 1.1 varnish-v4
X-Cache: HIT via 10.192.1.167 第二次访问命中缓存
Connection: keep-alive
[root@ceph_deploy ~]#
4、LNMP结合varnish实现动静分离
-
网络拓扑
- 环境准备
10.192.1.163 安装varnish
10.192.1.161 安装Nginx提供web静态页面服务
10.192.1.167 安装Nginx、php-fpm、mariadb提供动态站点服务 - varnish服务器配置
修改varnish主配置文件将监听端口修改为80端口
[root@node1 varnish]# vim varnish.params
VARNISH_LISTEN_PORT=80
启动varinish服务
[root@node1 varnish]# systemctl start varnish
修改varinish缓存规则文件
vcl 4.0;
backend default { #默认后端服务器
.host = "10.192.1.167";
.port = "80";
}
backend nginxsrvs { #静态网页web服务器
.host = "10.192.1.161";
.port = "80";
}
sub vcl_purge { #定义清除缓存规则
return (synth(200,"Purged"));
}
sub vcl_recv {
if (req.method == "PURGE") { #通过PURGE方法清除缓存
return (purge);
}
if (req.url ~ "(?i)^/(login|admin)") {
return(pass);
}
if (req.url ~ "(?i)\.(html|htm|css|svg|js|jpg|jpeg|png|gif|pdf)") { #访问静态内容转发到静态服务器
set req.backend_hint = nginxsrvs;
} else {
set req.backend_hint = default; #否则转发至默认主机
}
}
sub vcl_backend_response {
}
sub vcl_deliver {
if (obj.hits>0) {
set resp.http.X-Cache="HIT via " + server.ip;
} else {
set resp.http.X-Cache="MISS via " + server.ip;
}
}
使配置规则生效
[root@node1 varnish]# varnish_reload_vcl
- 配置nmp服务器
这里安装WordPress软件提供动态web站点,安装过程比较简单,官网下载,解压至web目录
这里是/data/wordpress目录
[root@localhost wordpress]# tar -xzvf wordpress-5.0.3-zh_CN.tar.gz -C /data/
启动php-fpm服务
[root@localhost wordpress]# systemctl start php-fpm
启动mariadb创建数据库并授权访问
[root@localhost wordpress]# systemctl start mariadb
MariaDB [(none)]> CREATE DATABASE wordpress;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'localhost' IDENTIFIED BY 'wordpress';
配置nginx,并启动nginx服务
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
#root /usr/share/nginx/html;
root /data/wordpress;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
index index.php index.html index.htm;
location / {
}
location ~ \.php$ {
root /data/wordpress;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
[root@localhost wordpress]# systemctl start nginx
wordpress url设置
WordPress地址(URL) http://www.mywp.com
- 配置静态web服务器
这个web服务器需要实时同步nmp服务器的WordPress目录,我们这里采用nfs方式
nmp服务器作为nfs服务端,静态web服务器作为nfs客户端,安装过程比较简单,如下命令
[root@localhost wordpress]# yum -y install rpcbind nfs-utils
静态服务器mount网络文件,同样放在/data/wordpress目录下
[root@ceph_deploy ~]# mount -t nfs 10.192.1.167:/data/wordpress /data/wordpress
配置nginx并启动
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
#root /usr/share/nginx/html;
root /data/wordpress;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
index index.html index.htm;
location / {
}
[root@ceph_deploy ~]# systemctl start nginx
编辑一个静态页面
[root@localhost wordpress]# vim index.html
static pag
- 测试
客户端主机设置hosts
10.192.1.163 www.mywp.com # 指向varinish服务器
压力测试压文章
[root@localhost wordpress]# ab -n 1000 -c 100 http://www.mywp.com/?p=10
压力测试压静态页面
[root@localhost wordpress]# ab -n 1000 -c 100 http://www.mywp.com/index.html
查看varnish服务器状态
[root@node1 ~]# varnishstat -1 -l -f MAIN.cache_hit -1 -l -f MAIN.cache_miss
MAIN.cache_hit 5773 0.10 Cache hits
MAIN.cache_miss 95 0.00 Cache misses