Keepalive + Nginx高可用

1. 简介

• 比较常见的高可用，keepalive实现IP的漂移，做到High Available，nginx实现反向代理，做到Load Balance

2. 架构图

• 被代理的可以是任何应用或者数据库，两个VIP都是可以被DNS解析到的地址，如果右侧机器宕机，VIP会飘到左边机器，实现HA

  
  
  

  image.png

3. 版本

• keepalived 1.3
• nginx 1.12

4. 安装与配置

• Keeaplived, base repo里面就有，可以直接安装

yum -y install keepalived
systemctl start keepalived
systemctl enable keepalived

配置文件 on server 1：/etc/keepalived/keepalived.conf

global_defs {
  notification_email {
    [email protected]
  }
  notification_email_from keepalived@node-10-210-149-21
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
}


vrrp_script check_webproxy {
  script "killall -0 nginx"
  interval 1
  weight 21
}

vrrp_script check_mantaince_down {
  script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
  interval 1
  weight 2
}


vrrp_instance vip_10_210_149_23 {
  state MASTER
  interface ens192
  virtual_router_id 23
  garp_master_delay 1
  mcast_src_ip 10.210.149.21
  lvs_sync_daemon_interface ens192
  priority 110
  advert_int 2
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  track_interface {
    ens192
  }
  virtual_ipaddress {
    10.210.149.23/24 dev ens192 label ens192:0
  }
  track_script {
    check_webproxy
    check_mantaince_down
  }
}

vrrp_instance nginx_vip_10_210.149_24 {
  state BACKUP
  interface ens192
  virtual_router_id 24
  garp_master_delay 1
  mcast_src_ip 10.210.149.21
  lvs_sync_daemon_interface ens192
  priority 100
  advert_int 2
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  track_interface {
    ens192
  }
  virtual_ipaddress {
    10.210.149.24/24 dev ens192 label ens192:1
  }
  track_script {
    check_webproxy
    check_mantaince_down
  }
}

配置文件 on server 2：/etc/keepalived/keepalived.conf

global_defs {
  notification_email {
    [email protected]
  }
  notification_email_from keepalived@node-10-210-149-21
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
}


vrrp_script check_webproxy {
  script "killall -0 nginx"
  interval 1
  weight 21
}

vrrp_script check_mantaince_down {
  script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
  interval 1
  weight 2
}


vrrp_instance vip_10_210_149_23 {
  state BACKUP
  interface ens192
  virtual_router_id 23
  garp_master_delay 1
  mcast_src_ip 10.210.149.22
  lvs_sync_daemon_interface ens192
  priority 100
  advert_int 2
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  track_interface {
    ens192
  }
  virtual_ipaddress {
    10.210.149.23/24 dev ens192 label ens192:0
  }
  track_script {
    check_webproxy
    check_mantaince_down
  }
}

vrrp_instance nginx_vip_10_210.149_24 {
  state MASTER
  interface ens192
  virtual_router_id 24
  garp_master_delay 1
  mcast_src_ip 10.210.149.22
  lvs_sync_daemon_interface ens192
  priority 110
  advert_int 2
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  track_interface {
    ens192
  }
  virtual_ipaddress {
    10.210.149.24/24 dev ens192 label ens192:1
  }
  track_script {
    check_webproxy
    check_mantaince_down
  }
}

• DNS记录，在对应的域解析文件下面添加如下的记录，bind服务器会默认使用RR算法平均分配解析请求，如果有特殊配置，配置项在rrset-order设置成cyclic就好

mysql                   IN      A       10.210.149.23
mysql                   IN      A       10.210.149.24

• Nginx

yum -y install nginx
systemctl start nginx
systemctl enable nginx

在/etc/nginx/conf.d/下面建立两个文件夹分别存放http和tcp请求的代理

mkdir /etc/nginx/conf.d/{http.d,tcp.d}

修改/etc/nginx/nginx.conf，分别include一个文件夹存放http和tcp反向代理，每个服务写成一个文件，方便管理，也方便使用consul-template

# HTTP七层反向代理要写在http段
http {
...
include /etc/nginx/conf.d/http.d/*.conf;
}
# TCP四层反向代理要写在stream段
stream {
...
include /etc/nginx/conf.d/tcp.d/*.conf;
}

两个节点的都一样，/etc/nginx/conf.d/tcp.d/mysql.conf

# mysql proxy
server {
    listen 3306;
    proxy_pass mysql;
}

upstream mysql {
    server 10.210.149.25:3306;
    server 10.210.149.26:3306;
    server 10.210.149.27:3306;
}

使用netstat可以看到端口在监听

tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      5502/nginx: master