HAProxy and HA Solution(1)Setup KeepAlived on Ubuntu

阅读更多

HAProxy and HA Solution(1)Setup KeepAlived on Ubuntu

Find the right Version for KeepAlived from here
https://www.keepalived.org/download.html

> wget https://www.keepalived.org/software/keepalived-2.0.17.tar.gz

Unzip and build/install
> tar zxvf keepalived-2.0.17.tar.gz
> ./configure --prefix=/home/carl/tool/keepalived-2.0.17

I get console message about this
*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

I do not think I need IPv6
> make
> sudo make install

Copy and prepare related files
> sudo ln -s /home/carl/tool/keepalived-2.0.17 /opt/keepalived-2.0.17
> sudo ln -s /opt/keepalived-2.0.17 /opt/keepalived

It seems not that easy to set up in service init.d
HAProxy author prefer keepalived.
http://www.voidcn.com/article/p-essgxbcx-bka.html

Directly install from ubuntu APT
> sudo apt-get install keepalived
> sudo snap install keepalived --classic

Check Version
> keepalived -version
Keepalived v1.3.9 (10/21,2017)
Copyright(C) 2001-2017 Alexandre Cassen,

> keepalived -version
Keepalived v2.0.10 (11/12,2018)

Need to Make sure the 2 machines are having the same version of KeepAlived

Finally, I get the 2 machine with the same version
> keepalived -version
Keepalived v2.0.17 (06/27,2019), git commit v2.0.17-8-g836230e+

> keepalived -version
Keepalived v2.0.17 (06/27,2019), git commit v2.0.17-8-g836230e+

Change the Setting
> sudo vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1

Check the Settings
> sudo sysctl -p
vm.max_map_count = 262144
net.ipv4.ip_nonlocal_bind = 1

Prepare the configuration on the master machine
> cat /etc/keepalived/keepalived.conf
global_defs {
  lvs_id haproxy_DH
}
vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
vrrp_instance VI_01 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 101
  virtual_ipaddress {
    192.168.56.110
  }
  track_script {
    check_haproxy
  }
}

Create the configuration on the Slave Machine
> cat /etc/keepalived/keepalived.conf
global_defs {
  lvs_id haproxy_DH_passive
}
vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}
vrrp_instance VI_01 {
  state SLAVE
  interface eth0
  virtual_router_id 51
  priority 100
  virtual_ipaddress {
    192.168.56.110
  }
  track_script {
    check_haproxy
  }
}

Suppose this command can start the service
> sudo service keepalived start

But I received these information
> sudo service keepalived start
Failed to start keepalived.service: Unit keepalived.service is masked.

> sudo service keepalived status
● keepalived.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)

> sudo service keepalived status
● keepalived.service
   Loaded: masked (Reason: Unit keepalived.service is masked.)
   Active: inactive (dead)

Solution:
https://serverfault.com/questions/678692/unable-to-start-keepalived
http://gayangunarathne.blogspot.com/2015/06/lvs-setup-in-oracle-virtualbox.html

Set Up HAProxy
Find the latest version from here
http://www.haproxy.org/#down

> wget http://www.haproxy.org/download/1.8/src/haproxy-1.8.20.tar.gz
> tar zxvf haproxy-1.8.20.tar.gz

Prepare
> sudo apt-get install libpcre3 libpcre3-dev

> make TARGET=generic ARCH=x86_64 USE_PCRE=1
> make install DESTDIR='/home/carl/tool/haproxy-1.8.20' PREFIX=''


> sudo ln -s /home/carl/tool/haproxy-1.8.20 /opt/haproxy-1.8.20
> sudo ln -s /opt/haproxy-1.8.20 /opt/haproxy

Add sbin to the PATH
export PATH="/opt/haproxy/sbin:$PATH"

Check the installation and version
> haproxy -v
HA-Proxy version 1.8.20 2019/04/25
Copyright 2000-2019 Willy Tarreau

Make sure we have the haproxy Settings right
> mkdir conf

Sample configuration is as follow:
> cat conf/haproxy.conf
global
        log 127.0.0.1 local0 info
        maxconn 5120
        chroot /opt/haproxy
        uid 99
        gid 99
        daemon
        quiet
        nbproc  2
        pidfile /opt/haproxy/haproxy.pid
defaults
       log        global
       mode       tcp
       option     tcplog
       option     dontlognull
       retries    3
       option redispatch
       maxconn 2000
       contimeout      5s
       clitimeout      120s
       srvtimeout      120s
      
listen rabbitmq_local_cluster
       bind 0.0.0.0:5670
       mode      tcp
       balance roundrobin
       server rabbit1 ubuntu-master:5672 check inter 5000 rise 2 fall 2
       server rabbit2 ubuntu-dev2:5672 check inter 5000 rise 2 fall 2
listen monitor
        bind 0.0.0.0:8100
        mode http
        option httplog
        stats enable
        stats uri /stats
        stats refresh 5s

Start RabbitMQ Cluster, Start HAProxy on Both Server
> sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 sbin/rabbitmq-server -detached

> sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit2 sbin/rabbitmq-server -detached

We can visit the UI to make sure it is running
http://ubuntu-master:15672/#/
http://ubuntu-dev2:15672/#/

Start HAProxy on Both Server
>sudo sbin/haproxy -f conf/haproxy.conf

Visit the UI to verify
http://ubuntu-master:8100/stats
http://ubuntu-dev2:8100/stats

Check the HAProxy
> pidof haproxy
2469 2468

Check IP Address
> ip addr

The latest Configuration, still not working
> cat /etc/keepalived/keepalived.conf
global_defs {
  lvs_id haproxy_DH
}
vrrp_script check_haproxy {
  script "pidof haproxy"
  interval 2
  weight 2
}
vrrp_instance VI_01 {
  state MASTER
  interface enp0s8
  virtual_router_id 51
  priority 101
  unicast_peer {
    192.168.56.3
  }
  virtual_ipaddress {
    192.168.56.110/27 label enp0s8:10
  }
  track_script {
    check_haproxy
  }
}

Try the older version
https://www.jianshu.com/p/dd93bc6d45f5

> wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
> tar zxvf keepalived-1.2.2.tar.gz

Try to compile
> ./configure --prefix=/usr/local/keepalived

Exception:
configure: error: Popt libraries is required

Solution:
> sudo apt-get install libpopt-dev
> ./configure --prefix=/usr/local/keepalived

> make
> make install

Prepare the files
> sudo cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
> sudo cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
> sudo mkdir /etc/sysconfig
> sudo cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
> sudo mkdir -p /etc/keepalived/
> sudo cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

I guess this is too old to use on the latest Ubuntu System
> /etc/init.d/keepalived start
/etc/init.d/keepalived: 12: .: Can't open /etc/rc.d/init.d/functions

Does not work based on these information, plan to try again tomorrow.
https://www.cnblogs.com/xxoome/p/8621677.html
https://blog.csdn.net/bbwangj/article/details/80346428

Checked some docs from systemctl, I think I get it right, I will try again here.
Start RabbitMQ both
> sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 sbin/rabbitmq-server -detached

> sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit2 sbin/rabbitmq-server -detached

Start HAProxy on both machines
> sudo sbin/haproxy -f conf/haproxy.conf

Visit the HAProxy Page
http://ubuntu-master:8100/stats
http://ubuntu-dev2:8100/stats

Finally, with these configuration, I get the keepalived working fine.
> cat /etc/keepalived/keepalived.conf
global_defs {
  lvs_id haproxy_DH
}
vrrp_script check_haproxy {
  script "pidof haproxy"
  interval 2
  weight 2
}
vrrp_instance VI_01 {
  state MASTER
  interface enp0s8
  virtual_router_id 51
  priority 110
  unicast_peer {
    192.168.56.106
  }
  virtual_ipaddress {
    192.168.56.110/27 label enp0s8:10
  }
  track_script {
    check_haproxy
  }
}

> cat /etc/keepalived/keepalived.conf
global_defs {
  lvs_id haproxy_DH_passive
}
vrrp_script check_haproxy {
  script "pidof haproxy"
  interval 2
  weight 2
}
vrrp_instance VI_01 {
  state BACKUP
  interface enp0s8
  virtual_router_id 51
  priority 100
  unicast_peer {
    192.168.56.101
  }
  virtual_ipaddress {
    192.168.56.110/27 label enp0s8:10
  }
  track_script {
    check_haproxy
  }
}


We can see the IP working on the master
enp0s8:10: flags=4163  mtu 1500
        inet 192.168.56.110  netmask 255.255.255.224  broadcast 0.0.0.0
        ether 08:00:27:7d:a0:7e  txqueuelen 1000  (Ethernet)

Visit this page, it will give us the status for HAproxy
http://192.168.56.110:8100/stats

Kill the KeepAlived on ubuntu-master

The floating IP will be on the ubuntu-dev2

References:
https://www.lisenet.com/2015/setting-up-a-load-balancing-haproxy-cluster-with-keepalived/
https://www.jianshu.com/p/dd93bc6d45f5
https://docs.oracle.com/cd/E37670_01/E41138/html/section_sm3_svy_4r.html
https://dasunhegoda.com/how-to-setup-haproxy-with-keepalived/833/
https://vexxhost.com/resources/tutorials/highly-available-web-servers-keepalived-floating-ips-ubuntu-16-04/