mycat 集群实现过程

规划

 

haproxy+keepalived： 192.168.121.219

 

haproxy+keepalived：192.168.121.244

 

mycat1： 192.168.121.52

mycat2：192.168.121.238

 

vip地址：192.168.121.103

 

 

1，准备2台mycat 
1.1 安装 192.168.121.52、192.168.121.238
在mycat群共享里面下载Mycat-server-1.5-alpha-20151221110028-linux.tar.gz
安装到/usr/local目录里面去：tar -xvf Mycat-server-1.5-alpha-20151221110028-linux.tar.gz -C /usr/local


1.2，启动：
/usr/local/mycat/bin/mycat start 


1.3，配置监听mycat是否存活
在2台mycat服务器上开启xinetd，启动mycat_status服务，在Mycat server1 Mycat server2上都需要添加检测端口48700的脚本，为此需要用到xinetd，xinetd为linux系统的基础服务，
安装xinetd：

 

yum install xinetd -y

添加服务：

 

[root@oracle_standby soft]# vim /etc/xinetd.d/mycat_status

service mycat_status
{
        flags           = REUSE
        socket_type     = stream
        port            = 48700
        wait            = no
        user            = root
        server          =/usr/local/bin/mycat_status
        log_on_failure  += USERID
        disable         = no
}

脚本/usr/local/bin/mycat_status脚本
mycat_status脚本

 

#!/bin/bash
#/usr/local/bin/mycat_status.sh
# This script checks if a mycat server is healthy running on localhost. It will
# return:
#
# "HTTP/1.x 200 OK\r" (if mycat is running smoothly)
#
# "HTTP/1.x 503 Internal Server Error\r" (else)
mycat=`/usr/local/mycat/bin/mycatstatus | grep'not running' | wc -l`
if [ "$mycat" = "0" ];
then
 /bin/echo-e "HTTP/1.1 200 OK\r\n"
 else
   /bin/echo-e "HTTP/1.1 503 Service Unavailable\r\n"
fi

在/etc/services中加入mycat_status服务
在末尾加入
[root@oracle_standby soft]# vim /etc/services
mycat_status    48700/tcp              # mycat_status
保存
重启xinetd服务
service xinetd restart


2、安装keepalived

 

先安装好openssl

 

 

2.1，安装openssl
yum install -y openssl*
看安装到哪里去了：
[root@cactitest ~]# find / -name ssl
/usr/include/c++/4.4.4/gnu/javax/net/ssl
/usr/include/c++/4.4.4/javax/net/ssl
/usr/include/boost/asio/ssl
/etc/cups/ssl
/etc/ssl

 

[root@cactitest ~]#

 

 

测试openssl是否ok：

 

[root@cactitest ~]# ldd /usr/bin/openssl 
	linux-vdso.so.1 =>  (0x00007fff28f09000)
	libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fe3a6554000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x000000340de00000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x000000340e200000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x000000340ca00000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x000000340d200000)
	libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fe3a616e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000003408200000)
	libz.so.1 => /lib64/libz.so.1 (0x0000003408a00000)
	libc.so.6 => /lib64/libc.so.6 (0x0000003407e00000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x000000340d600000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x000000340ee00000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003409a00000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003408600000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003407600000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003409600000)
[root@cactitest ~]# 
[root@cactitest ~]# which openssl
/usr/bin/openssl
[root@cactitest ~]# openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
[root@cactitest ~]# 

2.2,安装keepalived，
yum install -y keepalived*
[root@cactitest ~]# find / -name keepalived.conf
/etc/keepalived/keepalived.conf
[root@cactitest ~]# 
建检查haproxy是否存活癿脚本 
mkdir /etc/keepalived/scripts 
cd /etc/keepalived/scripts


（1）keepalived.conf配置
在haproxy1上面

 

vim /etc/keepalived/keepalived.conf
#! Configuration Filefor keepalived
vrrp_script chk_http_port {
    script"/etc/keepalived/scripts/check_haproxy.sh"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
   state MASTER              #主上改为Master
   interface eth0            #对外提供服务的网络接口
   virtual_router_id 51      #VRRP组名，两个节点的设置必须一样，以指明各个节点属于同一VRRP组
   priority 150              #数值愈大，优先级越高,172.17.210.84上改为120
   advert_int 1              #同步通知间隔
   authentication {          #包含验证类型和验证密码。类型主要有PASS、AH两种，通常使用的类型为PASS，据说AH使用时有问题
      auth_type PASS
      auth_pass 1111
   }
  
   track_script {
      chk_http_port            #调用脚本check_haproxy.sh检查haproxy是否存活
   }
  
   virtual_ipaddress {      #vip地址，这个ip必须与我们在lvs客户端设定的vip相一致
      192.168.121.103 dev eth0 scope globa
   }
   notify_master/etc/keepalived/scripts/haproxy_master.sh
   notify_backup/etc/keepalived/scripts/haproxy_backup.sh
   notify_fault /etc/keepalived/scripts/haproxy_fault.sh
   notify_stop  /etc/keepalived/scripts/haproxy_stop.sh
}

在haproxy2上面：

 

vim /etc/keepalived/keepalived.conf
#! Configuration Filefor keepalived
vrrp_script chk_http_port {
    script"/etc/keepalived/scripts/check_haproxy.sh"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
   state MASTER              #在主上改为Master
   interface eth1            #对外提供服务的网络接口
   virtual_router_id 51      #VRRP组名，两个节点的设置必须一样，以指明各个节点属于同一VRRP组
   priority 120              #数值愈大，优先级越高,172.17.210.64上改为120
   advert_int 1              #同步通知间隔
   authentication {          #包含验证类型和验证密码。类型主要有PASS、AH两种，通常使用的类型为PASS，据说AH使用时有问题
      auth_type PASS
      auth_pass 1111
   }
  
   track_script {
      chk_http_port            #调用脚本check_haproxy.sh检查haproxy是否存活
   }
  
   virtual_ipaddress {      #vip地址，这个ip必须与我们在lvs客户端设定的vip相一致
      192.168.121.103 dev eth1 scope globa
   }
   notify_master /etc/keepalived/scripts/haproxy_master.sh
   notify_backup /etc/keepalived/scripts/haproxy_backup.sh
   notify_fault /etc/keepalived/scripts/haproxy_fault.sh
   notify_stop  /etc/keepalived/scripts/haproxy_stop.sh
}

（2）check_haproxy.sh 如果没有haproxy进程存在，就启动haproxy，停止keepalived

 

vim /etc/keepalived/scripts/check_haproxy.sh
#!/bin/bash
STARTHAPROXY="/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg"
STOPKEEPALIVED="/etc/init.d/keepalived stop"
LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"
echo "[check_haproxy status]" >> $LOGFILE
A=`ps-C haproxy --no-header |wc-l`
echo "[check_haproxy status]" >> $LOGFILE
date >> $LOGFILE
if [ $A -eq 0 ];then
echo $STARTHAPROXY >> $LOGFILE
$STARTHAPROXY >> $LOGFILE  2>&1
sleep5
fi
if [ `ps -C haproxy --no-header |wc-l` -eq 0 ];then
exit 0
else
exit 1
fi

（3）/etc/keepalived/scripts/haproxy_master.sh(master和slave一样)

 

vim /etc/keepalived/scripts/haproxy_master.sh
#!/bin/bash
STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy-f /usr/local/haproxy/haproxy.cfg`
STOPHAPROXY=`ps-ef | grep sbin/haproxy | grep -vgrep |awk'{print $2}'|xargskill -s 9`
LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
echo "stop haproxy...." >> $LOGFILE 2>&1
$STOPHAPROXY >> $LOGFILE  2>&1
echo "start haproxy...." >> $LOGFILE 2>&1
$STARTHAPROXY >> $LOGFILE  2>&1
echo "haproxy stared ..." >> $LOGFILE

（4）/etc/keepalived/scripts/haproxy_backup.sh(master和slave一样)

 

vim /etc/keepalived/scripts/haproxy_backup.sh
#!/bin/bash
STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy-f /usr/local/haproxy/haproxy.cfg`
STOPHAPROXY=`ps-ef | grep sbin/haproxy | grep -vgrep |awk'{print $2}'|xargskill -s 9`
LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being backup...." >> $LOGFILE 2>&1
echo "stop haproxy...." >> $LOGFILE 2>&1
$STOPHAPROXY >> $LOGFILE  2>&1
echo "start haproxy...." >> $LOGFILE 2>&1
$STARTHAPROXY >> $LOGFILE  2>&1
echo "haproxy stared ..." >> $LOGFILE

（5）haproxy_fault.sh(master和slave一样)

 

vim /etc/keepalived/scripts/haproxy_fault.sh
#!/bin/bash
LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log
echo "[fault]" >> $LOGFILE
date >> $LOGFILE

（6）haproxy_stop.sh(master和slave一样)

 

vim /etc/keepalived/scripts/haproxy_stop.sh
#!/bin/bash
LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log
echo "[stop]" >> $LOGFILE
date >> $LOGFILE

（7）启用服务
service keepalived start


3、准备haproxy
3.1，下载当前比较新的版本
去http://fossies.org/linux/misc/haproxy-1.6.3.tar.gz/网站下载
wget http://fossies.org/linux/misc/haproxy-1.6.3.tar.gz


tar -xvf haproxy-1.6.3.tar.gz
cd haproxy-1.6.3
make TARGET=linux26 PREFIX=/usr/local/haprpxy ARCH=x86_64
make install PREFIX=/usr/local/haproxy
useradd haproxy
chown -R haproxy.haproxy *

其中报错如下，这个问题一时解决不了，只好重做系统才能解决：
[root@oracle_standby1 haproxy-1.6.3]# make TARGET=linux26 PREFIX=/usr/local/haprpxy ARCH=x86_64
gcc -Iinclude -Iebtree -Wall -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement       -DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DNETFILTER -DUSE_GETSOCKNAME  -DCONFIG_HAPROXY_VERSION=\"1.6.3\" -DCONFIG_HAPROXY_DATE=\"2015/12/25\" -c -o src/standard.o src/standard.c
src/standard.c: Assembler messages:
src/standard.c:1342: Error: Incorrect register `%r9' used with `l' suffix
src/standard.c:1372: Error: Incorrect register `%r9' used with `l' suffix
make: *** [src/standard.o] Error 1
[root@oracle_standby1 haproxy-1.6.3]# 

 

 

 

 

3.2，配置haproxy.cfg功能 
 haproxy.cfg
#cd /usr/local/haproxy
#touch haproxy.cfg
#vim /usr/local/haproxy/haproxy.cfg

 

[root@cactitest ~]# more /usr/local/haproxy/haproxy.cfg
global
log 127.0.0.1   local0 ##记日志的功能
    maxconn 4096
    chroot/usr/local/haproxy
    user haproxy
    group haproxy
    daemon
defaults
    log    global
    option    dontlognull
    retries    3
    option redispatch
    maxconn    2000
    contimeout    5000
    clitimeout    50000
    srvtimeout    50000
listen  admin_status 192.168.121.103:48800 ##VIP
      stats uri/admin-status        ##统计页面
      stats auth  admin:admin
      mode    http
      option  httplog
listen    allmycat_service 192.168.121.103:8096 ##转发到mycat的8066端口，即mycat的服务端口
      mode tcp
      option tcplog
        option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
      balance    roundrobin
        server    mycat1 192.168.121.52:8066 check port 48700 inter 5s rise 2 fall 3
        server    mycat2 192.168.121.238:8066 check port 48700 inter 5s rise 2 fall 3
      srvtimeout 20000
listen    allmycat_admin 192.168.121.103:8097 ##转发到mycat的9066端口，及mycat的管理控制台端口
      mode tcp
      option tcplog
        option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
      balance    roundrobin
        server    mycat1 192.168.121.52:9066 check port 48700 inter 5s rise 2 fall 3
        server    mycat2  192.168.121.238:9066 check port 48700 inter 5s rise 2 fall 3
      srvtimeout 20000
[root@cactitest ~]# 

 
 
3.3，记录日志
haproxy记录日志
默认haproxy是不记录日志的，为了记录日志还需要配置syslog模块，在linux下是rsyslogd服务，yum –y install rsyslog先安装rsyslog，然后
yum install -y rsyslog*
cd /etc/rsyslog.d/
vim haproxy.conf
$ModLoad imudp
$UDPServerRun 514
 
local0.* /var/log/haproxy.log


在#### RULES ####上面一行的地方加入以下内容：
$IncludeConfig /etc/rsyslog.d/*.conf
在
local7.*                                               /var/log/boot.log的下面加入以下内容（增加后的效果如下）：
# Save boot messages also to boot.log
local7.*                                               /var/log/boot.log
local0.*
重启
service rsyslog restart
可以看到新日志
[root@cactitest rsyslog.d]# ll /var/log/haproxy.log
-rw-------. 1 root root 0 Mar 13 19:38 /var/log/haproxy.log
[root@cactitest rsyslog.d]# 



3.4 启动haproxy 
启动haproxy1、haproxy2，启动haproxy前必须先启动keepalived，否则启动不了。
[root@cactitest soft]# /etc/rc.d/init.d/keepalived restart
Starting keepalived:                                       [  OK  ]
[root@cactitest soft]# 


/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg
启动完，可以去网站查看信息：

 

http://192.168.121.103:48800/admin-status

 

 

启动有报错，报错信息如下：

 

[root@mall_hc_244 rsyslog.d]# /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:4] : unknown keyword 'chroot/usr/local/haproxy' in 'global' section
[WARNING] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:14] : the 'contimeout' directive is now deprecated in favor of 'timeout connect', and will not be supported in future versions.
[WARNING] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:15] : the 'clitimeout' directive is now deprecated in favor of 'timeout client', and will not be supported in future versions.
[WARNING] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:16] : the 'srvtimeout' directive is now deprecated in favor of 'timeout server', and will not be supported in future versions.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:17] : 'listen' cannot handle unexpected argument '192.168.121.103:48800'.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:17] : please use the 'bind' keyword for listening addresses.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:18]: unknown stats parameter 'uri/admin-status', expects 'admin', 'uri', 'realm', 'auth', 'scope', 'enable', 'hide-version', 'show-node', 'show-desc' or 'show-legends'.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:22] : 'listen' cannot handle unexpected argument '192.168.121.103:8096'.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:22] : please use the 'bind' keyword for listening addresses.
[WARNING] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:29] : overwriting 'timeout server' which was already specified
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:30] : 'listen' cannot handle unexpected argument '192.168.121.103:8097'.
[ALERT] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:30] : please use the 'bind' keyword for listening addresses.
[WARNING] 073/084322 (1715) : parsing [/usr/local/haproxy/haproxy.cfg:37] : overwriting 'timeout server' which was already specified
[ALERT] 073/084322 (1715) : Error(s) found in configuration file : /usr/local/haproxy/haproxy.cfg
[WARNING] 073/084322 (1715) : config : proxy 'admin_status' has no 'bind' directive. Please declare it as a backend if this was intended.
[WARNING] 073/084322 (1715) : config : missing timeouts for proxy 'admin_status'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 073/084322 (1715) : config : log format ignored for proxy 'admin_status' since it has no log address.
[WARNING] 073/084322 (1715) : config : proxy 'allmycat_service' has no 'bind' directive. Please declare it as a backend if this was intended.
[WARNING] 073/084322 (1715) : config : missing timeouts for proxy 'allmycat_service'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 073/084322 (1715) : config : log format ignored for proxy 'allmycat_service' since it has no log address.
[WARNING] 073/084322 (1715) : config : proxy 'allmycat_admin' has no 'bind' directive. Please declare it as a backend if this was intended.
[WARNING] 073/084322 (1715) : config : missing timeouts for proxy 'allmycat_admin'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 073/084322 (1715) : config : log format ignored for proxy 'allmycat_admin' since it has no log address.
[ALERT] 073/084322 (1715) : Fatal errors found in configuration.
[root@mall_hc_244 rsyslog.d]# 

后续继续调试这个问题