Codis集群部署架构

Codis安装部署全架构

 

 

 

前期规划

 

机器与应用列表：

操作系统：CentOS6.5

 

1，操作系统：CentOS6.5

192.168.88.106    codis-server1

192.168.88.107    codis-server2

192.168.88.108    codis-server3

192.168.88.111    codis-ha1

192.168.88.112    codis-ha2

192.168.88.113  zookeeper-1（codis-proxy-1）

192.168.88.114  zookeeper-2（codis-proxy-2）

192.168.88.115  zookeeper-3（codis-proxy-3）

2，硬件配置：

ha              mem：8G   cpu：4   disk：100G

zookeeper     mem：16G  cpu：8   disk：300G

codis-server mem：16G  cpu：8   disk：200G

 

3，架构应用

1）， HA（192.168.88.111、192.168.88.112、VIP 192.168.88.159）

hostname:codisha-1   apps：keepalived master,haproxy                      prots:19000

hostname:codisha-2 apps：keepalived slave ,haproxy，codis-config        prots:19000,18087

 

2），zookeeper+codis-proxy

hostname:zookeeper-1     apps: zookeeper1, codis_proxy_1          prots:2811,19000

hostname:zookeeper-2     apps: zookeeper2, codis_proxy_2          prots:2811,19000

hostname:zookeeper-3     apps: zookeeper3, codis_proxy_3          prots:2811,19000

 

3），codis-server

codis-server(192.168.88.106、192.168.88.107、192.168.88.108)

hostname: codis-server1    apps: codis_server_master,slave,        ports:6379,6380，6389,6390

hostname: codis-server2    apps: codis_server_master,slave         ports:6379,6380，6389,6390

hostname: codis-server3    apps: codis_server_master,slave         ports:6379,6380，6389,6390

 

 

 

部署安装详细步骤

 

一，安装zookeeper

 

1，配置hosts文件 (所有机器上配置)

vim /etc/hosts

192.168.88.106    codis-server1

192.168.88.107    codis-server2

192.168.88.108    codis-server3

192.168.88.111    codis-ha1

192.168.88.112    codis-ha2

192.168.88.113  zookeeper-1(codis-proxy-1)

192.168.88.114  zookeeper-2(codis-proxy-2)

192.168.88.115  zookeeper-3(codis-proxy-3)

 

 

2，安装java环境

ZooKeeper 要求 JAVA 的环境才能运行，并且需要 JAVA6 以上的版本，可以从 SUN 官网上下载，并对 JAVA 环境变量进行设置。

yum -y install java-1.7.0-openjdk-devel

java -version

java version "1.7.0_75"

OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)

OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)

 

 

3，安装zookeeper

wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

tar zxvf zookeeper-3.4.6.tar.gz

mv zookeeper-3.4.6 /usr/local/zookeeper

mkdir -p /data/zookeeper/{data,logs}

配置zoo.cfg

vim /usr/local/zookeeper/conf/zoo.cfg 

tickTime=2000

initLimit=5

syncLimit=2

dataDir=/data/zookeeper/data

#dataLogDir=/data/zookeeper/logs

clientPort=2181

server.1=zookeeper-1:2888:3888

server.2=zookeeper-2:2888:3888

server.3=zookeeper-3:2888:3888

上述配置内容说明，可以参考http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

 

 

4，设置myid

在我们配置的dataDir指定的目录下面，创建一个myid文件，里面内容为一个数字，用来标识当前主机，conf/zoo.cfg文件中配置的server.X中X为什么数字，则myid文件中就输入这个数字

[root@zookeeper-1 ~]# echo 1 > /data/zookeeper/data/myid

[root@zookeeper-2 ~]# echo 2 > /data/zookeeper/data/myid

[root@zookeeper-3 ~]# echo 3 > /data/zookeeper/data/myid

 

 

5，启动zookeeper

启动顺序zookeeper-1>zookeeper-2>zookeeper-3

[root@zookeeper-1 zookeeper]# zkServer.sh start  

JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@zookeeper-1 zookeeper]# zkServer.sh status

JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Mode: leader

可以看到最先开始启动的是leader，其他两个是follower

 

设置开机启动

vim /etc/rc.local

/usr/local/zookeeper/bin/zkServer.sh start

 

设置环境变量

vim /etc/profile

export ZOOKEEPERPATH=/usr/local/zookeeper

export GOROOT=/usr/local/go

export CODISPATH=/usr/local/codis

export PATH=$PATH:$GOROOT/bin:$ZOOKEEPERPATH/bin:$CODISPATH/bin

 

source /etc/profile

 

二，安装codis集群

 

 

1，安装go

设置环境变量

vim /etc/profile

export GOROOT=/usr/local/go

export CODISPATH=/usr/local/codis

export PATH=$PATH:$GOROOT/bin:$CODISPATH/bin

source /etc/profile

下载安装go

cd /usr/local/

 

wget http://golangtc.com/static/go/go1.3.3.linux-amd64.tar.gz

 

tar -zxvf go1.3.3.linux-amd64.tar.gz                                                                             

go version

go version go1.3.3 linux/amd64

 

 

2，安装依赖环境

yum groupinstall "Development Tools"

 

 

3，安装codis

yum install -y git

go get github.com/wandoulabs/codis  #这个需要几分钟下载共30M文件

package github.com/wandoulabs/codis

imports github.com/wandoulabs/codis

imports github.com/wandoulabs/codis: no buildable Go source files in /usr/local/codis/src/github.com/wandoulabs/codis

cd $GOPATH/src/github.com/wandoulabs/codis

[root@localhost codis]# pwd

/usr/local/codis/src/github.com/wandoulabs/codis

#执行编译测试脚本，编译go和reids。 

./bootstrap.sh #这个需要十几分钟共下载50M文件

make gotest

# 将编译好后，把bin目录和一些脚本复制过去/usr/local/codis目录下：

mkdir -p /usr/local/codis/{logs,conf,scripts}

mkdir -p /data/codis_server/{logs,conf,data}

cp -rf bin /usr/local/codis/

cp sample/config.ini /usr/local/codis/conf/

cp sample/redis_conf/6381.conf /data/codis_server/conf/

cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/*.sh /usr/local/codis/scripts/

cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/usage.md /usr/local/codis/scripts/

cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.13

cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.21

ln -s /usr/local/codis/bin/redis-cli-2.8.21 /usr/local/codis/bin/redis-cli

 

 

4. 配置codis_proxy  ( zookeeper-1、zookeeper-2、zookeeper-3 机器上配置)

 

配置codis_proxy_1 ( zookeeper-1 机器上配置)

cd /usr/local/codis/conf

vim config.ini 

zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

product=jerrymin-codis

proxy_id=codis_proxy_1

net_timeout=50

dashboard_addr=192.168.88.112:18087

coordinator=zookeeper

 

配置codis_proxy_2 ( zookeeper-1 机器上配置)

cd /usr/local/codis/conf

vim config.ini 

zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

product=jerrymin-codis

proxy_id=codis_proxy_2

net_timeout=50

dashboard_addr=192.168.88.112:18087

coordinator=zookeeper

 

配置codis_proxy_3 ( zookeeper-1 机器上配置)

cd /usr/local/codis/conf

vim config.ini 

zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

product=jerrymin-codis

proxy_id=codis_proxy_3

net_timeout=50

dashboard_addr=192.168.88.112:18087

coordinator=zookeeper

 

 

5. 修改配置文件，启动codis-server服务（codis-server1、codis-server2、codis-server3 机器上配置）

cd /data/codis_server/conf/

grep -Ev "^#|^$" 6379.conf.bak >6379.conf

vim 6379.conf

修改如下参数: (生产环境，参数适当进行调整)

daemonize yes

timeout 300

pidfile /var/run/redis_6379.pid

port 6379

logfile "/data/codis_server/logs/codis_6379.log"

save 900 1

save 300 10

save 60 10000

dbfilename 6379.rdb

dir /data/codis_server/data

appendfilename "6379_appendonly.aof"

appendfsync everysec

 

具体配置文件如下：

daemonize yes

pidfile /var/run/redis_6379.pid

port 6379

tcp-backlog 511

timeout 300

tcp-keepalive 0

loglevel notice

logfile "/data/codis_server/logs/redis_6379.log"

databases 16

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename 6379.rdb

dir /data/codis_server/data

slave-serve-stale-data yes

slave-read-only yes

repl-disable-tcp-nodelay no

slave-priority 100

maxclients 10000

maxmemory 3gb

maxmemory-policy allkeys-lru

appendonly no

appendfilename "6379_appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

 

 

 

复制6380、6389、6390配置文件

cp 6379.conf 6380.conf

cp 6379.conf 6389.conf

cp 6379.conf 6390.conf

sed -i 's/6379/6380/g' 6380.conf

sed -i 's/6379/6389/g' 6380.conf

sed -i 's/6379/6390/g' 6380.conf

 

添加内核参数

echo "vm.overcommit_memory = 1" >>  /etc/sysctl.conf

sysctl -p

内核参数说明如下：                                                                                                                                       

overcommit_memory文件指定了内核针对内存分配的策略，其值可以是0、1、2。                               

0， 表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。 

1， 表示内核允许分配所有的物理内存，而不管当前的内存状态如何。

2， 表示内核允许分配超过所有物理内存和交换空间总和的内存

 

启动codis-server服务

/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf 

/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf 

/usr/local/codis/bin/codis-server /data/codis_server/conf/6389.conf 

/usr/local/codis/bin/codis-server /data/codis_server/conf/6390.conf 

[root@codis-server1 ~]# /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf 

[root@codis-server1 ~]# /usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf 

[root@codis-server1 ~]# /usr/local/codis/bin/codis-server /data/codis_server/conf/6389.conf 

[root@codis-server1 ~]# /usr/local/codis/bin/codis-server /data/codis_server/conf/6390.conf 

[root@codis-server1 ~]# ps aux |grep codis

root      7473  0.0  0.0 137388  9540 ?        Ssl  09:48   0:00 /usr/local/codis/bin/codis-server *:6379                           

root      7478  0.0  0.0 137388  9524 ?        Ssl  09:48   0:00 /usr/local/codis/bin/codis-server *:6380                           

root      7482  0.0  0.0 137388  9516 ?        Ssl  09:48   0:00 /usr/local/codis/bin/codis-server *:6389                           

root      7486  0.0  0.0 137388  9524 ?        Ssl  09:48   0:00 /usr/local/codis/bin/codis-server *:6390                           

root      7490  0.0  0.0 103252   856 pts/0    S+   09:49   0:00 grep --color=auto codis

[root@codis-server1 ~]# netstat -tulpn|grep codis

tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      7473/codis-server * 

tcp        0      0 0.0.0.0:6380                0.0.0.0:*                   LISTEN      7478/codis-server * 

tcp        0      0 0.0.0.0:6389                0.0.0.0:*                   LISTEN      7482/codis-server * 

tcp        0      0 0.0.0.0:6390                0.0.0.0:*                   LISTEN      7486/codis-server * 

tcp        0      0 :::6379                     :::*                        LISTEN      7473/codis-server * 

tcp        0      0 :::6380                     :::*                        LISTEN      7478/codis-server * 

tcp        0      0 :::6389                     :::*                        LISTEN      7482/codis-server * 

tcp        0      0 :::6390                     :::*                        LISTEN      7486/codis-server * 

 

 

 

6. 查看一下启动流程：

cat /usr/local/codis/scripts/usage.md

0. start zookeeper       //启动zookeeper服务

1. change config items in config.ini  //修改codis配置文件

2. ./start_dashboard.sh    //启动 dashboard

3. ./start_redis.sh        //启动redis实例

4. ./add_group.sh          //添加redis组，一个redis组只能有一个master

5. ./initslot.sh          //初始化槽

6. ./start_proxy.sh        //启动proxy

7. ./set_proxy_online.sh    //上线proxy项目

8. open browser to http://localhost:18087/admin     //访问web

这只是一个参考，有些顺序不是必须的，但启动dashboard前，必须启动zookeeper服务，这是必须的，后面有很多操作，都可以在web页面完成，例如添加/删除组，添加/删除redis实例等

 

 

7. 修改脚本，启动 dashboard。( 只需在一台机器上启动即可。codis-ha2上启动 ，后续大部分操作都可以在面板上操作)

cat /usr/local/codis/scripts/start_dashboard.sh

#!/bin/sh

CODIS_HOME=/usr/local/codis 

 

nohup $CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/dashboard.log dashboard --addr=:18087 --http-log=$CODIS_HOME/logs/requests.log &>/dev/null &

 

启动dashboard

[root@codis-ha2 scripts]# ls -lh start_dashboard.sh 

-rwxr-xr-x 1 root root 218 Jun 24 22:04 start_dashboard.sh

[root@codis-ha2 scripts]# ./start_dashboard.sh 

[root@codis-ha2 scripts]# ps aux |grep codis-config

root      2435  0.0  0.1 216444 11044 pts/1    Sl   10:06   0:00 /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini -L /usr/local/codis/logs/dashboard.log dashboard --addr=:18087 --http-log=/usr/local/codis/logs/requests.log

root      2441  0.0  0.0 103252   840 pts/1    S+   10:06   0:00 grep --color=auto codis-config

[root@codis-ha2 scripts]# netstat -tulpn |grep codis

tcp        0      0 :::10086                    :::*                        LISTEN      2435/codis-config   

tcp        0      0 :::18087                    :::*                        LISTEN      2435/codis-config   

 

访问dashboard

http://192.168.88.112:18087/admin/

 

 

8. 添加redis组 

通过管理页面添加组ID，为组添加主从实例，一个组里只能有一个redis-master：

http://192.168.88.112:18087/admin/（最好用Firefox浏览器或者谷歌浏览器）

登录http://192.168.88.112:18087/admin/,添加2个组，组里面有2个实例，一个主一个从，默认每个组里面第一个实例是主

group_1

192.168.88.106:6379   master

192.168.88.107:6380   slave

group_2

192.168.88.106:6389   master

192.168.88.108:6390   slave

 

group_3

192.168.88.107:6379   master

192.168.88.106:6380   slave

group_4

192.168.88.107:6389   master

192.168.88.108:6380   slave

 

group_5

192.168.88.108:6379   master

192.168.88.106:6390   slave

group_6

192.168.88.108:6389   master

192.168.88.107:6390   slave

 

9. 修改脚本，初始化槽 ( 在codis-ha2机器上配置，初始化solt是在group设置好之后 )

[root@codis-ha2 scripts]# cat initslot.sh 

#!/bin/sh

CODIS_HOME=/usr/local/codis

echo "slots initializing..."

$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini slot init -f

echo "done"

echo "set slot ranges to server groups..."

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 0 170 1 online

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 171 341 2 online

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 342 512 3 online

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 513 683 4 online

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 684 853 5 online

$CODIS_HOME/bin/codis-config -c  config.ini slot range-set 854 1023 6 online

echo "done"

 

 

10.测试一下redis-master和redis-slave是否正常同步数据了：

这个需要用redis-cli测试：

[root@codis-server1 ~]# redis-cli -h 192.168.88.106 -p 6379

192.168.88.106:6379> ping

PONG

192.168.88.106:6379> set name jerrymin

OK

192.168.88.106:6379> get name

"jerrymin"

192.168.88.106:6379> quit

[root@codis-server1 ~]# redis-cli -h 192.168.88.107 -p 6380

192.168.88.107:6380> get name

"jerrymin"

192.168.88.107:6380> set name foo

(error) READONLY You can't write against a read only slave.

192.168.88.107:6380> quit

 

 

11. 修改start_proxy.sh，启动codis-proxy服务 ( 在zookeeper-1、zookeeper-2、zookeeper-3上配置)

 

zookeeper-1上（其他上面就是codis_proxy_2、codis_proxy_3）

 

查看start_proxy脚本

[root@zookeeper-1 scripts]# cat start_proxy.sh 

#!/bin/sh

 

CODIS_HOME=/usr/local/codis

 

echo "shut down codis_proxy_1..."

$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy offline codis_proxy_1

echo "done"

 

echo "start new codis_proxy_1..."

nohup $CODIS_HOME/bin/codis-proxy --log-level info -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/codis_proxy_1.log  --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &

echo "done"

 

echo "sleep 3s"

sleep 3

tail -n 30 $CODIS_HOME/logs/codis_proxy_1.log.0

 

查看proxy_online脚本

[root@zookeeper-1 scripts]# cat set_proxy_online.sh 

#!/bin/sh

 

CODIS_HOME=/usr/local/codis

echo "set codis_proxy_1 online"

$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy online codis_proxy_1

echo "done"

 

启动proxy

[root@zookeeper-1 scripts]# ./start_proxy.sh 

shut down codis_proxy_1...

{

  "msg": "OK",

  "ret": 0

}

done

start new codis_proxy_1...

done

sleep 3s

nohup: appending output to `nohup.out'

2015/07/24 11:06:13 [INFO] set log level to %!s(log.LogLevel=7)

2015/07/24 11:06:13 [INFO] running on 0.0.0.0:19000

2015/07/24 11:06:13 [INFO] start proxy with config: &{proxyId:codis_proxy_1 productName:jerrymin-codis zkAddr:zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 fact:<nil> netTimeout:50 proto:tcp provider:zookeeper}

2015/07/24 11:06:13 [INFO] proxy info = {Id:codis_proxy_1 Addr:zookeeper-1:19000 LastEvent: LastEventTs:0 State:offline Description: DebugVarAddr:zookeeper-1:11000 Pid:8097 StartAt:2015-07-24 11:06:13.4791833 +0800 CST}

2015/07/24 11:06:13 [WARN] wait to be online: codis_proxy_1

 

上线proxy

[root@zookeeper-1 scripts]# ./set_proxy_online.sh 

set codis_proxy_1 online

{

  "msg": "OK",

  "ret": 0

}

done

 

 

 

三，配置HA

 

1，codis-ha1和codis-ha2上操作

安装keeplived和ipvsadm

设置VIP 192.168.88.159

codis-ha1  为LVS主

codis-ha2  为LVS备

如下是备的配置，主的配置与此类似

[root@codis-ha2 haproxy]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

     local@localhost

   }

   notification_email_from localhost@localhost

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL_112

}

vrrp_script chk_haproxy_port {

   script "/etc/keepalived/check_haproxy.sh"

   interval 2

   weight 2

}

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 159

    priority 50

    advert_int 3

    authentication {

        auth_type PASS

        auth_pass jerrymin

    }

    virtual_ipaddress {

        192.168.88.159

    }

    track_script {

      chk_haproxy_port

    }

}

检测脚本，保证keepalived切换成主时haproxy在工作

[root@codis-ha2 keepalived]# cat check_haproxy.sh 

#!/bin/bash

A=`ps -C haproxy --no-header |wc -l`

if [ $A -eq 0 ];then

/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

sleep 3

if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then

/etc/init.d/keepalived stop

fi

fi

 

 

2，安装配置haproxy

[root@codis-ha1 ~]#tar zxvf  haproxy-1.5.11.tar.gz 

[root@codis-ha1 ~]#cd  haproxy-1.5.11

[root@codis-ha1 haproxy-1.5.11]#make TARGET=linux26 PREFIX=/usr/local/haproxy 

[root@codis-ha1 haproxy-1.5.11]#make install PREFIX=/usr/local/haproxy

 

3，配置haproxy.cfg

global 

        maxconn 40000 

daemon

        user root

        group root 

        nbproc 4 

        log 127.0.0.1 local3 

        spread-checks 2 

defaults 

         timeout server  3s 

         timeout connect 3s 

         timeout client  60s 

         timeout http-request 3s 

         timeout queue   3s

frontend codis-proxy

        bind :19000 

        default_backend codis-proxy-19000

 

frontend web_haproxy_status 

        bind :8080 

        default_backend web_status

 

 

backend codis-proxy-19000

        mode    tcp 

        option  tcpka 

        balance roundrobin 

        server  msvr1 192.168.88.113:19000 check   inter 1s rise 5 fall 1 

        server  msvr2 192.168.88.114:19000 check   inter 1s rise 5 fall 1 

        server  msvr3 192.168.88.115:19000 check   inter 1s rise 5 fall 1 

timeout server   9s 

 

backend  web_status 

         mode http 

         stats enable 

         stats refresh 5s 

         stats uri /status 

         stats realm Haproxy\ statistics 

         stats auth jerrymin:jerrymin@2015

 

4.启动服务

启动服务：

[root@codis-ha1 haproxy]# /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

[root@codis-ha1 haproxy]# ps aux |grep haproxy

root      3147  0.0  0.0  18668  3020 ?        Ss   14:42   0:00 /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

root      3148  0.0  0.0  18668  2888 ?        Ss   14:42   0:00 /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

root      3149  0.0  0.0  18668  2888 ?        Ss   14:42   0:00 /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

root      3150  0.0  0.0  18668  3020 ?        Ss   14:42   0:00 /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

[root@codis-ha1 haproxy]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.88.159:19000 wrr

  -> 192.168.88.111:19000           Local   1      0          0