corosync+pacemaker+drbd实现web服务高可用
一：实验环境
| 节点 |
OS | IP | DRBD_IP | DRBD用硬盘 |
VIP |
| web1 | centos 5.10 | 192.168.10.11 | 172.16.1.1 | /dev/sdb | 192.168.10.100 |
| web2 | centos 5.10 | 192.168.10.12 | 172.16.1.2 | /dev/sdb |
注：
1.其中两节点IP地址已按上图设置好
2.两节点已配置相互ssh信任，并已做了时间同步
二：安装相关软件（节点1和2都安装）
1.安装corosync、pacemaker
到下面的地址下载对应的rpm包
http://clusterlabs.org/rpm/epel-5/x86_64/
本次下载的rpm包为：
cluster-glue-1.0.6-1.6.el5.x86_64.rpm
cluster-glue-libs-1.0.6-1.6.el5.x86_64.rpm
corosync-1.2.7-1.1.el5.x86_64.rpm
corosynclib-1.2.7-1.1.el5.x86_64.rpm
heartbeat-3.0.3-2.3.el5.x86_64.rpm
heartbeat-libs-3.0.3-2.3.el5.x86_64.rpm
libesmtp-1.0.4-5.el5.x86_64.rpm //这个包在epel源中下载
pacemaker-1.0.12-1.el5.centos.x86_64.rpm
pacemaker-libs-1.0.12-1.el5.centos.x86_64.rpm
resource-agents-1.0.4-1.1.el5.x86_64.rpm
安装：
[root@web1 ~]# for i in 1 2; do ssh web$i yum -y --nogpgcheck localinstall /root/*.rpm; done
3.安装apache并设置不随机启动
[root@web1 ~]# for i in 1 2; do ssh web$i yum -y install httpd; done
[root@web1 ~]# for i in 1 2; do ssh web$i chkconfig httpd off; done
4.安装drbd并设置不随机启动
[root@web1 ~]# for i in 1 2; do ssh web$i yum -y install drbd83 kmod-drbd83; done
[root@web1 ~]# for i in 1 2; do ssh web$i chkconfig drbd off; done
三：配置drbd
1. [root@web1 ~]# cp /usr/share/doc/drbd83-8.3.15/drbd.conf /etc/drbd.conf
2. [root@web1 ~]# cd /etc/drbd.d/
3.[root@web1 drbd.d]# cat global_common.conf //这里只列出更改部分，默认未列出
global {
usage-count no;
}
common {
protocol C;
net {
cram-hmac-alg sha1;
shared-secret"wjcaiyf";
}
syncer {
rate 56M;
}
}
4.新建资源r0
[root@web1 drbd.d]# touch r0.res
[root@web1 drbd.d]# cat r0.res
resource r0 {
device /dev/drbd0;
disk /dev/sdb;
meta-disk internal;
on web1 {
address 172.16.1.1:7789;
}
on web2 {
address 172.16.1.2:7789;
}
}
5.复制global_common.conf 和r0.res到web2对应目录下
[root@web1 drbd.d]# scp global_common.conf r0.res web2:/etc/drbd.d/
6.为资源r0创建元数据（节点1和2都创建）
[root@web1 ~]# for i in 1 2; do ssh web$i drbdadm create-md r0; done
7.启动drbd服务
[root@web1 ~]# /etc/init.d/drbd start
[root@web2 ~]# /etc/init.d/drbd start
8.设置web1为主节点开始同步数据
[root@web1 ~]# drbdadm -- --overwrite-data-of-peer primary r0
9.查看/proc/drbd以确定同步状态
[root@web1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash:0ce4d235fc02b5c53c1c52c53433d11a694eab8cbuild by [email protected], 2013-03-27 16:01:26
0:cs:Connected ro:Primary/Secondaryds:UpToDate/UpToDate C r-----
ns:4 nr:8 dw:12 dr:17 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
由以上红色字体可知同步完成
10.创建ext3文件系统
[root@web1 ~]# mkfs.ext3 /dev/drbd0
11.挂载/dev/drbd0到/var/www/html目录下，并放置一个测试网页，然后再卸载该挂载
[root@web1 ~]# mount /dev/drbd0 /var/www/html/
[root@web1 ~]# echo "This is a testPage" >/var/www/html/index.html
[root@web1 ~]# umount /var/www/html/
12.恢复drbd状态为secondary/secondary，并停止drbd服务
[root@web1 ~]# drbdadm secondary r0
[root@web1 ~]# for i in 1 2; do ssh web$i /etc/init.d/drbd stop; done
至此drbd配置完成
四：配置corosync
1.[root@web1 ~]# cd /etc/corosync/
2.[root@web1 corosync]# cp corosync.conf.example corosync.conf
3. 完成后的配置文件如下所示：
[root@web1 corosync]# cat corosync.conf
# Please read the corosync.conf.5 manualpage
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.10.0
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
#
# 以下为添加部分
service {
ver: 0
name: pacemaker
}
4.复制配置文件到web2上
[root@web1 corosync]# scp corosync.conf web2:/etc/corosync/
5.创建/var/log/cluster目录并启动corosync服务
[root@web1 ~]# for i in 1 2; do ssh web$i mkdir /var/log/cluster; done
[root@web1 ~]# /etc/init.d/corosync start
Starting Corosync Cluster Engine(corosync): [ OK ]
[root@web1 ~]# ssh web2 /etc/init.d/corosync start
Starting Corosync Cluster Engine(corosync): [ OK ]
6.设置corosync随机启动
[root@web1 ~]# for i in 1 2; do ssh web$i chkconfig corosync on; done
四：集群服务配置
1.查看目前的集群状态
[root@web1 ~]# crm status
Last updated: Tue Jun 23 15:28:58 2015
Last change: Tue Jun 23 15:23:58 2015 viacrmd on web1
Stack: classic openais (with plugin)
Current DC: web1 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Webs configured, 2 expected votes
0 Resources configured
Online: [ web1 web2 ]
由以上可知，节点1和2都在线，但还未配置任何资源
2.设置集群属性
[root@web1 ~]# crm configure
crm(live)configure# property stonith-enabled=false
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node web1
node web2
property cib-bootstrap-options: \
dc-version=1.1.10-14.el6-368c726 \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
3.添加一个名为webdrbd的drbd资源
在添加之前先了解一下克隆资源各个元数据的意义：
master-max：最多有多少份克隆资源可以被定义成主资源，默认是1；
master-node-max：每个节点上最多有多少份克隆资源可以被提升为主资源，默认是1；
clone-max: 在集群中最多能运行多少份克隆资源，默认和集群中的节点数相同；
clone-node-max：每个节点上最多能运行多少份克隆资源，默认是1；
notify：当成功启动或关闭一份克隆资源，要不要通知给其它的克隆资源，可用值为false,true；默认值是true；
globally-unique: 是否为集群中各节点的克隆资源取一个全局唯一名称，用来描述不同的功能，默认为true；
ordered：克隆资源是否按顺序(order)启动，而非一起(parallel)启动,可用值为false,true；默认值是true；
interleave：当对端的实例有了interleave，就可以改变克隆资源或主资源中的顺序约束；
开始添加drbd资源
crm(live)configure# primitive webdrbd ocf:linbit:drbd params \
> drbd_resource=r0 \
> op start timeout=240 \
> op stop timeout=100 \
> op monitor role=Master timeout=20 interval=20 \
> op monitor role=Slave timeout=20 interval=10
crm(live)configure# verify
4.接着添加一个名为ms_webdrbd的主备资源
crm(live)configure# ms ms_webdrbd webdrbd \
> meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
crm(live)configure# verify
现在提交
crm(live)configure# commit
返回到上一级，查看目前集群状态
crm(live)configure# cd
crm(live)# status
============
Last updated: Thu Jun 25 15:54:40 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
1 Resources configured.
============
Online: [ web1 web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web1 ]
Slaves: [ web2 ]
由以上可知，ms_webdrbd已启动，并且Master为web1，Slave为web2
注：由于drbd的特殊原因（当出现脑裂时，最好手动消除脑裂），所以不要设置资源切回操作(位置约束),如果是共享存储可以设置位置约束（哪个节点硬件资源更好一点，就更倾向于运行在哪个节点上）
5.添加一个名为webstore的文件系统资源并设置webstore必须和ms_webdrbd角色为Master的节点在一起，等到ms_webdrbd提升完毕Master的角色后才能启动webstore
crm(live)configure# primitive webstore ocf:heartbeat:Filesystem params \
> device=/dev/drbd0 directory=/var/www/html fstype=ext3 \
> op start timeout=60 \
> op stop timeout=60
crm(live)configure# verify
crm(live)configure# colocation webstore_with_ms_webdrbd_Master inf: webstore ms_webdrbd:Master
crm(live)configure# verify
crm(live)configure# order ms_webdrbd_before_webstore mandatory: ms_webdrbd:promote webstore:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
============
Last updated: Thu Jun 25 16:01:35 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ web1 web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web1 ]
Slaves: [ web2 ]
webstore (ocf::heartbeat:Filesystem): Started web1
6.添加httpd服务资源并设置httpd服务必须和webstore在一起，webstore必须先启动后，httpd服务才能启动
crm(live)configure# primitive httpd lsb:httpd
crm(live)configure# colocation httpd_with_httpd inf: httpd webstore
crm(live)configure# order webstore_before_httpd mandatory: webstore:start httpd:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
============
Last updated: Thu Jun 25 16:04:54 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
3 Resources configured.
============
Online: [ web1 web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web1 ]
Slaves: [ web2 ]
webstore (ocf::heartbeat:Filesystem): Started web1
httpd (lsb:httpd): Started web1
7.添加虚拟IP资源，并设置虚拟IP必须和httpd服务在一起，httpd服务启动后，才能启动虚拟IP
crm(live)configure# primitive webip ocf:heartbeat:IPaddr params \
> ip=192.168.10.100
crm(live)configure# colocation webip_with_httpd inf: webip httpd
crm(live)configure# order httpd_before_webip mandatory: httpd:start webip:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
============
Last updated: Thu Jun 25 16:21:09 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ web1 web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web1 ]
Slaves: [ web2 ]
webstore (ocf::heartbeat:Filesystem): Started web1
httpd (lsb:httpd): Started web1
webip (ocf::heartbeat:IPaddr): Started web1
五：高可用测试
1.使web1离线后，查看集群状态
[root@web1 ~]# crm node standby
[root@web1 ~]# crm status
============
Last updated: Thu Jun 25 16:22:21 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Node web1: standby
Online: [ web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web2 ]
Stopped: [ webdrbd:0 ]
webstore (ocf::heartbeat:Filesystem): Started web2
httpd (lsb:httpd): Started web2
webip (ocf::heartbeat:IPaddr): Started web2
由以上可知，资源切换到了web2上
2.使web1重新上线
[root@web1 ~]# crm node online
[root@web1 ~]# crm status
============
Last updated: Thu Jun 25 16:24:19 2015
Stack: openais
Current DC: web1 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ web1 web2 ]
Master/Slave Set: ms_webdrbd
Masters: [ web2 ]
Slaves: [ web1 ]
webstore (ocf::heartbeat:Filesystem): Started web2
httpd (lsb:httpd): Started web2
webip (ocf::heartbeat:IPaddr): Started web2
由以上可知，web1重新上线，并且角色为Slave
并且在节点离线和上线的过程中，不断访问http://192.168.10.100，都可以看到“This is a test Page”
至此corosync+pacemaker+drbd的apache高可用配置完成
本文参考以下文章：
http://www.linuxidc.com/Linux/2014-05/101304.htm
本文出自 “永不止步” 博客，谢绝转载！