一、DRBD定义
DRBD的全称为：Distributed Replicated Block Device (DRBD)分布式块设备复制,DRBD是由内核模块和相关脚本而构成，用以构建高可用性的集群。其实现方式是通过网络来镜像整个设备。它允许用户在远程机器上建立一个本地块设备的实时镜像。与心跳连接结合使用，也可以把它看作是一种网络RAID。DRBD因为是工作在系统内核空间，而不是用户空间，它直接复制的是二进制数据，这是它速度快的根本原因。
一个DRBD系统由两个以上节点构成，与HA集群类似，也有主用节点和备用节点之分，在带有主要设备的节点上，应用程序和操作系统可以运行和访问DRBD设备。
在主节点写入的数据通过drbd设备存储到主节点的磁盘设备中，同时，这个数据也会自动发送到备用节点相应的drbd设备，最终写入备用节点的磁盘设备中，在备用节点上，drbd只是将数据从drbd设备写入到备用节点的磁盘设备中。
DRBD在数据进入Buffer Cache时，先经过DRBD这一层，复制一份数据经过TCP/IP协议封装，发送到另一个节点上，另一个节点通过TCP/IP协议来接受复制过来的数据，同步到次节点的DRBD设备上。
DRBD内核中工作模型：
DRBD是有资源组成：
resource name：可以使用除空白字符以外的任意ACSII表中的字符。
drbd设备：drbd的设备的访问路径；设备文件/dev/drbd#。
disk：各节点为组成此drbd设备所提供的块设备。
网络属性：节点间为了实现跨主机的磁盘镜像而使用的网络配置。
DRBD的复制模式：
| 协议A(异步) | 数据一旦写入磁盘并发送到本地TCP/IP协议栈中就认为完成了写入操作。 |
| 协议B(半同步) | 收到对方接收确认即发送到对方的TCP/IP协议栈就认为完成了写入操作。 |
| 协议C(同步) | 等待对方写入完成后并返回确认信息才认为完成写入操作。 |
脑裂(split brain)自动修复方法：
丢弃比较新的主节点的所做的修改：
在这种模式下，当网络重新建立连接并且发现了裂脑，DRBD就会丢弃自切换到主节点后所修改的数据。
丢弃老的主节点所做的修改：
在这种模式下，DRBD将丢弃首先切换到主节点后所修改的数据。
丢弃修改比较少的主节点的修改：
在这种模式下，DRBD会检查两个节点的数据，然后丢弃修改比较少的主机上的节点。
一个节点数据没有发生变化的完美的修复裂脑：
在这种模式下，如果其中一台主机的在发生裂脑时数据没有发生修改，则可简单的完美的修复并声明已经解决裂脑问题。
了解基本概念后可以来安装配置DRBD了。
二、安装
由于在Linux 2.6.33以后的版本中，drbd已经集成到内核中；而之前的版本只能通过两种方法安装：
1、对内核打补丁；编译该模块
2、安装rpm包(要与内核完全匹配)
准备工作：
#下载与内核版本一致的内核rpm包和用户空间工具包
[root@node1 drbd]# ls
drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm
[root@node1 drbd]# uname -r
2.6.32-431.el6.x86_64
#两个测试节点
node1.soul.com 192.168.0.111
node2.soul.com 192.168.0.112
#保证时间同步；双机互信
[root@node1 ~]# ssh node2 'date';date
Wed Apr 23 18:13:30 CST 2014
Wed Apr 23 18:13:30 CST 2014
[root@node1 ~]#
Wed Apr 23 18:11:12 CST 2014
安装：
[root@node1 drbd]# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm warning: drbd-8.4.3-33.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 66534c2b: NOKEY Preparing... ########################################### [100%] 1:drbd-kmdl-2.6.32-431.el########################################### [ 50%] 2:drbd ########################################### [100%] [root@node1 drbd]# scp *.rpm node2:/root/drbd/ drbd-8.4.3-33.el6.x86_64.rpm 100% 283KB 283.3KB/s 00:00 drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm 100% 145KB 145.2KB/s 00:00 [root@node1 drbd]# ssh node2 "rpm -ivh /root/drbd/*.rpm" warning: /root/drbd/drbd-8.4.3-33.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 66534c2b: NOKEY Preparing... ################################################## drbd-kmdl-2.6.32-431.el6 ################################################## drbd ################################################## [root@node1 drbd]# #两个节点都安装相同的包
查看配置文件
[root@node1 ~]# cd /etc/drbd.d/
[root@node1 drbd.d]# ls
global_common.conf
[root@node1 drbd.d]#
[root@node1 drbd.d]# vim global_common.conf
global {
usage-count no; #在可以访问互联网的情况下drbd可以统计使用数据
# minor-count dialog-refresh disable-ip-verification
}
common {
handlers { #处理器
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
#启动下面这三项：
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup { #启动时执行脚本
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
options {
# cpu-mask on-no-data-accessible
}
disk { #drbd设备
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
on-io-error detach; #当io发生错误时直接拆掉磁盘
}
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
cram-hmac-alg "sha1";
shared-secret "node.soul.com"; #建议使用随机数值
protocol C; #复制协议
}
syncer {
rate 1000M; #传输速率
}
}
三、配置使用
提供一个磁盘分区；或单独一个磁盘；两节点需大小一致
[root@node1 ~]# fdisk -l /dev/sda Disk /dev/sda: 128.8 GB, 128849018880 bytes 255 heads, 63 sectors/track, 15665 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00076a20 Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 7859 62914560 8e Linux LVM /dev/sda3 7859 8512 5252256 83 Linux #该分区作为drbd资源 #node2也是如此
定义资源
[root@node1 ~]# vim /etc/drbd.d/web.res
resource web {
on node1.soul.com {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.0.111:7789;
meta-disk internal;
}
on node2.soul.com {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.0.112:7789;
meta-disk internal;
}
}
#复制到node2上一份
[root@node1 ~]# scp /etc/drbd.d/web.res node2:/etc/drbd.d/
web.res 100% 247 0.2KB/s 00:00
[root@node1 ~]# scp /etc/drbd.d/global_common.conf node2:/etc/drbd.d/
global_common.conf 100% 1945 1.9KB/s 00:00
[root@node1 ~]#
初始化资源并启动服务
[root@node1 ~]# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
#node2同样执行
[root@node1 ~]# service drbd start
Starting DRBD resources: [
create res: web
prepare disk: web
adjust disk: web
adjust net: web
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 0 seconds. [wfc-timeout]
(These values are for resource 'web'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 11]: #等待node2启动,node2启动即可
.
[root@node1 ~]
#查看状态
[root@node1 ~]# drbd-overview
0:web/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
[root@node1 ~]#
手动将其中一个节点提升为主节点
[root@node1 ~]# drbdadm primary --force web
[root@node1 ~]# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:321400 nr:0 dw:0 dr:329376 al:0 bm:19 lo:0 pe:1 ua:8 ap:0 ep:1 wo:f oos:4931544
[>...................] sync'ed: 6.3% (4812/5128)M
finish: 0:02:33 speed: 32,048 (32,048) K/sec
#显示正常同步进行
#显示的Primary/Secondary左边为自己；右边为对方节点
[root@node2 ~]# drbd-overview
0:web/0 Connected Secondary/Primary UpToDate/UpToDate C r-----
#node2上显示
手动做主从切换
#因为同时只能有一个为主；所以切换时；需要先将主的降级；才能提升
#且只有主的才能格式化被挂载使用
node1
[root@node1 ~]# drbdadm secondary web
[root@node1 ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
node2
[root@node2 ~]# drbdadm primary web
[root@node2 ~]# drbd-overview
0:web/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
#格式化挂载
[root@node2 ~]# mke2fs -t ext4 /dev/drbd0
This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@node2 ~]#
[root@node2 ~]# mount /dev/drbd0 /mnt/
[root@node2 ~]# ls /mnt/
lost+found
#挂载成功
#如果要为node1为主；这边需要先卸载降级
四、与pacemaker结合使用自动角色转换
[root@node1 ~]# drbdadm secondary web
[root@node1 ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@node1 ~]# service drbd stop
Stopping all DRBD resources: .
[root@node1 ~]# ssh node2 'service drbd stop'
Stopping all DRBD resources: .
[root@node1 ~]#
[root@node1 ~]# chkconfig drbd off
[root@node1 ~]# ssh node2 'chkconfig drbd off'
#卸载降级关闭服务；且关闭开启自动启动。两个节点相同操作
配置pacemaker
#具体配置就不详细介绍了；上一篇介绍的有
[root@node1 ~]# crm status
Last updated: Wed Apr 23 21:00:11 2014
Last change: Wed Apr 23 18:59:58 2014 via cibadmin on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node2.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ node1.soul.com node2.soul.com ]
crm(live)configure# show
node node1.soul.com
node node2.soul.com
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
#配置好直接查看相关信息
配置drbd为高可用资源
#drbd的资源代理是有linbit提供
crm(live)ra# info ocf:linbit:drbd
Parameters (* denotes required, [] the default):
drbd_resource* (string): drbd resource name
The name of the drbd resource from the drbd.conf file.
drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf
Full path to the drbd.conf file.
....
#查看详细信息
#定义资源
crm(live)configure# primitive webdrbd ocf:linbit:drbd params drbd_resource=web op monitor role=Master interval=40s timeout=30s op monitor role=Slave interval=60s timeout=30s op start timeout=240s op stop timeout=100s
crm(live)configure# verify
crm(live)configure# master MS_webdrbd webdrbd meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
crm(live)configure# verify
crm(live)configure# show
node node1.soul.com
node node2.soul.com
primitive webdrbd ocf:linbit:drbd \
params drbd_resource="web" \
op monitor role="Master" interval="40s" timeout="30s" \
op monitor role="Slave" interval="60s" timeout="30s" \
op start timeout="240s" interval="0" \
op stop timeout="100s" interval="0"
ms MS_webdrbd webdrbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
crm(live)configure# commit
定义完成后可以查看当前状态
crm(live)# status
Last updated: Wed Apr 23 21:39:49 2014
Last change: Wed Apr 23 21:39:32 2014 via cibadmin on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ node1.soul.com node2.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node1.soul.com ]
Slaves: [ node2.soul.com ]
#可以让node1下线测试
crm(live)# node standby node1.soul.com
crm(live)# status
Last updated: Wed Apr 23 21:44:31 2014
Last change: Wed Apr 23 21:44:27 2014 via crm_attribute on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
2 Resources configured
Node node1.soul.com: standby
Online: [ node2.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node2.soul.com ]
Stopped: [ node1.soul.com ]
#可以看出node2自动变成主节点了
五、定义webfs服务共享drbd资源
crm(live)configure# primitive webfs ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/var/www/html" fstype="ext4" op monitor interval=30s timeout=40s on-fail=restart op start timeout=60s op stop timeout=60s
crm(live)configure# verify
crm(live)configure# colocation webfs_with_MS_webdrbd_M inf: webfs MS_webdrbd:Master
crm(live)configure# verify
crm(live)configure# order webfs_after_MS_webdrbd_M inf: MS_webdrbd:promote webfs:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status
Last updated: Wed Apr 23 21:57:23 2014
Last change: Wed Apr 23 21:57:11 2014 via cibadmin on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
3 Resources configured
Online: [ node1.soul.com node2.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node2.soul.com ]
Slaves: [ node1.soul.com ]
webfs (ocf::heartbeat:Filesystem): Started node2.soul.com
[root@node2 ~]# ls /var/www/html/
issue lost+found
#在node2上测试挂载成功
#测试转移
crm(live)# node standby node2.soul.com
crm(live)# status
Last updated: Wed Apr 23 21:59:00 2014
Last change: Wed Apr 23 21:58:51 2014 via crm_attribute on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
3 Resources configured
Node node2.soul.com: standby
Online: [ node1.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node1.soul.com ]
Stopped: [ node2.soul.com ]
webfs (ocf::heartbeat:Filesystem): Started node1.soul.com
[root@node1 ~]# ls /var/www/html/
issue lost+found
[root@node1 ~]#
#测试转移成功
六、配置webip和webserver实现web服务高可用
crm(live)# configure primitive webip ocf:heartbeat:IPaddr params ip="192.168.0.222" op monitor it=30s on-fail=restart
crm(live)configure# primitive webserver lsb:httpd op monitor interval=30s timeout=30s on-fail=restart
crm(live)configure# group webcluster MS_webdrbd webip webfs webserver
INFO: resource references in colocation:webfs_with_MS_webdrbd_M updated
INFO: resource references in order:webfs_after_MS_webdrbd_M updated
crm(live)configure# show
group webcluster webip webfs webserver
ms MS_webdrbd webdrbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation webfs_with_MS_webdrbd_M inf: webcluster MS_webdrbd:Master
order webfs_after_MS_webdrbd_M inf: MS_webdrbd:promote webcluster:start
#查看信息正符合要求
crm(live)# status
Last updated: Wed Apr 23 23:02:26 2014
Last change: Wed Apr 23 22:55:36 2014 via cibadmin on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Node node2.soul.com: standby
Online: [ node1.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node1.soul.com ]
Stopped: [ node2.soul.com ]
Resource Group: webcluster
webip (ocf::heartbeat:IPaddr): Started node1.soul.com
webfs (ocf::heartbeat:Filesystem): Started node1.soul.com
webserver (lsb:httpd): Started node1.soul.com
#转移测试
crm(live)# node standby node1.soul.com
crm(live)# status
Last updated: Wed Apr 23 23:03:18 2014
Last change: Wed Apr 23 23:03:15 2014 via crm_attribute on node1.soul.com
Stack: classic openais (with plugin)
Current DC: node1.soul.com - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Node node1.soul.com: standby
Online: [ node2.soul.com ]
Master/Slave Set: MS_webdrbd [webdrbd]
Masters: [ node2.soul.com ]
Stopped: [ node1.soul.com ]
Resource Group: webcluster
webip (ocf::heartbeat:IPaddr): Started node2.soul.com
webfs (ocf::heartbeat:Filesystem): Started node2.soul.com
webserver (lsb:httpd): Started node2.soul.com
到此；drbd基础与pacemaker实现自动转移以完成。
如有错误；恳请指正。