HA集群之四：Corosync+Pacemaker+DRBD实现HA Mysql
一、drbd基础
1、存储
DAS: Direct Attached Storage 直接附加存储,块设备
DAS这种存储方式与我们普通的PC存储架构一样，外部存储设备都是直接挂接在服务器内部总线上，数据存储设备是整个服务器结构的一部分。是指将存储设备通过 ide, usb, sata, scsi, sas接口或光纤通道直接连接到一台计算机上。
NAS：Network Attached Storage 网络附加存储
它就是个文件服务器，是文件系统级别，NAS和传统的文件存储服务或直接存储设备不同的地方在于NAS设备上面的操作系统和软件只提供了数据存储、数据访问、以及相关的管理功能；
NAS用的是以文件为单位的通信协议，例如像是NFS（在UNIX系统上很常见）或是SMB（常用于Windows系统）。
SAN:Storage Area Network 存储区域网络
把SCSI协议借助于其它网络协议实现传送的；1991年，IBM公司在S/390服务器中推出了ESCON(Enterprise System Connection)技术。它是基于光纤介质，最大传输速率达17MB/s的服务器访问存储器的一种连接方式。在此基础上，进一步推出了功能更强的ESCON Director(FC SWitch)，构建了一套最原始的SAN系统。
存储区域网络（SAN）用的是以区块为单位的通信协议、通常是通过SCSI再转为光纤通道或是iSCSI。（还有其他各种不同的SAN通信协议，像是ATA over Ethernet和HyperSCSI，不过这些都不常见。）
集群文件系统： (支持的节点不多，最多16个)
GFS2, OCFS2,cLVM2
2、drbd
drbd：跨主机的块设备镜像系统
基于网络实现数据镜像，工作于内核
用户空间管理工具：drbdadm, drbdsetup, drbdmeta
工作特性：实时、透明、同步或异步；
每组drbd设备都由"drbd resource"进行定义：
名字：只能由空白字符之外ASCII字符组成；
drbd设备：/dev/drbd#
主设备号：147,
次设备号：从0开始编号
磁盘配置：各主机上用于组成此drbd设备的磁盘或分区；
网络配置：数据同步时的网络通信属性配置；
数据同步模型：
三种协议：Protocal A, B, C
A: Async 异步
数据发送本地网卡发送队列
B: Semi-Sync 半同步
数据发送到对方网卡接收队列
C: sync 同步
数据发送到对方硬盘
drbd工作模型：
master/slave：主/从
dual master: 双主；要求必须在HA集群使用集群文件系统；
二、案例：Corosync+Pacemaker+DRBD实现HA Mysql
drbd：可靠性不是很高，有可能丢失数据
drbd：分为两部分
用户空间工具：跟内核版本关系比较松散，只要是能适用于CentOS 6及对应硬件平台的就OK;
内核模块：必须与当下内核版本严格对应；其中drbd内核模块代码已经整合进Linux内核2.6.33以后的版本中，因此，如果您的内核版本高于此版本的话，你只需要安装管理工具即可；
否则，您需要同时安装内核模块和管理工具两个软件包，并且此两者的版本号一定要保持对应。
1、配置前提：时间同步、基于主机名访问，双机互信
详细过程见前面的博文
2、为两个节点准备等同大小的磁盘分区，分区好之后不需要格式化，分好区并且识别出就可以了；
[root@BAIYU_180 ~]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
First cylinder (3982-5222, default 3982):
Using default value 3982
Command (m for help): n
First cylinder (4244-5222, default 4244):
Using default value 4244
Last cylinder, +cylinders or +size{K,M,G} (4244-5222, default 5222): +1G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: 设备或资源忙.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@BAIYU_180 ~]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
BLKPG: Device or resource busy
error adding partition 4
BLKPG: Device or resource busy
error adding partition 5
BLKPG: Device or resource busy
error adding partition 6
[root@BAIYU_180 ~]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
BLKPG: Device or resource busy
error adding partition 4
BLKPG: Device or resource busy
error adding partition 5
BLKPG: Device or resource busy
error adding partition 6
BLKPG: Device or resource busy
error adding partition 7
BLKPG: Device or resource busy
error adding partition 8
[root@BAIYU_180 ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b52c1
Device Boot Start End Blocks Id System
/dev/sda1 * 1 66 524288 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 66 1371 10485760 83 Linux
/dev/sda3 1371 2677 10485760 83 Linux
/dev/sda4 2677 5222 20446208 5 Extended
/dev/sda5 2677 3982 10485760 83 Linux
/dev/sda6 3983 4244 2097152 82 Linux swap / Solaris
/dev/sda7 3982 3982 743+ 83 Linux
/dev/sda8 4244 4375 1059340 83 Linux
3、安装drdb；
[root@node1 ~]# uname -r # 查看系统内核版本 2.6.32-431.el6.x86_64 [root@BAIYU_179 ~]# rpm -q centos-release centos-release-6-5.el6.centos.11.1.x86_64
下载对应的程序包: drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm # drdb内核模块 drbd-8.4.3-33.el6.x86_64.rpm # drbd主程序 安装drdb: [root@BAIYU_179 ~]# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm warning: drbd-8.4.3-33.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 66534c2b: NOKEY error: Failed dependencies: drbd-kmdl-8.4.3-33.el6 is needed by drbd-8.4.3-33.el6.x86_64 [root@BAIYU_179 ~]# rpm -ivh drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el 6.x86_64.rpm warning: drbd-8.4.3-33.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 66534c2b: NOKEY Preparing... ########################################### [100%] 1:drbd-kmdl-2.6.32-431.el########################################### [ 50%] 2:drbd ########################################### [100%]
4、配置drbd
配置文件：
/etc/drbd.conf 主配置文件
/etc/drbd.d/global_common.conf: 提供全局配置，及多个drbd设备相同的配置；
/etc/drbd.d/*.res: 资源定义；
global: 全局属性，定义drbd自己的工作特性；
common: 通用属性，定义多组drbd设备通用特性；
*.res：资源特有的配置
1）配置/etc/drbd.d/global_common.conf
[root@BAIYU_180 ~]# vi /etc/drbd.d/global_common.conf
global { #全局配置
usage-count no; #这个默认为yes表示如果你本机可以连接互联网时drbd会通过互联网收集到你安装drbd的信息，不用可以改为no
# minor-count dialog-refresh disable-ip-verification
}
common {
handlers { #处理器
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
#fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup { 定义一个节点启动时另一个节点应该怎么做
# wfc-timeout(等待另一个节点上线的超时时长)
# degr-wfc-timeout(等待超时后做降级处理)
# outdated-wfc-timeout(过期的等待超时)
# wait-after-sb(脑裂之后等待多长时长)
}
options { #定义同步属性
# cpu-mask on-no-data-accessible
}
disk {
on-io-error detach; # IO发生错误，就拆除此节点
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}
net {
protocol C; # 同步模型
cram-hmac-alg "sha1"; # 数据同步时使用的加密协议
shared-secret "www.magedu.com"; # 密钥
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
}
syncer {
rate 1000M; # 最大速率
}
}
2）、定义资源，包括资源名，drbd设备，disk以及网络属性，主要是这四个方面；
# cd /etc/drbd.d/
# vim mystore.res
resource mystore { #定义一个资源，用关键字resource；
2 on BAIYU_179 { #on说明在哪个节点上，跟uname -n保持一致,有多少个>
节点就定义多少个；
3 device /dev/drbd0; #在磁盘上表现的drbd叫什么名；
4 disk /dev/sda8; #所使用的磁盘设备是哪个；
5 address 192.168.100.179:7789; #在node1这个节点上监听的套接字,默认监
听在7789端口上；
6 meta-disk internal; #保存drbd元数据信息的，表示就放在自己的磁盘区分上，
也可以放在外部的磁盘上；
7 }
8 on BAIYU_180 {
9 device /dev/drbd0;
10 disk /dev/sda8;
11 address 192.168.100.180:7789;
12 meta-disk internal;
13 }
14 }
复制一份到别一个节点上，使它们的配置文件要保持一致：
[root@BAIYU_179 drbd.d]# ls global_common.conf global_common.conf.orig mystore.res mystore.res.orig [root@BAIYU_179 drbd.d]# scp global_common.conf mystore.res BAIYU_180:/etc/drbd .d root@baiyu_180's password: global_common.conf 100% 1995 2.0KB/s 00:00 mystore.res 100% 675 0.7KB/s 00:00
在各节点分别初始化已定义的资源并启动服务：
[root@BAIYU_179 drbd.d]# drbdadm create-md mystore Writing meta data... initializing activity log NOT initializing bitmap lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory New drbd meta data block successfully created. #提示已经创建成功 lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory #警告信息可以忽略不计 [root@BAIYU_179 lib]# service drbd start Starting DRBD resources: [ create res: mystore prepare disk: mystore adjust disk: mystore adjust net: mystore ] .......... *************************************************************** DRBD's startup script waits for the peer node(s) to appear. - In case this node was already a degraded cluster before the reboot the timeout is 0 seconds. [degr-wfc-timeout] - If the peer was available before the reboot the timeout will expire after 0 seconds. [wfc-timeout] (These values are for resource 'mystore'; 0 sec -> wait forever) To abort waiting enter 'yes' [ -- ]: [ 43]: To abort waiting enter 'yes' [ -- ]:[ 215]: # 在等待其它节点启动，其它节点也启动后才能启动完成 . [root@BAIYU_179 lib]# cat /proc/drbd # 查看drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1059268 [root@BAIYU_179 lib]# date 2015年 10月 27日 星期二 18:53:38 CST [root@BAIYU_179 lib]# drbd-overview # 查看drbd 0:mystore/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
从上面查看的结果:Secondary/Secondary Inconsistent/Inconsistent 知道2个节点都是从，数据没有同步；要把其中一个提升为主才会数据同步：
[root@BAIYU_179 lib]# drbdadm primary --force mystore # 将该节点提升为主节点 [root@BAIYU_180 drbd.d]# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n- ns:47888 nr:0 dw:0 dr:54936 al:0 bm:2 lo:2 pe:2 ua:8 ap:0 ep:1 wo:f oos:1013188 [>....................] sync'ed: 4.7% (1013188/1059268)K #开始在同步，已经同步了4.7% finish: 0:01:49 speed: 9,216 (9,216) K/sec
OK、看到两个节点上的数据正在同步了，磁盘越大同步时需要时间越久；
[root@BAIYU_180 drbd.d]# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1059268 nr:0 dw:0 dr:1059932 al:0 bm:65 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
当前节点为主对方为从，数据已同步
注意：哪个是主节点哪个就可以挂载使用，不是主节点的连挂载都不可以挂载；
挂载格式化drbd0，挂载测试使用：
[root@BAIYU_179 ~]# mke2fs -t ext4 /dev/drbd0 mke2fs 1.41.12 (17-May-2010) 文件系统标签= 操作系统:Linux 块大小=4096 (log=2) 分块大小=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 66240 inodes, 264817 blocks 13240 blocks (5.00%) reserved for the super user 第一个数据块=0 Maximum filesystem blocks=272629760 9 block groups 32768 blocks per group, 32768 fragments per group 7360 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 正在写入inode表: 完成 Creating journal (8192 blocks): 完成 Writing superblocks and filesystem accounting information: 完成 This filesystem will be automatically checked every 38 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. [root@BAIYU_179 ~]# mount /dev/drbd0 /mnt [root@BAIYU_179 ~]# ls /mnt lost+found [root@BAIYU_179 ~]# cp /etc/inittab /mnt [root@BAIYU_179 ~]# umount /dev/drbd0 [root@BAIYU_179 ~]# drbdadm secondary mystore # 先卸载再降级成从节点 [root@BAIYU_179 ~]# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r----- ns:49880 nr:0 dw:49880 dr:1377 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 [root@BAIYU_179 ~]# ssh BAIYU_180 # 登录到另一个节点 Last login: Tue Oct 27 18:03:41 2015 from 192.168.100.88 You have entered the system: 168.100.180 your username is: root WARNING:Proceed with caution! Have any questions please contact the system administrator [root@BAIYU_180 ~]# drbd-overview 0:mystore/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- [root@BAIYU_180 ~]# drbdadm primary mystore # 提升至主节点 [root@BAIYU_180 ~]# drbd-overview 0:mystore/0 Connected Primary/Secondary UpToDate/UpToDate C r----- [root@BAIYU_180 ~]# mount /dev/drbd0 /mnt [root@BAIYU_180 ~]# ls /mnt # drbd0中的文件还在 inittab lost+found
配置drbd就这么顺利的完成了。
5、将drbd加入HA资源
要自动完成drbd的角色切换得要借助于corosync+pacmaker，那接下来我们就来安装配置corosync和pacemaker；
1）为了让高可用的配置顺利，首先将两个节点都降级为从，停止运行，禁止开机自动启动：
[root@BAIYU_179 ~]# cat /proc/drbd version: 8.4.3 (api:1/proto:86-101) GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:49880 nr:24 dw:49904 dr:1377 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 [root@BAIYU_179 ~]# ssh BAIYU_180 Last login: Tue Oct 27 21:49:46 2015 from 192.168.100.179 You have entered the system: 168.100.180 your username is: root WARNING:Proceed with caution! Have any questions please contact the system administrator [root@BAIYU_180 ~]# umount /mnt [root@BAIYU_180 ~]# drbdadm secondary mystore [root@BAIYU_180 ~]# service drbd stop Stopping all DRBD resources: . [root@BAIYU_180 ~]# chkconfig drbd off [root@BAIYU_180 ~]# exit logout Connection to BAIYU_180 closed. [root@BAIYU_179 ~]# service drbd stop Stopping all DRBD resources: . [root@BAIYU_179 ~]# chkconfig drbd off
2）安装配置并启动corosync+pacemaker+crm_sh
详细过程上篇博文
[root@BAIYU_179 yum.repos.d]# yum install corosync pacemaker -y [root@BAIYU_179 corosync]# service corosync start; ssh BAIYU_180 service corosync start Starting Corosync Cluster Engine (corosync): [确定] Starting Corosync Cluster Engine (corosync): [确定] [root@BAIYU_179 corosync]# service pacemaker start; ssh BAIYU_180 service pacemaker start Starting Pacemaker Cluster Manager[确定] Starting Pacemaker Cluster Manager[确定] [root@BAIYU_179 corosync]# crm crmadmin crm_error crm_mon crm_resource crm_standby crm_attribute crm_failcount crm_node crm_shadow crm_ticket crm_diff crm_master crm_report crm_simulate crm_verify [root@BAIYU_179 corosync]# crm_mon Attempting connection to the cluster... Last updated: Tue Oct 27 22:20:21 2015 Last change: Tue Oct 27 22:20:16 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ BAIYU_179 BAIYU_180 ] 安装crmsh: [root@BAIYU_180 corosync]# crm crm(live)# status Last updated: Tue Oct 27 22:35:54 2015 Last change: Tue Oct 27 22:20:16 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ BAIYU_179 BAIYU_180 ] crm(live)configure# property stonith-enabled=false crm(live)configure# property no-quorum-policy=ignore crm(live)configure# show node BAIYU_179 node BAIYU_180 property cib-bootstrap-options: \ dc-version=1.1.11-97629de \ cluster-infrastructure="classic openais (with plugin)" \ expected-quorum-votes=2 \ stonith-enabled=false \ no-quorum-policy=ignore crm(live)configure# verify crm(live)configure# commit
3）将drbd加入HA
root@BAIYU_179 ~]# crm crm(live)# ra crm(live)ra# classes lsb ocf / heartbeat linbit pacemaker service stonith crm(live)ra# --help back classes help ls quit -h bye end info meta up ? cd exit list providers crm(live)ra# classes lsb ocf / heartbeat linbit pacemaker service stonith crm(live)ra# list ocf CTDB ClusterMon Delay Dummy Filesystem HealthCPU HealthSMART IPaddr IPaddr2 IPsrcaddr LVM MailTo Route SendArp Squid Stateful SysInfo SystemHealth VirtualDomain Xinetd apache conntrackd controld db2 dhcpd drbd ethmonitor exportfs iSCSILogicalUnit mysql named nfsnotify nfsserver nginx pgsql ping pingd postfix remote rsyncd symlink tomcat crm(live)ra# list ocf linbit drbd crm(live)ra# info ocf:linbit:drbd # 查看默认设置和参数
注意：clone资源首先是基本资源
在pacemaker中定义克隆资源的专用属性：
clone-max：最多克隆出的资源份数；
clone-node-max：在单个节点上最多运行几份克隆；
notify：当一份克隆资源启动或停止时，是否通知给其它的副本；
master-max：最多启动几份master资源；
master-node-max：同一个节点最多运行几份master类型资源；
resource cleanup ms_mystor #清理资源的报错信息
node clearstate BAIYU_180 #清理节点状态，清理后节点pacemaker会停止，再次启动pacemaker，资源停止运行；当资源无法停止时使用此方法，
crm(live)# crm(live)# configure crm(live)configure# primitive mystor ocf:linbit:drbd params drbd_resource="mystore" op monitor r ole="Master" interval=10s timeout=20s op monitor role="Slave" interval=20s timeout=20s op start timeout==240s op stop timeout=100s crm(live)configure# ms ms_mystor mystor meta clone-max="2" clone-node-max="1" master-max="1" mas ter-node-max="1" notify="true" crm(live)configure# verify crm(live)configure# commit crm(live)configure# show node BAIYU_179 \ attributes standby=off node BAIYU_180 primitive mystor ocf:linbit:drbd \ params drbd_resource=mystore \ op monitor role=Master interval=10s timeout=20s \ op monitor role=Slave interval=20s timeout=20s \ op start timeout="=240s" interval=0 \ op stop timeout=100s interval=0 ms ms_mystor mystor \ meta clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true property cib-bootstrap-options: \ dc-version=1.1.11-97629de \ cluster-infrastructure="classic openais (with plugin)" \ expected-quorum-votes=2 \ stonith-enabled=false \ no-quorum-policy=ignore crm(live)configure# cd crm(live)# status Last updated: Wed Oct 28 01:35:05 2015 Last change: Wed Oct 28 01:34:53 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 2 Resources configured Online: [ BAIYU_179 BAIYU_180 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_180 ] Slaves: [ BAIYU_179 ]
可以看到现在180是主，我们把180备用，看179是否会自动提升为主：
crm(live)# node standby BAIYU_180 crm(live)# status Last updated: Wed Oct 28 01:38:42 2015 Last change: Wed Oct 28 01:38:36 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 2 Resources configured Node BAIYU_180: standby Online: [ BAIYU_179 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_179 ] Stopped: [ BAIYU_180 ]
再让180上线，180是否又会自动提升为主？
crm(live)# node online BAIYU_180 crm(live)# status Last updated: Wed Oct 28 01:40:21 2015 Last change: Wed Oct 28 01:40:19 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 2 Resources configured Online: [ BAIYU_179 BAIYU_180 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_179 ] Slaves: [ BAIYU_180 ]
发现是不会的。
4）此时定义brbd资源，HA只能自动切换节点为主从，不能实现自动挂载。所以我们还要定义一个文件系统资源，来实现自动挂载
[root@BAIYU_179 ~]# mkdir /mydata [root@BAIYU_179 ~]# ssh BAIYU_180 mkdir /mydata #在两个节点上分别创建/mydata挂载目录
因为节点是主才能挂载使用brbd,所以要定义位置约束和顺序约束：
crm(live)configure# primitive mydata ocf:heartbeat:Filesystem params device="/dev/drbd0" directo ry="/mydata" fstype="ext4" op monitor interval=20s timeout=40s op start timeout=60s op stop time out=60s crm(live)configure# verify crm(live)configure# colocation mydata_with_ms_mystor_master inf: mydata ms_mystor:Master crm(live)configure# order mydata_after_ms_mystor_master Mandatory: ms_mystor:promote mydata:star t crm(live)configure# verify crm(live)configure# commit crm(live)configure# cd crm(live)# status Last updated: Wed Oct 28 01:54:01 2015 Last change: Wed Oct 28 01:53:54 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Online: [ BAIYU_179 BAIYU_180 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_179 ] Slaves: [ BAIYU_180 ] mydata (ocf::heartbeat:Filesystem): Started BAIYU_179
此时查看179：
[root@BAIYU_179 ~]# mount /dev/sda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/sda3 on /data type ext4 (rw) /dev/sda5 on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/drbd0 on /mydata type ext4 (rw)
将179备用再查看179和180：
crm(live)# node standby crm(live)# status Last updated: Wed Oct 28 01:56:15 2015 Last change: Wed Oct 28 01:56:12 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Node BAIYU_179: standby Online: [ BAIYU_180 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_180 ] Stopped: [ BAIYU_179 ] mydata (ocf::heartbeat:Filesystem): Started BAIYU_180 [root@BAIYU_179 ~]# mount /dev/sda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/sda3 on /data type ext4 (rw) /dev/sda5 on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) [root@BAIYU_179 ~]# ssh BAIYU_180 mount /dev/sda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/sda3 on /data type ext4 (rw) /dev/sda5 on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/drbd0 on /mydata type ext4 (rw)
共享存储配置完成。
5）安装mysql
注意：节点上的mysql版本要保持一致，此时哪个节点是主就在该节点上配置mysql
要确保2个节点上都有mysql用户，并且ID要一样，否则没有对/mydata的访问权限
[root@BAIYU_180 ~]# groupadd -r -g 306 mysql;ssh BAIYU_179 groupadd -r -g 306 mysql [root@BAIYU_180 ~]# useradd -r -g 306 -u 306 mysql;ssh BAIYU_179 useradd -r -g 306 -u 306 mysql [root@BAIYU_180 ~]# id mysql;ssh BAIYU_179 id mysql uid=306(mysql) gid=306(mysql) 组=306(mysql) uid=306(mysql) gid=306(mysql) 组=306(mysql) [root@BAIYU_180 ~]# ls -ld /mydata;ssh BAIYU_179 ls -ld /mydata drwxr-xr-x 3 root root 4096 10月 27 21:11 /mydata drwxr-xr-x 2 root root 4096 10月 28 01:43 /mydata [root@BAIYU_180 ~]# mkdir /mydata/data [root@BAIYU_180 ~]# mkdir /mydata/data [root@BAIYU_180 ~]# ls -ld /mydata/data drwxr-xr-x 2 root root 4096 10月 28 02:22 /mydata/data [root@BAIYU_180 ~]# chown -R mysql.mysql /mydata/data /data [root@BAIYU_180 ~]# ls -ld /mydata/data drwxr-xr-x 2 mysql mysql 4096 10月 28 02:22 /mydata/data [root@BAIYU_180 ~]# yum install -y mysql mysql-server
配置mysql主配置文件/etc/my.cnf
[root@BAIYU_180 mysql]# \cp /usr/share/doc/mysql-server-5.1.71/my-large.cnf /etc/my.cnf 在[client]段将socket = /var/lib/mysql/mysql.sock 改成socket = /mydata/data/mysql.sock 在[mysqld]段将socket = /var/lib/mysql/mysql.sock 改成socket = /mydata/data/mysql.sock 并添加datadir = /mydata/data行
启动mysql：
注意：另一个节点一定要先把drbd切换过去再启动mysqld，此时mysqld不会再初始化，要不然它会在本地初始化生成数据，造成数据错乱
[root@BAIYU_179 ~]# service mysql start #第一次启动mysql将初始化 [root@BAIYU_179 ~]# ls /var/lib/mysql/ [root@BAIYU_179 ~]# ls /mydata/data/ BAIYU_179.err ib_logfile0 mysql mysql-bin.000002 mysql-bin.index test ibdata1 ib_logfile1 mysql-bin.000001 mysql-bin.000003 mysql.sock [root@BAIYU_180 mysql]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.1.71-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY '123456' ; # 授权root用户可以在任何主机上以123456为密码登录此mysql Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; # 保存 Query OK, 0 rows affected (0.00 sec) mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | +--------------------+ 3 rows in set (0.00 sec) mysql> CREATE DATABASE xiexiaojun; Query OK, 1 row affected (0.00 sec) mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | | xiexiaojun | +--------------------+ 4 rows in set (0.00 sec) mysql> exit; Bye [root@BAIYU_180 data]# service mysqld stop 停止 mysqld： [确定] [root@BAIYU_180 data]# service mysqld stop # 停止此节点上的mysql，要不然切换节点时mysql不正常关闭 [root@BAIYU_180 data]# ls BAIYU_180.err ib_logfile0 mysql mysql-bin.000002 mysql-bin.index test ibdata1 ib_logfile1 mysql-bin.000001 mysql-bin.000003 mysql.sock xiexiaojun crm(live)# node standby BAIYU_180 crm(live)# node online BAIYU_179 # 切换至179 crm(live)# status Last updated: Wed Oct 28 03:58:43 2015 Last change: Wed Oct 28 03:58:41 2015 Stack: classic openais (with plugin) Current DC: BAIYU_179 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Node BAIYU_180: standby Online: [ BAIYU_179 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_179 ] Stopped: [ BAIYU_180 ] mydata (ocf::heartbeat:Filesystem): Started BAIYU_179 [root@BAIYU_179 ~]# service mysqld start 正在启动 mysqld： [确定] # 179节点的mysql不会进行初始化 [root@BAIYU_179 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.1.71-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create database xiena; Query OK, 1 row affected (0.00 sec) mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | | xiexiaojun | +--------------------+ 4 rows in set (0.02 sec) mysql> quit Bye
在179节点上创建的xiexiejun数据库在180上也显示了，说明使用的是共享存储。OK，drbd加入HA就成功了。
注意：另一个节点一定要先把drbd切换过去再启动mysqld，此时mysqld不会再初始化，要不然它会在本地初始化生成数据，造成数据错乱
最后一步就是将vip:192.168.100.27和mysqld加入HA就完成了；
crm(live)# configure crm(live)configure# primitive myip ocf:heartbeat:IPaddr params ip="192.168.100.27" op monitor in terval=10s teimeout=20s crm(live)configure# verify crm(live)configure# primitive myserver lsb:mysqld op monitor interval=20s timeout=20s crm(live)configure# verify crm(live)configure# colocation myip_with_ms_mystor_master inf: myip ms_mystor:Master crm(live)configure# colocation myserver_with_mydata inf: myserver mydata crm(live)configure# order myserver_after_mydata inf: mydata:start myserver:start crm(live)configure# verify crm(live)configure# order myserver_after_myip inf: myip:start myserver:start crm(live)configure# verify crm(live)configure# commit crm(live)configure# status ERROR: configure.status: No such command crm(live)configure# cd crm(live)# status Last updated: Wed Oct 28 08:36:34 2015 Last change: Wed Oct 28 08:36:28 2015 Stack: classic openais (with plugin) Current DC: BAIYU_180 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 5 Resources configured Online: [ BAIYU_179 BAIYU_180 ] Master/Slave Set: ms_mystor [mystor] Masters: [ BAIYU_180 ] Slaves: [ BAIYU_179 ] mydata (ocf::heartbeat:Filesystem): Started BAIYU_180 myip (ocf::heartbeat:IPaddr): Started BAIYU_180 myserver (lsb:mysqld): Started BAIYU_180
约束资源太繁杂了，用分组简单很多，详见上篇博文；
6、测试：
从远程主机连接192.168.100.27的mysql:
[root@node1 ~]# mysql -h 192.168.100.27 -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 5 Server version: 5.1.71-log Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | | xiexiaojun | +--------------------+ 4 rows in set (0.00 sec)
此时将资源切换到179；
crm(live)# node standby crm(live)# status Last updated: Wed Oct 28 08:54:30 2015 Last change: Wed Oct 28 08:54:23 2015 Stack: classic openais (with plugin) Current DC: BAIYU_180 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 5 Resources configured Node BAIYU_180: standby Online: [ BAIYU_179 ] Master/Slave Set: ms_mystor [mystor] # 资源切换需要一点时间 Masters: [ BAIYU_179 ] Stopped: [ BAIYU_180 ] mydata (ocf::heartbeat:Filesystem): Started BAIYU_179 myip (ocf::heartbeat:IPaddr): Started BAIYU_179 myserver (lsb:mysqld): Started BAIYU_179 crm(live)#
再在远程主机上查看：
mysql> SHOW DATABASES; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 2 Current database: *** NONE *** +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | | xiexiaojun | +--------------------+ 5 rows in set (20.06 sec) mysql>
OK.
附：配置DRBD报错及解决方法总结
Q1.'ha' ignored, since this host (node2.centos.bz) is not mentioned with an 'on' keyword.？
Error Meaage:
执行指令 drbdadm create-md ha 时出现如下错误信息
'ha' ignored, since this host (node2.centos.bz) is not mentioned with an 'on' keyword.
Ans:
因为在 drbd 设定 drbd.conf 中 on 写的是 node1、node2 ，将node1和node2分别改为node1.centos.bz,node2.centos.bz。
Q2.drbdadm create-md ha: exited with coolcode 20？
Error Meaage:
执行指令 drbdadm create-md ha 时出现如下错误信息
open(/dev/hdb1) failed: No such file or directory
Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit coolcode 20
drbdadm create-md ha: exited with coolcode 20
Ans:
因为忘了执行 fdisk /dev/hdb 指令建立分割区所造成，将 /dev/hdb 建立分割区后指令即可正常执行
#fdisk /dev/hdb //准备为 hdb 建立分割区
Q3.drbdadm create-md ha: exited with coolcode 40？
Error Meaage:
执行指令 drbdadm create-md ha 时出现如下错误信息
Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
* use external meta data (recommended)
* shrink that filesystem first
* zero out the device (destroy the filesystem)
Operation refused.
Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit coolcode 40
drbdadm create-md ha: exited with coolcode 40
Ans:
使用 dd 指令将一些资料塞到 /dev/hdb 后再执行 drbdadm create-md ha 指令即可顺利执行
#dd if=/dev/zero of=/dev/hdb1 bs=1M count=100
Q4.DRBD 状态始终是 Secondary/Unknown？
Error Meaage:
Node1、Node2 主机启动 DRBD 后状态始终是 Secondary/Unknown
#service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16
m:res cs ro ds p mounted fstype
0:ha WFConnection Secondary/Unknown Inconsistent/DUnknown C
Ans:
1、Node1、Node2 没有打开相对应的 Port，请开启相对应的 Port 或先把 IPTables 服务关闭即可。
2、可能发生了脑裂行为，一般出现在ha切换时，解决方法：
在一节点执行:
drbdadm secondary resource
drbdadm connect --discard-my-data resource
另一节点执行：
drbdadm connect resource
Q5. Failure: (104) Can not open backing device
Error Meaage:
执行drbdadm up r0时出现：
1: Failure: (104) Can not open backing device.
Command 'drbdsetup attach 1 /dev/sdb1 /dev/sdb1 internal' terminated with exit code 10
Ans:
可能因为你挂载了/dev/sdb1,执行umount /dev/sdb1即可。