Mysql+Corosync+Pacemaker+DRBD构建高可用Mysql

本次实验主要介绍Mysql的高可用集群构建；其他的不多说了，下面直接开始安装配置

一、环境介绍及准备

1、本次配置有两个节点：nod1.allen.com(172.16.14.1) 与 nod2.allen.com(172.16.14.2)

######在NOD1与NOD2节点执行如下命令
cat > /etc/hosts << EOF
172.16.14.1 nod1.allen.com nod1
172.16.14.2 nod2.allen.com nod2
EOF
注释：让所有节点的主机名称与对应的IP地址可以正常解析

2、每个节点的主机名称须跟"uname -n"命令的执行结果一样

######NOD1节点执行
sed -i 's@\(HOSTNAME=\).*@\1nod1.allen.com@g' /etc/sysconfig/network
hostname nod1.allen.com
######NOD2节点执行
sed -i 's@\(HOSTNAME=\).*@\1nod2.allen.com@g' /etc/sysconfig/network
hostname nod2.allen.com
注释：修改文件须重启系统生效，这里先修改文件然后执行命令修改主机名称可以不用重启

3、nod1与nod2两个节点上各提供了一个相同大小的分区作为DRBD设备，这里我们在两个节点上分别创建"/dev/sda3"作为DRBD设备，大小容量为2G

######在NOD1与NOD2节点上分别创建分区，分区大小必须保持一样
fdisk /dev/sda
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (7859-15665, default 7859):
Using default value 7859
Last cylinder, +cylinders or +size{K,M,G} (7859-15665, default 15665): +2G
Command (m for help): w
partx /dev/sda  #让内核重新读取分区
######查看内核有没有识别分区，如果没有需要重新启动，这里没有识别需要重启系统
cat /proc/partitions
major minor  #blocks  name
   8        0  125829120 sda
   8        1     204800 sda1
   8        2   62914560 sda2
 253        0   20971520 dm-0
 253        1    2097152 dm-1
 253        2   10485760 dm-2
 253        3   20971520 dm-3
reboot

4、关闭两台服务器的SELinux、Iptables与NetworkManager

setenforce 0            #关闭SELinux
service iptables stop   #关闭Iptables
chkconfig iptables off  #禁止Iptables开机启动
service NetworkManager stop
chkconfig NetworkManager off
chkconfig --list NetworkManager
NetworkManager  0:off   1:off   2:off   3:off   4:off   5:off   6:off
chkconfig network on
chkconfig --list network
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
注意：做的过程中必须关闭NetworkManager服务关闭并设置开机不能自动启动；将network服务设置开机自启动；否则作实验过程中会带来不必要的麻烦，造成集群系统不能正常运行

5、配置好YUM源并同步时间，且保证两个节点的时间要同步 epel源下载

######配置epel源
######在NOD1与NOD2节点分别安装
rpm -ivh epel-release-6-8.noarch.rpm

6、做双机互信

[root@nod1 ~]# ssh-keygen -t rsa
[root@nod1 ~]# ssh-copy-id -i .ssh/id_rsa.pub nod2
==================================================
[root@nod2 ~]# ssh-keygen -t rsa
[root@nod2 ~]# ssh-copy-id -i .ssh/id_rsa.pub nod1

7、系统版本：CentOS 6.4_x86_64

8、使用软件： 其中pacemaker与corosync在光盘映像中有

pssh-2.3.1-2.el6.x86_64 下载见附件

crmsh-1.2.6-4.el6.x86_64 下载见附件

drbd-8.4.3-33.el6.x86_64 DRBD下载地址：http://rpmfind.net

drbd-kmdl-2.6.32-358.el6-8.4.3-33.el6.x86_64

mysql-5.5.33-linux2.6-x86_64 点此下载

pacemaker-1.1.8-7.el6.x86_64

corosync-1.4.1-15.el6.x86_64

二、安装配置DRBD DRBD详解

1、在NOD1与NOD2节点上安装DRBD软件包

######NOD1
[root@nod1 ~]# ls drbd-*
drbd-8.4.3-33.el6.x86_64.rpm  drbd-kmdl-2.6.32-358.el6-8.4.3-33.el6.x86_64.rpm
[root@nod1 ~]# yum -y install drbd-*.rpm
######NOD2
[root@nod2 ~]# ls drbd-*
drbd-8.4.3-33.el6.x86_64.rpm  drbd-kmdl-2.6.32-358.el6-8.4.3-33.el6.x86_64.rpm
[root@nod2 ~]# yum -y install drbd-*.rpm

2、查看DRBD配置文件

ll /etc/drbd.conf;ll /etc/drbd.d/
-rw-r--r-- 1 root root 133 May 14 21:12 /etc/drbd.conf #主配置文件
total 4
-rw-r--r-- 1 root root 1836 May 14 21:12 global_common.conf #全局配置文件
######查看主配置文件内容
cat /etc/drbd.conf
######主配置文件中包含了全局配置文件及"drbd.d/"目录下以.res结尾的文件
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";

3、修改配置文件如下：

[root@nod1 ~]#vim /etc/drbd.d/global_common.conf
global {
    usage-count no;  #是否参加DRBD使用统计，默认为yes
    # minor-count dialog-refresh disable-ip-verification
}
common {
    protocol C;      #使用DRBD的同步协议
    handlers {
        # These are EXAMPLE handlers only.
        # They may have severe implications,
        # like hard resetting the node under certain circumstances.
        # Be careful when chosing your poison.
        pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
        # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
        # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
        # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
        # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
        # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
    }
    startup {
        # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
    }
    options {
        # cpu-mask on-no-data-accessible
    }
    disk {
        on-io-error detach; #配置I/O错误处理策略为分离
        # size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
        # disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
    }
    net {
        cram-hmac-alg "sha1";       #设置加密算法
        shared-secret "allendrbd"; #设置加密密钥
        # protocol timeout max-epoch-size max-buffers unplug-watermark
        # connect-int ping-int sndbuf-size rcvbuf-size ko-count
        # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
        # after-sb-1pri after-sb-2pri always-asbp rr-conflict
        # ping-timeout data-integrity-alg tcp-cork on-congestion
        # congestion-fill congestion-extents csums-alg verify-alg
        # use-rle
    }
    syncer {
        rate 1024M;    #设置主备节点同步时的网络速率
    }
}

4、添加资源文件:

[root@nod1 ~]# vim /etc/drbd.d/drbd.res
resource drbd {
  on nod1.allen.com {    #第个主机说明以on开头，后面是主机名称
    device    /dev/drbd0;#DRBD设备名称
    disk      /dev/sda3; #drbd0使用的磁盘分区为"sda3"
    address   172.16.14.1:7789; #设置DRBD监听地址与端口
    meta-disk internal;
  }
  on nod2.allen.com {
    device    /dev/drbd0;
    disk      /dev/sda3;
    address   172.16.14.2:7789;
    meta-disk internal;
  }
}

5、将配置文件为NOD2提供一份

[root@nod1 ~]# scp /etc/drbd.d/{global_common.conf,drbd.res} nod2:/etc/drbd.d/
The authenticity of host 'nod2 (172.16.14.2)' can't be established.
RSA key fingerprint is 29:d3:28:85:20:a1:1f:2a:11:e5:88:cd:25:d0:95:c7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nod2' (RSA) to the list of known hosts.
root@nod2's password:
global_common.conf                                                             100% 1943     1.9KB/s   00:00
drbd.res                                                                       100%  318     0.3KB/s   00:00

6、初始化资源并启动服务

######在NOD1与NOD2节点上初始化资源并启动服务
[root@nod1 ~]# drbdadm create-md drbd
Writing meta data...
initializing activity log
NOT initializing bitmap
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
New drbd meta data block successfully created.  #提示已经创建成功
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
######启动服务
[root@nod1 ~]# service drbd start
Starting DRBD resources: [
     create res: drbd
   prepare disk: drbd
    adjust disk: drbd
     adjust net: drbd
]
..........
***************************************************************
 DRBD's startup script waits for the peer node(s) to appear.
 - In case this node was already a degraded cluster before the
   reboot the timeout is 0 seconds. [degr-wfc-timeout]
 - If the peer was available before the reboot the timeout will
   expire after 0 seconds. [wfc-timeout]
   (These values are for resource 'drbd'; 0 sec -> wait forever)
 To abort waiting enter 'yes' [  12]: yes

7、初始化设备同步

[root@nod1 ~]# drbdadm -- --overwrite-data-of-peer primary drbd
[root@nod1 ~]# cat /proc/drbd     #查看同步进度
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-05-27 04:30:21
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-
    ns:1897624 nr:0 dw:0 dr:1901216 al:0 bm:115 lo:0 pe:3 ua:3 ap:0 ep:1 wo:f oos:207988
    [=================>..] sync'ed: 90.3% (207988/2103412)K
    finish: 0:00:07 speed: 26,792 (27,076) K/sec
######当同步完成时如以下状态
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-05-27 04:30:21
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:2103412 nr:0 dw:0 dr:2104084 al:0 bm:129 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
注释： drbd：为资源名称
######查看同步进度也可使用以下命令
drbd-overview

8、创建文件系统

######格式化文件系统
[root@nod1 ~]# mkfs.ext4 /dev/drbd0

9、禁止NOD1与NOD2节点上的DRBD服务开机自启动

[root@nod1 ~]# chkconfig drbd off
[root@nod1 ~]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
=====================================================================
[root@nod2 ~]# chkconfig drbd off
[root@nod2 ~]# chkconfig --list drbd
drbd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

三、安装Mysql

1、安装Mysql并配置

######在NOD1节点上安装Mysql
[root@nod1 ~]# mkdir /mydata
[root@nod1 ~]# mount /dev/drbd0 /mydata/
[root@nod1 ~]# mkdir /mydata/data
[root@nod1 ~]# tar xf mysql-5.5.33-linux2.6-x86_64.tar.gz -C /usr/local/
[root@nod1 ~]# cd /usr/local/
[root@nod1 local]# ln -s mysql-5.5.33-linux2.6-x86_64 mysql
[root@nod1 local]# cd mysql
[root@nod1 mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@nod1 mysql]# cp support-files/mysql.server /etc/init.d/mysqld
[root@nod1 mysql]# chmod +x /etc/init.d/mysqld
[root@nod1 mysql]# chkconfig --add mysqld
[root@nod1 mysql]# chkconfig mysqld off
[root@nod1 mysql]# vim /etc/my.cnf
datadir = /mydata/data
innodb_file_per_table = 1
[root@nod1 mysql]# echo "PATH=/usr/local/mysql/bin:$PATH" >> /etc/profile
[root@nod1 mysql]# . /etc/profile
[root@nod1 mysql]# useradd -r -u 306 mysql
[root@nod1 mysql]# chown mysql.mysql -R /mydata
[root@nod1 mysql]# chown root.mysql *
[root@nod1 mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/mydata/data/
[root@nod1 mysql]# service mysqld start
Starting MySQL.....                                        [  OK  ]
[root@nod1 mysql]# chkconfig --list mysqld
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@nod1 mysql]# service mysqld stop
Shutting down MySQL.                                       [  OK  ]

######在NOD2节点上安装Mysql
[root@nod2 ~]# scp nod1:/root/mysql-5.5.33-linux2.6-x86_64.tar.gz ./
[root@nod2 ~]# mkdir /mydata
[root@nod2 ~]# tar xf mysql-5.5.33-linux2.6-x86_64.tar.gz -C /usr/local/
[root@nod2 ~]# cd /usr/local/
[root@nod2 local]# ln -s mysql-5.5.33-linux2.6-x86_64 mysql
[root@nod2 local]# cd mysql
[root@nod2 mysql]# cp support-files/my-large.cnf /etc/my.cnf
######修改配置文件添加如下配置
[root@nod2 mysql]# vim /etc/my.cnf
datadir = /mydata/data
innodb_file_per_table = 1
[root@nod2 mysql]# cp support-files/mysql.server /etc/init.d/mysqld
[root@nod2 mysql]# chkconfig --add mysqld
[root@nod2 mysql]# chkconfig mysqld off
[root@nod2 mysql]# useradd -r -u 306 mysql
[root@nod2 mysql]# chown -R root.mysql *

2、卸载NOD1节点上的DRBD设备然后降级

[root@nod1 ~]# drbd-overview
  0:drbd/0  Connected Primary/Secondary UpToDate/UpToDate C r-----
[root@nod1 ~]# umount /mydata/
[root@nod1 ~]# drbdadm secondary drbd
[root@nod1 ~]# drbd-overview
  0:drbd/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----

3、在NOD2节点升级DBRD为主然后挂载DRBD设备

[root@nod2 ~]# drbd-overview
  0:drbd/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@nod2 ~]# drbdadm primary drbd
[root@nod2 ~]# drbd-overview
  0:drbd/0  Connected Primary/Secondary UpToDate/UpToDate C r-----
[root@nod2 ~]# mount /dev/drbd0 /mydata/

4、在NOD2节点上启动Mysql服务进行测试

[root@nod2 ~]# chown -R mysql.mysql /mydata
[root@nod2 ~]# service mysqld start
Starting MySQL..                                           [  OK  ]
[root@nod2 ~]# service mysqld stop
Shutting down MySQL.                                       [  OK  ]
[root@nod2 ~]# chkconfig --list mysqld
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off

5、将DRBD服务都设置为备用节点如：

[root@nod2 ~]# drbdadm secondary drbd
[root@nod2 ~]# drbd-overview
  0:drbd/0  Connected Secondary/Secondary UpToDate/UpToDate C r-----

6、卸载DRBD设备并停止NOD1与NOD2节点上的DRBD服务

[root@nod2 ~]# umount /mydata/
[root@nod2 ~]# service drbd stop
Stopping all DRBD resources: .
[root@nod1 ~]# service drbd stop
Stopping all DRBD resources: .

四、安装Corosync+Pacemaker软件

1、在NOD1与NOD2节点上安装

[root@nod1 ~]# yum -y install crmsh*.rpm pssh*.rpm pacemaker corosync
[root@nod2 ~]# scp nod1:/root/{pssh*.rpm,crmsh*.rpm} ./
[root@nod2 ~]# yum -y install crmsh*.rpm pssh*.rpm pacemaker corosync

2、在NOD1上配置Corosync

[root@nod1 ~]# cd /etc/corosync/
[root@nod1 corosync]# ls
corosync.conf.example  corosync.conf.example.udpu  service.d  uidgid.d
[root@nod1 corosync]# cp corosync.conf.example corosync.conf
[root@nod1 corosync]# vim corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
    version: 2    #版本号
    secauth: on   #是否开启安全认证
    threads: 0    #多少个现成认证，0 为无限制
    interface {
        ringnumber: 0
        bindnetaddr: 172.16.0.0 #通过哪个网络通信
        mcastaddr: 226.94.14.12 #组播地址
        mcastport: 5405         #组播端口
        ttl: 1
    }
}
logging {
    fileline: off
    to_stderr: no    #是否发送标准错误输出
    to_logfile: yes  #是否开启日志
    to_syslog: no    #是否开启系统日志，建议关闭一个
    logfile: /var/log/cluster/corosync.log #日志存放路径，须手动创建目录
    debug: off
    timestamp: on    #日志中是否记录时间
    logger_subsys {
        subsys: AMF
        debug: off
    }
}
amf {
    mode: disabled
}
service {                #添加支持使用Pacemaker
    ver:   0
    name:  pacemaker
}
aisexec {                #是否使用openais，有时可能会用到
    user:  root
    group: root
}

3、生成节点之间通信时用到的认证密钥文件

[root@nod1 corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Press keys on your keyboard to generate entropy (bits = 152).
Press keys on your keyboard to generate entropy (bits = 216).
注释：生成密钥时如果出现以上问题，说明随机数不够用，可以安装软件来解决

4、将配置文件及认证文件拷贝到NOD2节点一份

[root@nod1 corosync]# scp authkey corosync.conf nod2:/etc/corosync/
authkey                                    100%  128     0.1KB/s   00:00
corosync.conf                              100%  522     0.5KB/s   00:00

5、启动Corosync服务

[root@nod1 ~]# service corosync start
Starting Corosync Cluster Engine (corosync):               [  OK  ]
######查看corosync引擎是否正常启动
[root@nod1 ~]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log
Sep 19 18:44:36 corosync [MAIN  ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
Sep 19 18:44:36 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
######查看启动过程是否产生错误信息；如下信息可以忽略
[root@nod1 ~]# grep ERROR: /var/log/cluster/corosync.log
Sep 19 18:44:36 corosync [pcmk  ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon.
Sep 19 18:44:36 corosync [pcmk  ] ERROR: process_ais_conf:  Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN
######查看初始化成员节点通知是否正常发出
[root@nod1 ~]# grep  TOTEM  /var/log/cluster/corosync.log
Sep 19 18:44:36 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Sep 19 18:44:36 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Sep 19 18:44:36 corosync [TOTEM ] The network interface [172.16.14.1] is now up.
Sep 19 18:44:36 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
######查看pacemaker是否正常启动
[root@nod1 ~]# grep pcmk_startup /var/log/cluster/corosync.log
Sep 19 18:44:36 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
Sep 19 18:44:36 corosync [pcmk  ] Logging: Initialized pcmk_startup
Sep 19 18:44:36 corosync [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Sep 19 18:44:36 corosync [pcmk  ] info: pcmk_startup: Service: 9
Sep 19 18:44:36 corosync [pcmk  ] info: pcmk_startup: Local hostname: nod1.allen.com

6、启动NOD2节点上Corosync服务

[root@nod1 ~]# ssh nod2 'service corosync start'
Starting Corosync Cluster Engine (corosync): [  OK  ]
######查看集群节点启动状态
[root@nod1 ~]# crm status
Last updated: Thu Sep 19 19:01:33 2013
Last change: Thu Sep 19 18:49:09 2013 via crmd on nod1.allen.com
Stack: classic openais (with plugin)
Current DC: nod1.allen.com - partition with quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
0 Resources configured.
Online: [ nod1.allen.com nod2.allen.com ] #两个节点都已正常启动

7、查看Corosync启动的相关进程

[root@nod1 ~]# ps auxf
root     10336  0.3  1.2 556824  4940 ?        Ssl  18:44   0:04 corosync
305      10342  0.0  1.7  87440  7076 ?        S    18:44   0:01  \_ /usr/libexec/pacemaker/cib
root     10343  0.0  0.8  81460  3220 ?        S    18:44   0:00  \_ /usr/libexec/pacemaker/stonit
root     10344  0.0  0.7  73088  2940 ?        S    18:44   0:00  \_ /usr/libexec/pacemaker/lrmd
305      10345  0.0  0.7  85736  3060 ?        S    18:44   0:00  \_ /usr/libexec/pacemaker/attrd
305      10346  0.0  4.7 116932 18812 ?        S    18:44   0:00  \_ /usr/libexec/pacemaker/pengin
305      10347  0.0  1.0 143736  4316 ?        S    18:44   0:00  \_ /usr/libexec/pacemaker/crmd

五、配置资源

1、Corosync默认启用了Stonith，而当前集群并没有相应的Stonith，会出现以下错误；需要禁用Stonith

[root@nod1 ~]# crm_verify -L -V
   error: unpack_resources:     Resource start-up disabled since no STONITH resources have been defined
   error: unpack_resources:     Either configure some or disable STONITH with the stonith-enabled option
   error: unpack_resources:     NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
  -V may provide more details

######禁用Stonith并查看
[root@nod1 ~]# crm configure property stonith-enabled=false
[root@nod1 ~]# crm configure show
node nod1.allen.com
node nod2.allen.com
property $id="cib-bootstrap-options" \
    dc-version="1.1.8-7.el6-394e906" \
    cluster-infrastructure="classic openais (with plugin)" \
    expected-quorum-votes="2" \
    stonith-enabled="false"

2、查看当前的集群系统支持的类型

[root@nod1 ~]# crm ra classes
lsb
ocf / heartbeat linbit pacemaker redhat
service
stonith
注释：linbit 资源类型只有安装DRBD服务才会有

3、如何查看某种类型下所用可用的资源代理列表？

crm ra list lsb
crm ra list ocf heartbeat
crm ra list ocf pacemaker
crm ra list stonith
crm ra list ocf linbit

4、配置VIP资源与Mysqld资源

[root@nod1 ~]# crm        #进入crm交互模式
crm(live)# configure
crm(live)configure# property no-quorum-policy="ignore"
crm(live)configure# primitive MyVip ocf:heartbeat:IPaddr params ip="172.16.14.10"    #定义虚拟IP资源
crm(live)configure# primitive Mysqld lsb:mysqld #定义Mysql服务资源
crm(live)configure# verify     #检查语法错误
crm(live)configure# commit     #提交
crm(live)configure# show       #查看配置
node nod1.allen.com
node nod2.allen.com
primitive MyVip ocf:heartbeat:IPaddr \
    params ip="172.16.14.10"
primitive Mysqld lsb:mysqld
property $id="cib-bootstrap-options" \
    dc-version="1.1.8-7.el6-394e906" \
    cluster-infrastructure="classic openais (with plugin)" \
    expected-quorum-votes="2" \
    stonith-enabled="false" \
    no-quorum-policy="ignore"

5、配置DRBD主从资源

crm(live)configure# primitive Drbd ocf:linbit:drbd params drbd_resource="drbd" op monitor interval=10s role="Master" op monitor interval=20s role="Slave" op start timeout=240s op stop timeout=100
crm(live)configure# master My_Drbd Drbd meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show Drbd
primitive Drbd ocf:linbit:drbd \
    params drbd_resource="drbd" \
    op monitor interval="10s" role="Master" \
    op monitor interval="20s" role="Slave" \
    op start timeout="240s" interval="0" \
    op stop timeout="100s" interval="0"
crm(live)configure# show My_Drbd
ms My_Drbd Drbd \
    meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

6、定义一个文件系统资源

crm(live)configure# primitive FileSys ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mydata" fstype="ext4" op start timeout="60s" op stop timeout="60s"
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show FileSys
primitive FileSys ocf:heartbeat:Filesystem \
    params device="/dev/drbd0" directory="/mydata" fstype="ext4" \
    op start timeout="60s" interval="0" \
    op stop timeout="60s" interval="0"

7、定将资源之间的位置和启动顺序约束

crm(live)configure# colocation FileSys_on_My_Drbd inf: FileSys My_Drbd:Master #让文件系统与DRBD主节点运行在一起
crm(live)configure# order FileSys_after_My_Drbd inf: My_Drbd:promote FileSys:start  #让DRBD服务比文件系统先启动
crm(live)configure# verify
crm(live)configure# colocation Mysqld_on_FileSys inf: Mysqld FileSys #让Mysql服务与文件系统运行在一起
crm(live)configure# order Mysqld_after_FileSys inf: FileSys Mysqld:start #让文件系统比Mysql服务先运行
crm(live)configure# verify
crm(live)configure# colocation MyVip_on_Mysqld inf: MyVip Mysqld #让虚拟IP与Mysql服务运行在一起
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# bye #断开crm交互连接

8、查看服务状态如下：

[root@nod1 ~]# crm status
Last updated: Thu Sep 19 21:18:20 2013
Last change: Thu Sep 19 21:18:06 2013 via crmd on nod1.allen.com
Stack: classic openais (with plugin)
Current DC: nod2.allen.com - partition with quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
5 Resources configured.
Online: [ nod1.allen.com nod2.allen.com ]
 Master/Slave Set: My_Drbd [Drbd]
     Masters: [ nod2.allen.com ]
     Slaves: [ nod1.allen.com ]
 FileSys    (ocf::heartbeat:Filesystem):    Started nod2.allen.com
Failed actions:
    Mysqld_start_0 (node=nod1.allen.com, call=60, rc=1, status=Timed Out): unknown error
    MyVip_start_0 (node=nod2.allen.com, call=47, rc=1, status=complete): unknown error
    Mysqld_start_0 (node=nod2.allen.com, call=13, rc=1, status=complete): unknown error
    FileSys_start_0 (node=nod2.allen.com, call=39, rc=1, status=complete): unknown error
注释：出现以上错误是因为我们在定义资源提交时，期间会检测服务是否运行；如果没有运行可能会尝试启动，而资源还没有完全定义好，所以会报错误；执行如下命令清除错误即可
[root@nod1 ~]# crm resource cleanup Mysqld
[root@nod1 ~]# crm resource cleanup MyVip
[root@nod1 ~]# crm resource cleanup FileSys

9、在上一步清除完错误后再次查看：

[root@nod1 ~]# crm status
Last updated: Thu Sep 19 21:26:49 2013
Last change: Thu Sep 19 21:19:35 2013 via crmd on nod2.allen.com
Stack: classic openais (with plugin)
Current DC: nod2.allen.com - partition with quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
5 Resources configured.
Online: [ nod1.allen.com nod2.allen.com ]
 Master/Slave Set: My_Drbd [Drbd]
     Masters: [ nod1.allen.com ]
     Slaves: [ nod2.allen.com ]
 MyVip  (ocf::heartbeat:IPaddr):    Started nod1.allen.com
 Mysqld (lsb:mysqld):   Started nod1.allen.com
 FileSys    (ocf::heartbeat:Filesystem):    Started nod1.allen.com
======================================================================
注释：由上可见，DRBD_Master、MyVip、Mysqld、FileSys都运行在NOD1节点上，也已经正常运行

六、验证服务运行是否正常

1、在NOD1节点上查看是否已经运行Mysqld服务及配置好虚拟IP地址和文件系统

[root@nod1 ~]# netstat -anpt|grep mysql
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      22564/mysqld
[root@nod1 ~]# mount | grep drbd0
/dev/drbd0 on /mydata type ext4 (rw)
[root@nod1 ~]# ifconfig eth0:0
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:3D:3F:44
          inet addr:172.16.14.10  Bcast:172.16.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

2、登录数据库并创建数据库用于验证

[root@nod1 ~]# mysql
mysql> create database allen;
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| allen              |
| mysql              |
| performance_schema |
| test               |
+--------------------+

3、模拟主节点出现故障,将主节点设置为"Standby"状态，查看服务是否转移到备用节点上；当前主节点为：nod1.allen.com 备用节点：nod2.allen.com

[root@nod1 ~]# crm node standby nod1.allen.com
[root@nod1 ~]# crm status
Last updated: Thu Sep 19 22:23:50 2013
Last change: Thu Sep 19 22:23:42 2013 via crm_attribute on nod2.allen.com
Stack: classic openais (with plugin)
Current DC: nod1.allen.com - partition with quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
5 Resources configured.
Node nod1.allen.com: standby
Online: [ nod2.allen.com ]
 Master/Slave Set: My_Drbd [Drbd]
     Masters: [ nod2.allen.com ]
     Stopped: [ Drbd:1 ]
 MyVip  (ocf::heartbeat:IPaddr):    Started nod2.allen.com
 Mysqld (lsb:mysqld):   Started nod2.allen.com
 FileSys    (ocf::heartbeat:Filesystem):    Started nod2.allen.com
----------------------------------------------------------------------
######由上可见，所有服务已经切换到NOD2节点服务器上面

4、在NOD2节点上登录Mysql验证是否有"allen"数据库

[root@nod2 ~]# mysql
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| allen              |
| mysql              |
| performance_schema |
| test               |
+--------------------+

5、假如NOD1已修复好重新上线；这时NOD2节点上的服务是不会重新切换回NOD1节点上面的；如果想让切换也不是不可以，这需要设置资源粘性；但建议不要切换，避免服务切换时浪费不必要的资源

[root@nod1 ~]# crm node online nod1.allen.com
[root@nod1 ~]# crm status
Last updated: Thu Sep 19 22:34:55 2013
Last change: Thu Sep 19 22:34:51 2013 via crm_attribute on nod1.allen.com
Stack: classic openais (with plugin)
Current DC: nod1.allen.com - partition with quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
5 Resources configured.
Online: [ nod1.allen.com nod2.allen.com ]
 Master/Slave Set: My_Drbd [Drbd]
     Masters: [ nod2.allen.com ]
     Slaves: [ nod1.allen.com ]
 MyVip  (ocf::heartbeat:IPaddr):    Started nod2.allen.com
 Mysqld (lsb:mysqld):   Started nod2.allen.com
 FileSys    (ocf::heartbeat:Filesystem):    Started nod2.allen.com

6、设置资源粘性命令；这里就不在做测试了，如果各位博友有兴趣可以测试一下

crm configure rsc_defaults resource-stickiness=100

由上可见，所有服务都可以正常工作；到这里Mysql高可用已全部完成，而且还验证了Mysql服务的正常运行与数据；在这里感谢广大博友的关注与支持，我会继续努力的！！！ 加油 加油 加油