Mysql-5.5+Heartbeat-3.0.5+DRBD

环境：

CentOS 6.5

MySQL_Master  

eth0  192.168.1.10

eth1  192.168.2.10        

MySQL_Slave    

eth0  192.168.1.11

eth1  192.168.2.11

HA    192.168.1.254

------------------------------------------------

――创建RAID1+0组合[存储数据 四块硬盘]

1）创建两块RAID1 [4块硬盘、MySQL主从节点执行]

[root@master ~]# mdadm --create /dev/md0 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@master ~]# mdadm --create /dev/md1 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.

2）利用两个 RAID1 创建 RAID0

[root@CentOS ~]# mdadm --create /dev/md2 --level=raid0 --raid-devices=2 /dev/md0 /dev/md1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.

3）将raid信息写入配置文件

[root@master ~]# mdadm --detail --scan > /etc/mdadm.conf       
[root@master ~]# vi /etc/mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=CentOS:0 UUID=b4c4c7b4:0f9f6e60:7eb24578:29682c96
devices /dev/sdb1 /dev/sdc1
ARRAY /dev/md1 metadata=1.2 name=CentOS:1 UUID=f5afcda6:86847677:c752fcdd:fbb91e00
devices /dev/sdd1 /dev/sde1
ARRAY /dev/md2 metadata=1.2 name=CentOS:2 UUID=00f120ec:bab2f3fe:80d88cb9:3ee4b76b
devices /dev/md0 /dev/md1

――安装DRBD

1）解压并安装DRBD

[root@master Linux]# yum -y install gcc kernel-devel kernel-headers flex perl
[root@master Linux]# http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
[root@master Linux]# tar fzvx drbd-8.4.4.tar.gz
[root@master Linux]# cd drbd-8.4.4
[root@master drbd-8.4.4]# ./configure --prefix=/usr/local/drbd-8.4 --with-km
[root@master drbd-8.4.4]# make KDIR=/usr/src/kernels/2.6.32-358.el6.x86_64/
[root@master drbd-8.4.4]# make install
[root@master drbd-8.4.4# mkdir -p /usr/local/drbd-8.4/var/run/drbd
[root@master drbd-8.4.4]# cp /usr/local/drbd-8.4/etc/rc.d/init.d/drbd /etc/init.d/
[root@master drbd-8.4.4]# chkconfig --add drbd
[root@master drbd-8.4.4]# chkconfig drbd on

2）安装drbd模块

[root@master drbd-8.4.4]# cd drbd
[root@master drbd]# make clean
[root@master drbd]# make KDIR=/usr/src/kernels/2.6.32-358.el6.x86_64/
[root@master drbd]# cp drbd.ko /lib/modules/2.6.32-358.el6.x86_64/kernel/lib/
[root@master drbd]# depmod

3）配置global_common.conf

[root@master drbd]# cd /usr/local/drbd-8.4/etc/drbd.d/
[root@master drbd.d]# cp global_common.conf global_common.conf.bak
[root@master drbd.d]# vi global_common.conf
global {
        usage-count no;
}
common {
        handlers {
                  pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                  pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                  local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                  fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                  out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
        }
        startup {
                 wfc-timeout 30;
                 degr-wfc-timeout 30;
                 outdated-wfc-timeout 30;
        }
        disk {
　　            #磁盘读写速度与同步速率的30%
                resync-rate     30M;
                on-io-error     detach;
                fencing         resource-only;
        }
        net {
                protocol C;
                cram-hmac-alg   sha1;
                shared-secret   "mysql-ha";
                csums-alg       sha1;
                verify-alg crc32c;
        }
}

4）创建r0资源

[root@master drbd.d]# vi r0.res
resource r0{
        on master{
                device          /dev/drbd0; #逻辑设备的路径
                disk            /dev/md2;   #物理设备
                address         192.168.2.10:7788;
                meta-disk       internal;
        }
        on slave{
                device          /dev/drbd0;
                disk            /dev/md2;
                address         192.168.2.11:7788;
                meta-disk       internal;
        }
}

5）建立 drbd resource

[root@master drbd.d]# modprobe drbd
[root@master drbd.d]# drbdadm create-md r0
[root@master drbd.d]# drbdadm up r0

6）设置Primary [在master节点操作]

[root@master drbd.d]# drbdadm primary --force r0

7）创建DRBD文件系统 [在Mysql主节点的master上执行]

[root@master drbd.d]# mkfs.ext4 /dev/drbd0
[root@master drbd.d]# mount /dev/drbd0 /raid10/

8）DRBD同步测试

1、首先，在主服务器上先将设备卸载，同时将主服务器降为备用服务器：

[root@master drbd]# mkdir -p /raid10/mysql/data
[root@master drbd]# cd /
[root@master /]# umount /dev/drbd0
[root@master /]# drbdadm secondary r0

2、然后，登录备用服务器，将备用服务器升为主服务器，同时挂载drbd0设备到 /raid10目录：

[root@slave drbd]# drbdadm up r0
[root@slave drbd]# drbdadm primary r0
[root@slave drbd]# mount /dev/drbd0 /raid10/
[root@slave drbd]# cd /raid10/
[root@slave raid10]# ls
lost+found  mysql

――使用中出现脑裂以及解决办法

[root@slave ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@slave, 2013-12-03 09:50:30 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/Outdated   r-----
ns:0 nr:0 dw:2 dr:1684 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4

2）查看日志

[root@slave /]# tail -f /var/log/messages
Dec  3 11:06:05 slave kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code
127 (0x7f00)
Dec  3 11:06:05 slave kernel: drbd r0: conn( WFReportParams -> Disconnecting )
Dec  3 11:06:05 slave kernel: drbd r0: error receiving ReportState, e: -5 l: 0!
Dec  3 11:06:05 slave kernel: drbd r0: asender terminated
Dec  3 11:06:05 slave kernel: drbd r0: Terminating drbd_a_r0
Dec  3 11:06:05 slave kernel: drbd r0: Connection closed
Dec  3 11:06:05 slave kernel: drbd r0: conn( Disconnecting -> StandAlone )
Dec  3 11:06:05 slave kernel: drbd r0: receiver terminated
Dec  3 11:06:05 slave kernel: drbd r0: Terminating drbd_r_r0
Dec  3 11:06:41 slave kernel: block drbd0: role( Secondary -> Primary )

――解决方法：

1）我们需要将现在的master角色修改为secondary

[root@master ~]# drbdadm secondary r0
#该命令告诉slave，secondary上的数据不正确，以primary上的数据为准。
[root@master ~]# drbdadm -- --discard-my-data connect r0

2）我们还需要在slave上执行下面操作

#这样master就能和slave开始连接上了，并且保证数据不会丢失：
[root@slave ~]# drbdadm connect r0
[root@slave ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@slave, 2013-12-03 09:50:30
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:4 dw:6 dr:1688 al:1 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@master ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@master, 2013-12-03 09:49:22
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:4 nr:0 dw:1 dr:1015 al:1 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

――安装mysql

1）mastr节点

[root@master Linux]# yum install gcc gcc-c++ autoconf automake ncurses-devel libtool-ltdl-devel* -y
[root@master Linux]# useradd -M -s /sbin/nologin mysql
[root@master Linux]# tar zfvx cmake-2.8.12.tar.gz
[root@master Linux]# cd cmake-2.8.12
[root@master cmake-2.8.12]# ./configure
[root@master cmake-2.8.12]# gmake && make install
[root@master cmake-2.8.12]# cd ..
[root@master Linux]# tar zfxv mysql-5.5.25.tar.gz
[root@master Linux]# cd mysql-5.5.25
[root@master mysql-5.5.25]# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql-5.5
[root@master mysql-5.5.25]# make && make install
[root@master mysql-5.5.25]# cp support-files/my-medium.cnf  /raid10/mysql/my.cf
[root@master mysql-5.5.25]# rm -rf /etc/my.cnf
[root@master mysql-5.5.25]# ln -sv  /raid10/mysql/my.cf /etc/
[root@master mysql-5.5.25]# cd /usr/local/mysql-5.5/
[root@master mysql-5.5]# chown -R root:mysql .
[root@master mysql-5.5]# chown -R mysql:mysql /raid10/mysql/data/
[root@master mysql-5.5]# ./scripts/mysql_install_db --user=mysql \
--basedir=/usr/local/mysql-5.5/ \
--datadir=/raid10/mysql/data/
[root@master mysql-5.5]# cp support-files/mysql.server /etc/init.d/mysqld
[root@master mysql-5.5]# chmod +x /etc/init.d/mysqld
[root@master mysql-5.5]# chkconfig --add mysqld
[root@master mysql-5.5]# vi /etc/init.d/mysqld
datadir=/raid10/mysql/data

2）savle节点安装

――安装mysql[MySQL主节点的savle节点安装]
[root@slave Linux]# yum install gcc gcc-c++ autoconf automake ncurses-devel libtool-ltdl-devel* -y
[root@slaveLinux]# useradd -M -s /sbin/nologin mysql
[root@slave Linux]# tar zfvx cmake-2.8.12.tar.gz
[root@slave Linux]# cd cmake-2.8.12
[[email protected]]# ./configure
[[email protected]]# gmake && make install
[[email protected]]# cd ..
[root@slave Linux]# tar zfxv mysql-5.5.25.tar.gz
[root@slave Linux]# cd mysql-5.5.25
[root@slave mysql-5.5.25]# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql-5.5
[root@slave mysql-5.5.25]# make && make install
[root@slave mysql-5.5.25]# cd /usr/local/mysql-5.5/
[root@slave mysql-5.5]# chown -R root:mysql .
[root@slave mysql-5.5]# cp support-files/mysql.server /etc/init.d/mysqld
[root@slave mysql-5.5]# vi /etc/init.d/mysqld
datadir=/raid10/mysql/data

――MySQL主节点实现高可用

注意：以下操作在MySQL主节点的master、slave节点执行

1）添加hosts主机信息

[root@master Linux]# vi /etc/hosts
192.168.2.10    master
192.168.2.11    slave

2）添加用户和组

[root@master Linux]# groupadd haclient
[root@master Linux]# useradd -g haclient -M -s /sbin/nologin hacluster

3）安装heartbeat

1.安装相关软件依赖包

[root@master Linux]# yum install libtool automake autoconf \
glib2-devel \
libxml2-devel \
bzip2-devel \
libtool-ltdl-devel \
libxslt-devel \
docbook* -y

2.安装glue

[root@master Linux]# wget http://hg.linux-ha.org/glue/archive/glue-1.0.9.tar.bz2
[root@master Linux]# tar jfvx glue-1.0.9.tar.bz2
[root@master Linux]# cd Reusable-Cluster-Components-glue--glue-1.0.9/
[root@master Reusable-Cluster-Components-glue--glue-1.0.9]# ./autogen.sh
[root@master Reusable-Cluster-Components-glue--glue-1.0.9]# ./configure LIBS='/lib64/libuuid.so.1'
[root@CentOS Reusable-Cluster-Components-glue--glue-1.0.9]# make && make install

3.安装agents

[root@master Heartbeat-3-0-7e3a82377fa8]# cd ..
[root@master Linux]# wget https://codeload.github.com/ClusterLabs/resource-agents/legacy.tar.gz/v3.9.2
[root@master Linux]# tar zfvx ClusterLabs-resource-agents-v3.9.2-0-ge261943.tar.gz
[root@master Linux]# cd ClusterLabs-resource-agents-b735277/
[root@master ClusterLabs-resource-agents-b735277]# ./autogen.sh
[root@master ClusterLabs-resource-agents-b735277]# ./configure LIBS='/lib64/libuuid.so.1'
[root@master ClusterLabs-resource-agents-b735277]# make && make install

4.安装heartbeat

[root@master ClusterLabs-resource-agents-b735277]# cd ..
[root@master Linux]# wget http://hg.linux-ha.org/heartbeat-STABLE_3_0/archive/7e3a82377fa8.tar.bz2
[root@master Linux]# tar jfvx heartbeat-3.0.5.tar.bz2
[root@master Linux]# cd Heartbeat-3-0-7e3a82377fa8/
[root@master Heartbeat-3-0-7e3a82377fa8]# ./bootstrap
[root@master Heartbeat-3-0-7e3a82377fa8]# ./ConfigureMe configure LIBS='/lib64/libuuid.so.1'
[root@master Heartbeat-3-0-7e3a82377fa8]# make && make install

4）配置heartbeat

[root@master Heartbeat-3-0-7e3a82377fa8]# cd doc
[root@master doc]# cp authkeys haresources ha.cf /etc/ha.d/
[root@master doc]# cp /usr/etc/ha.d/shellfuncs /etc/ha.d/
[root@master doc]# rm -rf /usr/etc/ha.d
[root@master doc]# ln -sv /etc/ha.d /usr/etc
[root@master doc]# vi /etc/ha.d/ha.cf
#开启日志
logfile /var/log/ha-log
#设置syslog()/logger设备
logfacility     local0
#心跳发送时间间隔/秒
keepalive 2
#15秒没有收到主机心跳、确认主机故障
deadtime 15
#警告次数
warntime 5
#守护进程启动30后 启动服务资源
initdead 30
#监听端口
udpport 694
#另一个节点IP、通过检测来保证心跳的可用性
ucast eth0 192.168.1.11
ucast eth1 192.168.2.11
#两个节点的名字 [uname -n 获取]
node    master
node    slave
#开启DPOD
respawn hacluster /usr/lib64/heartbeat/ipfail
respawn hacluster /usr/lib64/heartbeat/dopd
apiauth ipfail gid=haclient uid=hacluster
apiauth dopd gid=haclient uid=hacluster
[root@master doc]# vi /etc/ha.d/authkeys
auth 1
1 sha1 HI!
[root@master doc]# chmod 600 /etc/ha.d/authkeys
[root@master doc]# vi /etc/ha.d/haresources
master drbddisk::r0 Filesystem::/dev/drbd0::/raid10::ext4 mysqld IPaddr::192.168.1.254/24/eth0
[root@master doc]# cd /Linux/drbd-8.4.4/scripts/
[root@master scripts]# cp drbddisk /etc/ha.d/resource.d/
[root@master scripts]# cp /etc/init.d/mysqld /etc/ha.d/resource.d/
[root@master scripts]# chkconfig --add heartbeat
[root@master scripts]# chkconfig heartbeat on

――测试

1）启动master节点启动heartbeat

[root@master scripts]# /etc/init.d/heartbeat start

2）启动slave节点启动heartbeat

[root@slave scripts]# /etc/init.d/heartbeat start

3）在master节点上查看启动日志

[root@master ha.d]# tail -f /var/log/ha-log
Apr 26 21:11:48 master heartbeat: [42033]: info: Starting child client "/usr/lib64/heartbeat/ipfail" (501,501)
Apr 26 21:11:48 master heartbeat: [42033]: info: Starting child client "/usr/lib64/heartbeat/dopd" (501,501)
Apr 26 21:11:48 mster heartbeat: [42062]: info: Starting "/usr/lib64/heartbeat/ipfail" as uid 501  gid 501 (pid 42062)
Apr 26 21:11:48 mster heartbeat: [42064]: info: Starting "/usr/lib64/heartbeat/dopd" as uid 501  gid 501 (pid 42064)
Apr 26 21:11:48 mster heartbeat: [42061]: info: Local Resource acquisition completed.
Apr 26 21:11:48 mster heartbeat: [42033]: info: Initial resource acquisition complete (req_our_resources)
Apr 26 21:11:48 mster ipfail: [42062]: ERROR: auto_failback set to incompatible legacy option.
Apr 26 21:11:48 mster heartbeat: [42033]: WARN: Managed /usr/lib64/heartbeat/ipfail process 42062 exited with return code 100.
Apr 26 21:11:48 mster heartbeat: [42033]: info: Status update for slave: status active
harc[42104]:    2014/04/26_21:11:48 info: Running /usr/etc/ha.d//rc.d/status status
#说明启动成功

4）切换主备

1.在master节点停掉heartbeat

[root@master scripts]# /etc/init.d/heartbeat stop

2.查看slave日志是否能自动切换至slave节点

[root@slave scripts]# tail -f /var/log/ha-log
#显示master节点已经shutdown
Apr 26 21:21:27 slave heartbeat: [42054]: info: Received shutdown notice from 'master'.
Apr 26 21:21:27 slave heartbeat: [42054]: info: Resources being acquired from master.
harc[42118]:    2014/04/26_21:21:27 info: Running /usr/etc/ha.d//rc.d/status status
Apr 26 21:21:27 slave heartbeat: [42119]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys slave] to acquire.
mach_down[42148]:       2014/04/26_21:21:27 info: Taking over resource group drbddisk::r0
ResourceManager[42175]: 2014/04/26_21:21:27 info: Acquiring resource group: master drbddisk::r0 Filesystem::/dev/drbd0::/raid10::ext4 mysqld IPaddr::192.168.1.254/24/eth0
ResourceManager[42175]: 2014/04/26_21:21:27 info: Running /etc/ha.d/resource.d/drbddisk r0 start
Filesystem[42239]:      2014/04/26_21:21:27 INFO:  Resource is stopped
ResourceManager[42175]: 2014/04/26_21:21:27 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /raid10 ext4 start
Filesystem[42320]:      2014/04/26_21:21:27 INFO: Running start for /dev/drbd0 on /raid10
Filesystem[42312]:      2014/04/26_21:21:28 INFO:  Success
ResourceManager[42175]: 2014/04/26_21:21:28 info: Running /etc/ha.d/resource.d/mysqld  start
Apr 26 21:21:39 slave heartbeat: [42054]: WARN: node master: is dead
Apr 26 21:21:39 slave heartbeat: [42054]: info: Dead node master gave up resources.
Apr 26 21:21:39 slave heartbeat: [42054]: info: Resources being acquired from master.
Apr 26 21:21:39 slave heartbeat: [42054]: info: Link master:eth0 dead.
Apr 26 21:21:39 slave heartbeat: [42054]: info: Link master:eth1 dead.
Apr 26 21:21:41 slave heartbeat: [42614]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys slave] to acquire.
Apr 26 21:21:41 slave heartbeat: [42054]: info: Initial resource acquisition complete (req_our_resources)
IPaddr[42642]:  2014/04/26_21:21:42 INFO:  Resource is stopped
#显示slave节点的VIP已经运行
ResourceManager[42175]: 2014/04/26_21:21:42 info: Running /etc/ha.d/resource.d/IPaddr 192.168.1.254/24/eth0 start
IPaddr[42727]:  2014/04/26_21:21:42 INFO: Using calculated netmask for 192.168.1.254: 255.255.255.0
IPaddr[42727]:  2014/04/26_21:21:42 INFO: eval ifconfig eth0:0 192.168.1.254 netmask 255.255.255.0 broadcast 192.168.1.255
IPaddr[42701]:  2014/04/26_21:21:42 INFO:  Success
mach_down[42148]:       2014/04/26_21:21:42 info: mach_down takeover complete for node master.
harc[42820]:    2014/04/26_21:21:42 info: Running /usr/etc/ha.d//rc.d/status status
mach_down[42837]:       2014/04/26_21:21:42 info: Taking over resource group drbddisk::r0
ResourceManager[42864]: 2014/04/26_21:21:43 info: Acquiring resource group: master drbddisk::r0 Filesystem::/dev/drbd0::/raid10::ext4 mysqld IPaddr::192.168.1.254/24/eth0
#显示slave的mysql和drbd服务已经启动成功
Filesystem[42906]:      2014/04/26_21:21:43 INFO:  Running OK
IPaddr[42987]:  2014/04/26_21:21:43 INFO:  Running OK
mach_down[42837]:       2014/04/26_21:21:43 info: mach_down takeover complete for node master.