mysql+drbd+heartbeat

1．Drbd是基于块设备的高可用集群，所以在使用drbd 之前我需要创建一个分区（mysql-a和mysql-b都要去创建分区。分区为/dev/sdb1

当然分区也可以一样，也可以不一样，可以在drbd的配置文件指定。

第一步，节点上分区

[root@mysql-a /]# fdisk -l /dev/sdb

Disk /dev/sdb: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0xf5c6ec8b

 

   Device Boot      Start        End      Blocks  Id  System

/dev/sdb1              1        2610    20964793+ 83  Linux

[root@mysql-b /]# fdisk -l /dev/sdb

 

Disk /dev/sdb: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x274e7b9c

 

   Device Boot      Start         End      Blocks  Id  System

/dev/sdb1              1        2610    20964793+ 83  Linux

第二步安装drbd

解压后

安装之前的准备工作

参考资料 

http://drbd.linbit.com/users-guide-8.4/ap-man-pages.html   drbd的手册页 主要包括drbd.confConfigurationfile for DRBD's devices

drbdadmAdministrationtool for DRBD

drbdsetupSetuptool for DRBD

drbdmetaDRBD'smeta data management tool

http://drbd.linbit.com/users-guide-8.4/  版本手册

http://drbd.linbit.com/users-guide-8.4/s-build-from-source.html#s-build-prepare-checkout安装参考资料

阅读完安装手册进行安装

make, gcc, the glibcdevelopment libraries, and the flex scanner generator must be installed

[root@mysql-a drbd-8.4.4]#  ./configure --prefix=/usr/local/drbd--localstatedir=/usr/local/ --sysconfdir=/etc --with-km

[root@mysql-a drbd-8.4.4]# make && make install

CompilingDRBD as a kernel module

[root@mysql-a drbd-8.4.4]# cd drbd

[root@mysql-a drbd]# make clean all 

Buildingagainst precompiled kernel headers

查看当前我正在用的内核，将机器重启看到我又更新内核

我使用的是2.6.32.-504.8.1

[root@mysql-a drbd]# make clean

rm -rf .tmp_versions Module.markers Module.symversmodules.order

rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

rm -f compat/*.[oas] compat/.*NaNd

[root@mysql-a drbd]# make KDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/build/

Buildingagainst a kernel source tree

[root@mysql-a drbd]# make clean

rm -rf .tmp_versions Module.markers Module.symversmodules.order

rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

rm -f compat/*.[oas] compat/.*NaNd

[root@mysql-a drbd]# makeKDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/source

 

Using anon-default C compiler

[root@mysql-a drbd]# make clean

rm -rf .tmp_versions Module.markers Module.symvers modules.order

rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

rm -f compat/*.[oas] compat/.*NaNd

[root@mysql-a drbd]# make CC=gcc

检查

If themodule build completes successfully, you should see a kernel module file named drbd.ko in the drbd directory. You mayinterrogate the newly-built module with /sbin/modinfo drbd.ko if you are soinclined.

[root@mysql-a kernels]# find / -namedrbd.ko

/usr/local/src/drbd-8.4.4/drbd/drbd.ko

/lib/modules/2.6.32-504.8.1.el6.x86_64/updates/drbd.ko

查看drbd的配置文件，因为我安装的时候指定drbd配置文件的路径为/etc/可以直接去/etc/修改

[root@mysql-a kernels]# cd /etc/dr

dracut.conf   dracut.conf.d/ drbd.conf     drbd.d/       

[root@mysql-a kernels]# cd /etc/drbd.conf

[root@mysql-a /]# ls /etc/init.d/drbd   也不需要去cp启动脚本了 发现这样好方便

/etc/init.d/drbd

[root@mysql-a /]#

修改drbd的配置文件

http://drbd.linbit.com/users-guide-8.4/re-drbdconf.html   drbd.conf 官网说明

配置参考

http://drbd.linbit.com/users-guide/s-configure-resource.html#s-drbdconf-example 

 

[root@mysql-a /]# vim /etc/drbd.conf

# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

The file /etc/drbd.conf is read bydrbdadm.   Drbdadm 是读取drbd.conf 这个主配置文件的，但是默认drbd里面又是读取

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

这两个文件，我们可以去找到drbd.conf.example 模板直接修改drbd.conf 可以分别去修改这个文件

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

然后在drbd.conf 启用加载这两个文件就可以了。

[root@mysql-a /]#find / -name drbd.conf.example

/usr/local/src/drbd-8.4.4/scripts/drbd.conf.example

[root@mysql-a /]#cp /usr/local/src/drbd-8.4.4/scripts/drbd.conf.example /etc/drbd.conf

cp: overwrite`/etc/drbd.conf'? y

drbd不建议使用模板的drbd.conf 去配置，那样的配置比较混乱，容易出错。

drbd 参数介绍

[root@mysql-a ~]# scp /etc/drbd.d/*  20.0.0.36:/etc/drbd.d/     将修改的drbd.d 下的配置文件拷贝到备节点上启动drbd测试

初始化元数据 两个节点都需要执行  http://drbd.linbit.com/users-guide-8.4/s-first-time-up.html

drbd的配置文件参数

[root@mysql-aetc]# cat /etc/drbd.conf

# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

 

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

[root@mysql-aetc]# egrep -v "#|^$" /etc/drbd.d/global_common.conf   把带#的都过滤掉

global {

        usage-countno;

}

common {

        protocol C;

        handlers {

               pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f";

               pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f";

               local-io-error "/usr/lib/drbd/notify-io-error.sh;/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;halt -f";

                fence-peer "/usr/lib/drbd/crm-fence-peer.sh";

                split-brain "/usr/lib/drbd/notify-split-brain.sh root";

                out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";

        }

        startup {

               wfc-timeout 120;

               degr-wfc-timeout 120;  

        }

        options {

        }

        disk {

                on-io-error   detach;

            }

        net {

                cram-hmac-alg"sha1";

               shared-secret "allendrbd";

        }

        syncer {

          rate100M;

}

}

 [root@mysql-a etc]# cat/etc/drbd.d/drbd_mysql.res         

resource r0 {

 

        on mysql-a{

               device    /dev/drbd0;

                disk      /dev/sdb1;

               meta-disk internal;

               address   40.0.0.1:7789;

                }

        on mysql-b{

               device    /dev/drbd0;

               disk      /dev/sdb1;

               meta-disk internal;

               address   40.0.0.2:7789;

               }

}

在配置drbd的时候心跳是一个网段，数据同步是一个网段，业务是一个网段

Drbd 开始初始化元数据

[root@mysql-a ~]#drbdadm create-md r0                  

  --==  Thank you for participating in the globalusage survey  ==--

The server's response is:

 

you are the 3978th user to install this version

Writing meta data...

initializing activity log

NOT initializing bitmap

New drbd meta data block successfully created.

success

 [root@mysql-b drbd.d]# drbdadmcreate-md r0              

  --==  Thank you for participating in the globalusage survey  ==--

The server's response is:

 

you are the 3979th user to install this version

Writing meta data...

initializing activity log

NOT initializing bitmap

New drbd meta data block successfully created.

success

启动drbd 两个节点都启动 ，注意启动a节点的时候再去启动b节点，此时ab两个节点会进行通信

 [root@mysql-a ~]# /etc/init.d/drbd start  也可以使用drbdadm 去启动  参数 [root@mysql-b /]#drbdadm �Chelp 去查看参数

Starting DRBD resources: [

     create res:r0/usr/local/run/drbd: No such file or directory

 

   prepare disk:r0/usr/local/run/drbd: No such file or directory

 

    adjust disk: r0

     adjust net: r0

]

outdated-wfc-timeout has to be shorter thandegr-wfc-timeout

outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s)

..........

***************************************************************

 DRBD's startupscript waits for the peer node(s) to appear.

 - In case thisnode was already a degraded cluster before the

   reboot thetimeout is 120 seconds. [degr-wfc-timeout]

 - If the peer wasavailable before the reboot the timeout will

   expire after 120seconds. [wfc-timeout]

   (These valuesare for resource 'r0'; 0 sec -> wait forever)

 To abort waitingenter 'yes' [  23]:yes

[root@mysql-a ~]#

 [root@mysql-b drbd.d]#/etc/init.d/drbd start

Starting DRBD resources: [

     create res:r0/usr/local/run/drbd: No such file or directory

 

   prepare disk:r0/usr/local/run/drbd: No such file or directory

 

    adjust disk: r0

     adjust net: r0

]

outdated-wfc-timeout has to be shorter thandegr-wfc-timeout

outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s)

..........

***************************************************************

 DRBD's startupscript waits for the peer node(s) to appear.

 - In case thisnode was already a degraded cluster before the

   reboot the timeout is 120 seconds.[degr-wfc-timeout]

 - If the peer wasavailable before the reboot the timeout will

   expire after 120seconds. [wfc-timeout]

   (These valuesare for resource 'r0'; 0 sec -> wait forever)

 To abort waitingenter 'yes' [  11]:

[root@mysql-b drbd.d]#

在任意节点查看节点状态

[root@mysql-a ~]#cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42

 

 1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116

[root@mysql-a ~]#

[[email protected]]# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build byroot@mysql-b, 2015-03-13 19:58:56

 

 1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116

初始化设备同步，手动设置主备节点

默认是没有主次之分的，需要手动设置

[root@mysql-a ~]# drbdadm primary all  设置主节点

[root@mysql-b drbd.d]# cat /proc/drbd                    

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-13 19:58:56

 0: cs:SyncTargetro:Secondary/Secondary ds:Inconsistent/UpToDate C r-----

    ns:0 nr:439296dw:439296 dr:0 al:0 bm:26 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:20524820

       [>....................] sync'ed: 2.2% (20040/20472)M

        finish: 0:08:33speed: 39,936 (39,936) want: 78,120 K/sec

[root@mysql-a drbd.d]# cat /proc/drbd                    

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42

 0: cs:SyncSourcero:Secondary/Secondary ds:UpToDate/Inconsistent C r-----

    ns:831488 nr:0dw:0 dr:831488 al:0 bm:50 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20132628

       [>....................] sync'ed: 4.0% (19660/20472)M

        finish:0:08:28 speed: 39,536 (39,592) K/sec

参数说明我的资源为r0

此时两个节点上都显示的是从，需要手动去配置主 从节点

在主节点上手动设置drbd 主从关系

[root@mysql-a etc]# drbdadm �Chelp  参数

           

如果有以下提示需要执行：

[root@mysql-a ~]# drbdadm primary all   设置主从有提示依次执行以下命令

0: State change failed: (-2) Need access to UpToDate data

Command 'drbdsetup primary 0' terminated with exit code17

[root@mysql-a ~]# drbdadm -- --overwrite-data-of-peerprimary all

[root@mysql-a ~]# drbdadm primary all 

[root@mysql-a ~]# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

 0: cs:SyncSourcero:Primary/Secondary ds:UpToDate/Inconsistent C r-----

    ns:4277528 nr:0dw:0 dr:4279960 al:0 bm:261 lo:1 pe:1 ua:3 ap:0 ep:1 wo:f oos:16686868

       [===>................] sync'ed: 20.5% (16292/20472)M   元数据正在同步

        finish:0:06:29 speed: 42,852 (32,400) K/se

[root@mysql-b ~]# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

 0: cs:SyncTargetro:Secondary/Primary ds:Inconsistent/UpToDate C r-----

    ns:0 nr:5178368dw:5178368 dr:0 al:0 bm:316 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:15785748

       [===>................] sync'ed: 24.8% (15412/20472)M

        finish:0:06:06 speed: 43,064 (33,624) want: 41,000 K/sec

[root@mysql-a /]# mkfs.ext4 /dev/drbd0   只需要在主节点进行格式化分区

挂载drbd0 到mysql的数据存储位置

[root@mysql-a data]# /etc/init.d/mysql status 

 SUCCESS! MySQLrunning (1308)

[root@mysql-a data]# /etc/init.d/mysql stop

Shutting down MySQL... SUCCESS!

[root@mysql-a data]# ls

auto.cnf ib_logfile0  mysql        mysql-a.pid         test

ibdata1  ib_logfile1  mysql-a.err  performance_schema

[root@mysql-a data]# /etc/init.d/mysql stop

Shutting down MySQL.. SUCCESS!

[root@mysql-a data]# mv * /mnt/

[root@mysql-a /]# mount /dev/drbd0 /data/

[root@mysql-a /]# cd /mnt/

[root@mysql-a mnt]# ls

auto.cnf ibdata1  ib_logfile0  ib_logfile1 mysql  mysql-a.err  performance_schema  test

[root@mysql-a mnt]# mv * /data/

B节点也需要去清空以下mysql的元数据

[root@mysql-a mnt]# /etc/init.d/mysql start

Starting MySQL...... ERROR! The server quit withoutupdating PID file (/data/mysql-a.pid)

[root@mysql-a /]# chown -R mysql. /data/

[root@mysql-a mnt]# /etc/init.d/mysql restart

Shutting down MySQL.. SUCCESS!

Starting MySQL..... SUCCESS!

[root@mysql-a /]# mysql -uroot mysql可以正常登陆，

然后在主节点卸载umount /data

然后在从节点上进行挂载，看数据是否ok。

Drbd 测试

[root@mysql-a ~]# drbdadm secondary r0    将主节点设置为从

[root@mysql-a ~]# cat /proc/drbd           

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

 0: cs:Connectedro:Secondary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:28672dw:28672 dr:664 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@mysql-b ~]# drbdadm primary r0

[root@mysql-b ~]# cat /proc/drbd   

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

 0: cs:Connectedro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:28672 nr:0dw:0 dr:29336 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

手动切换造成drbd 脑裂 处理方法：

处理方法：

第一步在从节点上：

[root@mysql-a /]# drbdadm disconnect r0

r0: Failure: (162) Invalid configuration request

additional info from kernel:

unknown connection

Command 'drbdsetup disconnect ipv4:30.0.0.1:7789ipv4:30.0.0.2:7789' terminated with exit code 10

[root@mysql-a /]# drbdadm secondary r0

[root@mysql-a /]# drbdadm connect  --discard-my-data r0

[root@mysql-a /]# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

 0: cs:WFConnectionro:Primary/Unknown ds:UpToDate/Outdated C r-----

    ns:0 nr:0 dw:0dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:668436

查看还是报错

在主节点上面的处理

[root@mysql-b log]# drbdadm connect r0

[root@mysql-b log]# drbdadm role r0

Secondary/Primary

再去1节点查看状态

[root@mysql-a /]# drbdadm role r0

Primary/Secondary

故障解决

[root@mysql-a /]# drbdadm dstate r0   查看硬盘状态正常

UpToDate/UpToDate

Yum安装heartbeat

http://www.linux-ha.org/wiki/Downloads  linux-ha官网的提供的yum源

http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/

但是官网没有heartbeat 

wget http://mirrors.sohu.com/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm

rpm -ivh epel-release-6-8.noarch.rpm 安装yum源

[root@mysql-a local]# yum install heartbeat* 

[root@mysql-a local]# yum -y install pacemaker*  集群管理器

配置heartbeat

[root@mysql-a /]# cd /usr/share/doc/heartbeat-3.0.4/

[root@mysql-a heartbeat-3.0.4]# ls

apphbd.cf  authkeys AUTHORS  ChangeLog  COPYING COPYING.LGPL  ha.cf haresources  README

[root@mysql-a heartbeat-3.0.4]# cp authkeys ha.cfharesources  /etc/ha.d/   拷贝模板文件到/etc/ha.d/ 目录中

[root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/ha.cf

debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility    local0

keepalive 2

deadtime 10

warntime 5

initdead 30

udpport 694

baud    19200

ucast eth1 30.0.0.2

auto_failback off

node    mysql-a

node    mysql-b

ping 30.0.0.10

respawn hacluster /usr/lib64/heartbeat/ipfail

[root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/haresources

mysql-a IPaddr::20.0.0.252/8/eth0:1 drbddisk::r0  Filesystem::/dev/drbd0::/data::ext4 mysql

[root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/authkeys

auth 2

2 sha1 oldboy!

[root@mysql-b ~]# 

启动heartbeat

在实际操作中遇到drbd脑裂问题处理

[root@mysql-a ~]# netstat �Cltuanp查看心跳ip是否正常

[root@mysql-b ~]# cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)

GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

 0: cs:StandAlonero:Primary/Unknown ds:UpToDate/DUnknown  r-----

    ns:0 nr:0dw:256 dr:6889 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:364

解决思路：

 在从节点上

drbdadm secondary r0

drbdadm -- --discard-my-data connect r0

在到主节点上

drbdadm connect r0

解决脑裂

集群测试！

断掉心跳线， 节点关机，vip是否正常切换