DRDB实现mariadb高可用
------本文大纲
简介
DRBD资源属性
原理
安装DRDB
DRDB实现高可用
============
一、简介
DRBD是一种块设备，可以被用于高可用(HA)之中。它类似于一个网络RAID-1功能。当你将数据写入本地文件系统时，数据还将会被发送到网络中另一台主机上。以相同的形式记录在一个文件系统中。本地(主节点)与远程主机(备节点)的数据可以保证实时同步。当本地系统出现故障时，远程主机上还会保留有一份相同的数据，可以继续使用。在高可用(HA)中使用DRBD功能，可以代替使用一个共享盘阵。因为数据同时存在于本地主机和远程主机上。切换时，远程主机只要使用它上面的那份备份数据，就可以继续进行服务了。
二、DRBD资源属性
resource name
可以使用除空白字符外的任意ACSII表中的字符；
drbd设备
drbd的设备的访问路径，设备文件/dev/drbd#；
disk
各节点为组成此drbd设备所提供的块设备，通常是一个磁盘分区；
网络属性
节点间为了实现跨主机磁盘镜像而使用的网络配置；
注意：用户空间工具与drdb与内核中使用的模块版本要保持一致，只有在使用drbdadm工具时才会读取配置文件，对多个资源的公共配置，可以提取出来只配置一次，通常保存在Common中，此外还有global配置，这种配置跟资源本身没有关系的；
三、原理
需要知道，DRBD是工作在系统内核空间，而不是用户空间，它直接复制的是二进制数据，这是它速度快的根本原因。
其次，DRBD至少需要两个节点来工作，一主一次。
DRBD的文件同步过程和普通复制过程的不同：
DRBD在数据进入Buffer Cache时，先经过DRBD这一层，复制一份数据经过TCP/IP协议封装，发送到另一个节点上，另一个节点通过TCP/IP协议来接受复制过来的数据，同步到次节点的DRBD设备上。
DRBD底层设备支持
DRBD需要构建在底层设备之上，然后构建出一个块设备出来。对于用户来说，一个DRBD设备，就像是一块物理的磁盘，可以在上面内创建文件系统。DRBD所支持的底层设备有以下这些类：
1、一个磁盘，或者是磁盘的某一个分区；
2、一个soft raid 设备；
3、一个LVM的逻辑卷；
DRBD在远程传输上支持三种模式：
协议A:异步复制协议
只要主节点完成本地写操作就认为写操作完成，并且需要复制的数据包会被存放本地TCP发送缓存中，当发生failover故障，数据可能会丢失。
协议B：内存同步（半同步，semi-synchronous）复制协议
当本地磁盘的写己经完成，并且复制数据包己经到达对应从节点，此时主节点才认为磁盘写入己经完成，failover不会导致数据丢失，但当所有节点同时出现故障时，主节点上多数最新写入的数据可能丢失。
协议C:同步复制
只有在本地和远程磁盘都确定写入己完成后，主节点才会认为写入完成。这样可确保发生单点故障时不会导致任何数据丢失，如果发生数据丢失，那也只会在所有节点同时存在错误存储时才会发生这种情况
角色、模式
角色
在drbd构造的集群中，资源具有角色的概念，分别为primary和secondary。
所有设为primary的资源将不受限制进行读写操作。可以创建文件系统，可以使用裸设备,甚至直接io。所有设为secondary的设备中不能挂载，不能读写模式
drbd也有drbd mode：单主模型（主从），双主模型
在单主模型下drbd可以使用任意的文件系统
单在双主模型下只能使用集群文件系统，常用的开源的集群文件系统有:ocfs2和gfs2
四、安装DRDB
前提：
设置主机名（要与uname -n一致，主机名可解析）
时间同步
无密钥登录
本机环境
| 系统 | 主机名 | ip地址 |
| Centos6.5 x86_64 | essun.node1.com | 192.168.1.109 |
| Centos6.5 x86_64 | essun.node2.com | 192.168.1.108 |
注：
内核模块程序包一定要跟你的系统内核保持一致，uname -r查看你的内核版本，内核模块的版本必须要严格对应，而用户空间的模块就不那么严格要求了，两个节点的时间也需要保持一致性；
第一步、下载与内核版本相应的内核模块与用户空间工具（地址：http://packages.atrpms.net/dist/el6/drbd/）
[root@essun download]# uname -r 2.6.32-431.el6.x86_64 [root@essun download]# ls drbd-8.4.3-33.el6.x86_64.rpm drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm [root@essun download]# yum isntall drbd-kmdl-2.6.32-431.el6-8.4.3-33.el6.x86_64.rpm drbd-8.4.3-33.el6.x86_64.rpm -y
修改配置文件（/etc/drbd.d/global_common.conf）
[root@essun drbd.d]# vim global_common.conf [root@essun drbd.d]# cat global_common.conf global { usage-count no; # 是否参加DRBD使用者统计.默认是yes # minor-count dialog-refresh disable-ip-verification } common { handlers { # These are EXAMPLE handlers only. # They may have severe implications, # like hard resetting the node under certain circumstances. # Be careful when chosing your poison. pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f"; # fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; # split-brain "/usr/lib/drbd/notify-split-brain.sh root"; # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root"; # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k"; # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh; } startup { # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb #wfc-timeout 120; #(等待另一个节点上线的超时时长) # degr-wfc-timeout 240; #(等待超时后做降级处理) #outdated-wfc-timeout #(过期的等待超时) # wait-after-sb #(脑裂之后等待多长时长) } options { # cpu-mask on-no-data-accessible } disk { # size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes on-io-error detach; # 同步错误的做法是分离 # disk-drain md-flushes resync-rate resync-after al-extents # c-plan-ahead c-delay-target c-fill-target c-max-rate # c-min-rate disk-timeout } net { protocol C; # 使用协议C.表示收到远程主机的写入确认后,则认为写入完成. cram-hmac-alg "sha1"; shared-secret "drdb.essun.drdb.org"; # 设置主备机之间通信使用的信息算法. # protocol timeout max-epoch-size max-buffers unplug-watermark # connect-int ping-int sndbuf-size rcvbuf-size ko-count # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri # after-sb-1pri after-sb-2pri always-asbp rr-conflict # ping-timeout data-integrity-alg tcp-cork on-congestion # congestion-fill congestion-extents csums-alg verify-alg # use-rle } syncer { rate 1000M; } # 设置主备节点同步时的网络速率最大值,单位是字节. }提供物理块设备（两个节点的分区号最好相同），不需要格式化
[root@essun drbd.d]# fdisk -cu /dev/s
scd0 sda1 sg0 shm/ snd/ stderr stdout
sda sda2 sg1 snapshot sr0 stdin systty
[root@essun drbd.d]# fdisk -cu /dev/sda
Command (m for help): p
Disk /dev/sda: 128.8 GB, 128849018880 bytes
255 heads, 63 sectors/track, 15665 cylinders, total 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006c47d
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 411647 204800 83 Linux
/dev/sda2 411648 126240767 62914560 8e Linux LVM
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First sector (126240768-251658239, default 126240768):
Using default value 126240768
Last sector, +sectors or +size{K,M,G} (126240768-251658239, default 251658239): +10G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
[root@essun drbd.d]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
定义DRBD磁盘资源
[root@essun drbd.d]# vim dbresouce.res
resource dbresource {
on essun.node1.com { #on说明在哪个节点上，跟uname -n保持一致
device /dev/drbd0; #在磁盘上表现的drbd叫什么名；
disk /dev/sda3; #所使用的物理磁盘设备是哪个；
address 192.168.1.109:7789; #在node1这个节点上监听的套接字,默认监听在7789端口上；
meta-disk internal; #保存drbd源数据信息，表示就放在自己的磁盘区分上，也可以放在外部的磁盘上；
}
on essun.node2.com{
device /dev/drbd0;
disk /dev/sda3;
address 192.168.1.108:7789;
meta-disk internal;
}
}
将这两个文件拷贝到node1中的相同目录下
[root@essun drbd.d]# scp dbresouce.res global_common.conf essun.node1.com:/etc /drbd.d/ dbresouce.res 100% 259 0.3KB/s 00:00 global_common.conf 100% 2044 2.0KB/s 00:00 [root@essun drbd.d]#
在各自的节点上初始化资源
[root@essun drbd.d]# drbdadm create-md dbresource
#初始化资源列表
Writing meta data...
initializing activity log
NOT initializing bitmap
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
New drbd meta data block successfully created.
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory
[root@essun drbd.d]# service drbd start
#启动服务
Starting DRBD resources: [
create res: dbresource
prepare disk: dbresource
adjust disk: dbresource
adjust net: dbresource
]
.
[root@essun drbd.d]# drbd-overview
#查看drbd的状态
0:dbresource/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
[root@essun drbd.d]# watch -n 1 'cat /proc/drbd'
#实时查看drbd的状态
Every 1.0s: cat /proc/drbd Thu May 1 08:48:23 2014
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404
上面表示两个节点的drbd都是从的，这是因为我们并没有对其中任何一个提升主的。
将其中一个节点的DRBD资源提升为主资源
[root@essun drbd.d]# drbdadm primary --force dbresource
#将当前节点的drbd资源提升为主资源
[root@essun drbd.d]# drbd
[root@essun drbd.d]# drbd-overview
0:dbresource/0 SyncSource Primary/Secondary UpToDate/Inconsistent C r---n-
[>...................] sync'ed: 6.1% (9624/10236)M
初始化之后（角色己经发生改变了）
[root@essun drbd.d]# drbd-overview 0:dbresource/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
格式化DRBD
[root@essun drbd.d]# mke2fs -t ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621351 blocks
131067 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@essun drbd.d]# mount /dev/drbd0 /mnt
[root@essun drbd.d]# cd mnt
-bash: cd: mnt: No such file or directory
[root@essun drbd.d]# cd /mnt
[root@essun mnt]# ls
lost+found
[root@essun mnt]# cp /etc/inittab .
[root@essun mnt]# ls
inittab lost+found
[root@essun mnt]#
提升另一个节点的DRBD资源
[root@essun /]# umount /mnt #卸载drbd [root@essun /]# drbdadm secondary dbresource # 降级DRBD资源 [root@essun /]# drbd-overview 0:dbresource/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- #以下是在另一个节点操作 [root@essun ~]# drbdadm primary dbresource [root@essun ~]# mount /dev/drbd0 /mnt [root@essun ~]# cd /mnt [root@essun mnt]# ls inittab lost+found
如果想让DRBD能够自动切换的话，就要借助于messages layer程序了。
首先在对DRBD降级，停止服务，禁止开机自动启动
[root@essun /]# umount /mnt [root@essun /]# drbdadm secondary dbresource [root@essun /]# drbd-overview 0:dbresource/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- [root@essun /]# service drbd stop Stopping all DRBD resources: . [root@essun /]# chkconfig drbd off [root@essun /]# chkconfig --list drbd drbd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
第二步、安装mariadb
[root@essun download]# tar xf mariadb-10.0.10-linux-x86_64.tar.gz [root@essun download]# mv mariadb-10.0.10-linux-x86_64 /usr/local [root@essun download]# cd /usr/local/ [root@essun local]# ln -s mariadb-10.0.10-linux-x86_64 mysql [root@essun local]# groupadd -r mysql [root@essun local]# useradd -r -g mysql mysql [root@essun local]# chown -R mysql.mysql mari/
启动DRBD服务，提升主资源
[root@essun ~]# service drbd start
Starting DRBD resources: [
create res: dbresource
prepare disk: dbresource
adjust disk: dbresource
adjust net: dbresource
]
.
[root@essun local]# drbd-overview
0:dbresource/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@essun local]# drbd-overview
0:dbresource/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
挂载数据目录
[root@essun local]# mkdir /mariadb
[root@essun local]# mount /dev/drbd0 /mariadb/
[root@essun local]# cd /mariadb/
[root@essun mariadb]# mkdir {data,logs}
[root@essun mariadb]# ls
data logs lost+found
[root@essun /]# chown -R mysql.mysql /mariadb/
[root@essun /]# ll /mariadb/
total 24
drwxr-xr-x 2 mysql mysql 4096 May 1 11:09 data
drwxr-xr-x 2 mysql mysql 4096 May 1 11:09 logs
drwx------ 2 mysql mysql 16384 May 1 09:42 lost+found
注：
此目录在另一个节点上也要一样。
修改配置文件与添加启动脚本
[root@essun mari]# cp support-files/my-large.cnf /etc/cnf [root@essun mari]# vim /etc/my.cnf [root@essun mari]# grep -v "^#" /etc/my.cnf |grep -v " [client] port = 3306 socket = /tmp/mysql.sock [mysqld] port = 3306 socket = /tmp/mysql.sock skip-external-locking key_buffer_size = 256M max_allowed_packet = 1M table_open_cache = 256 sort_buffer_size = 1M read_buffer_size = 1M read_rnd_buffer_size = 4M myisam_sort_buffer_size = 64M thread_cache_size = 8 query_cache_size= 16M thread_concurrency = 8 datadir=/mariadb/data log-bin=/mariadb/logs/mysql-bin binlog_format=mixed server-id = 1 [mysqldump] quick max_allowed_packet = 16M [mysql] no-auto-rehash [myisamchk] key_buffer_size = 128M sort_buffer_size = 128M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout [root@essun mari]# cp support-files/mysql.server /etc/rc.d/init.d/mari [root@essun mari]# chmod +x /etc/rc.d/init.d/mari [root@essun mari]# chkconfig --add mari [root@essun mari]# chkconfig mari off
初始化数据库,并登录测试
[root@essun mari]# scripts/mysql_install_db --user=mysql --datadir=/mariadb/data/ Installing MariaDB/MySQL system tables in '/mariadb/data/' ... 140501 11:22:15 [Note] InnoDB: The InnoDB memory heap is disabled 140501 11:22:15 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140501 11:22:15 [Note] InnoDB: Compressed tables use zlib 1.2.3 ........省略中........ Please report any problems at http://mariadb.org/jira The latest information about MariaDB is available at http://mariadb.org/. You can find additional information about the MySQL part at: http://dev.mysql.com Support MariaDB development by buying support/new features from SkySQL Ab. You can contact us about this at [email protected]. Alternatively consider joining our community based development effort: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/ [root@essun local]# service mari start Starting MySQL........ [ OK ] [root@essun local]# /usr/local/mysql/bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 4 Server version: 10.0.10-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (0.00 sec) MariaDB [(none)]>
到另一个节点测试一下，先停止服务，卸载、降级
[root@essun local]# service mari stop Shutting down MySQL.. [ OK ] [root@essun local]# umount /mariadb/ [root@essun local]# drbdadm secondary dbresource [root@essun local]# drbd-overview 0:dbresource/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- [root@essun local]# scp /etc/my.cnf essun.node1.com:/etc/ my.cnf 100% 4936 4.8KB/s 00:00 [root@essun local]# scp -p /etc/rc.d/init.d/mari essun.node1.com:/etc/rc.d/init.d/ mari 100% 11KB 11.4KB/s 00:00 [root@essun local]# ==============以下是另一个节点的操作 [root@essun ~]# mkdir /mariadb [root@essun ~]# drbdadm primary dbresource [root@essun ~]# drbd-overview 0:dbresource/0 Connected Primary/Secondary UpToDate/UpToDate C r----- [root@essun ~]# chown -R mysql.mysql /mariadb/ [root@essun ~]# mount /dev/drbd0 /mariadb/ [root@essun ~]# cd /mariadb/ [root@essun mariadb]# ll total 24 drwxr-xr-x 5 mysq mysql 4096 May 1 11:29 data drwxr-xr-x 2 mysq mysql 4096 May 1 11:29 logs drwx------ 2 mysq mysql 16384 May 1 09:42 lost+found [root@essun /]# service mari start Starting MySQL....... [ OK ] [root@essun /]# /usr/local/mysql/bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 4 Server version: 10.0.10-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (0.03 sec) MariaDB [(none)]>
两个节点都要停止服务
[root@essun /]# service mari stop Shutting down MySQL.. [ OK ] [root@essun /]# chkconfig mari off [root@essun /]# chkconfig --list mari mari 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@essun /]# service drbd stop Stopping all DRBD resources: . [root@essun /]# chkconfig drbd off [root@essun /]# chkconfig --list drbd drbd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
第三步、安装corosync、pacemaker
[root@essun download]# yum install corosync pacemaker -y [root@essun download]# yum install crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm -y
修改配置文件，生成密钥
[root@essun /]# cd /etc/corosync/
[root@essun corosync]# ls
corosync.conf.example service.d
corosync.conf.example.udpu uidgid.d
[root@essun corosync]# cp corosync.conf.example corosync.conf
[root@essun corosync]# vim corosync.conf
[root@essun corosync]# cat corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.1.1
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
service {
name: pacemaker
ver: 0
}
aisexce {
user: root
group: root
}
[root@essun corosync]# mv /dev/random /dev/x
[root@essun corosync]# mv /dev/urandom /dev/random
[root@essun corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@essun corosync]# mv /dev/random /dev/urandom
[root@essun corosync]# mv /dev/x /dev/random
将这两个文件复制到另一个节点上一份
[root@essun corosync]# scp authkey corosync.conf essun.node1.com:/etc/corosync/ authkey 100% 128 0.1KB/s 00:00 corosync.conf 100% 554 0.5KB/s 00:00 [root@essun corosync]#
启动服务（两个节点都要启动服务）
[root@essun corosync]# service corosync start Starting Corosync Cluster Engine (corosync): [ OK ] [root@essun corosync]# crm status Last updated: Thu May 1 10:05:25 2014 Last change: Thu May 1 10:05:21 2014 via crmd on essun.node2.com Stack: classic openais (with plugin) Current DC: essun.node2.com - partition with quorum Version: 1.1.10-14.el6_5.3-368c726 2 Nodes configured, 2 expected votes 0 Resources configured Online: [ essun.node1.com essun.node2.com ] [root@essun corosync]#
禁用stonith裁决，忽略投票规则
crm(live)configure# property stonith-enabled=false
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node essun.node1.com
node essun.node2.com
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6_5.3-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
定义VIP，mariadb服务
crm(live)configure# primitive vip ocf:heartbeat:IPaddr params ip=192.168.1.150 op monitor interval=20s timeout=20s on-fail=restart
crm(live)configure# verify
crm(live)configure# primitive myserver lsb:mari op monitor interval=20s timeout=20s on-fail=restart
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
crm(live)configure# show
node essun.node1.com \
attributes standby="off"
node essun.node2.com
primitive myserver lsb:mari \
op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \
params ip="192.168.1.150" \
op monitor interval="20s" timeout="20s" on-fail="restart"
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6_5.3-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
配置drdb集群资源
crm(live)configure# primitive dbdrbd ocf:linbit:drbd params drbd_resource=dbresource op monitor role=Master interval=50s timeout=30s op monitor role=Slave interval=60s timeout=30s op start timeout=240s interval=0 op stop timeout=100s interval=0
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node essun.node1.com \
attributes standby="off"
node essun.node2.com
node essun.node1.com \
attributes standby="off"
node essun.node2.com
primitive dbdrbd ocf:linbit:drbd \
params drbd_resource="dbresource" \
op monitor role="Master" interval="50s" timeout="30s" \
op monitor role="Slave" interval="60s" timeout="30s" \
op start timeout="240s" interval="0" \
op stop timeout="100s" interval="0"
primitive myserver lsb:mari \
op monitor interval="20s" timeout="20s" on-fail="restart"
primitive vip ocf:heartbeat:IPaddr \
params ip="192.168.1.150" \
op monitor interval="20s" timeout="20s" on-fail="restart"
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6_5.3-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
drbd需要同时运行在两个节点上，但只能有一个节点（primary/secondary模型）是Master，而另一个节点为Slave；因此，它是一种比较特殊的集群资源，其资源类型为多态（Multi-state）clone类型，即主机节点有Master和Slave之分，且要求服务刚启动时两个节点都处于slave状态，一般来说，只要是定义克隆资源，首先其本身必须是主资源
crm(live)configure# ms ms_dbdrdb dbdrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show ms_dbdrdb
ms ms_dbdrdb dbdrbd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
到node1验证一下，看看是不是变成了主资源
[root@essun /]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:bf:c2:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.109/24 brd 255.255.255.255 scope global eth0
inet 192.168.1.150/24 brd 255.255.255.255 scope global secondary eth0
inet6 fe80::20c:29ff:febf:c282/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
定义文件系统资源（以便挂载DRBD的主资源，此处挂载的是mariadb的数据目录)
crm(live)configure# primitive dbstore ocf:heartbeat:Filesystem params device=/dev/drbd0 directory=/mariadb fstype=ext4 op monitor interval40s timeout=40s op start timeout=60s interval=0 op stop timeout=60 interval=0
crm(live)configure# verify
crm(live)configure# commit
WARNING: CIB changed in the meantime: won't touch it!
Do you still want to commit? y
crm(live)configure# show dbstore
primitive dbstore ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/mariadb" fstype="ext4" \
op monitor timeout="40s" interval="0" interval40s \
op start timeout="60s" interval="0" \
op stop timeout="60" interval="0
结合drbd的特性，只能在drbd服务将某节点设置为Primary以后方可启动。因此，还需要为这两个资源建立排列约束和顺序约束。
colocation vip_with_myserver inf: vip myserver
crm(live)configure# colocation myserver_with_dbststore inf: myserver
dbstore
crm(live)configure# colocation dbstore_with_ms_dbdrbd inf: dbstore ms_dbdrdb:Master
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# order ms_dbdrbd_before_dbstore inf: ms_dbdrdb:promote dbstore:start
crm(live)configure# verify
crm(live)configure# order vip_before_myserver inf: vip myserver
crm(live)configure# verify
crm(live)configure# order dbstore_before_myserver inf: dbstore:start myserver:start
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status
Last updated: Thu May 1 15:29:04 2014
Last change: Thu May 1 15:28:55 2014 via crmd on essun.node2.com
Stack: classic openais (with plugin)
Current DC: essun.node2.com - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Online: [ essun.node1.com essun.node2.com ]
vip (ocf::heartbeat:IPaddr): Started essun.node1.com
myserver (lsb:mari): Started essun.node1.com
Master/Slave Set: ms_dbdrdb [dbdrbd]
Masters: [ essun.node1.com ]
Slaves: [ essun.node2.com ]
dbstore (ocf::heartbeat:Filesystem): Started essun.node1.com
到节点node1上看一下maridb下是否己经挂载DRBD的主资源
[root@essun mariadb]# hostname essun.node1.com [root@essun mariadb]# ls data logs lost+found
这己表示在DRBD己经挂载到node1的mariadb目下
第四步、测试高可用mariadb
授权测试用户
[root@essun /]# /usr/local/mysql/bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 4 Server version: 10.0.10-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> grant all on *.* to root@'192.168.1.%' identified by 'mari'; Query OK, 0 rows affected (0.00 sec)
使用vip登录测试
[root@essun mariadb]# hostname essun.node2.com [root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150 Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 6 Server version: 10.0.10-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (0.08 sec)
模拟节点node1故障
crm(live)node# standby essun.node1.com [root@essun ~]# crm status Last updated: Thu May 1 16:28:59 2014 Last change: Thu May 1 16:27:34 2014 via crmd on essun.node2.com Stack: classic openais (with plugin) Current DC: essun.node1.com - partition with quorum Version: 1.1.10-14.el6_5.3-368c726 2 Nodes configured, 2 expected votes 5 Resources configured Node essun.node1.com: standby Online: [ essun.node2.com ] vip (ocf::heartbeat:IPaddr): Started essun.node2.com myserver (lsb:mari): Started essun.node2.com Master/Slave Set: ms_dbdrdb [dbdrbd] Masters: [ essun.node2.com ] Stopped: [ essun.node1.com ] dbstore (ocf::heartbeat:Filesystem): Started essun.node2.com再次使用vip登录测试
[root@essun mariadb]# hostname essun.node1.com [root@essun mariadb]# /usr/local/mysql/bin/mysql -uroot -p -h192.168.1.150 Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 10.0.10-MariaDB-log MariaDB Server Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | test | +--------------------+ 4 rows in set (1.28 sec)
===================================完====================================================