DRBD学习笔记--实现单主模式可靠性集群方案
一、部署环境：
服务器版本：CentOS6.5
单主模式：
VIP：192.168.3.30
主机网络参数：
| 接口 | MySQL01(Primary) | MySQL02(Secondary) | 说明 |
| eth1 | 192.168.3.23 | 192.168.3.24 | 内网管理IP |
| eth2 | 192.168.5.23 | 192.168.5.24 | 心跳线 |
| eth3 | 192.168.2.23 | 192.168.2.24 | 外网（临时下载文件用） |
网络拓扑：
二、需求分析：
MySQL01及MySQL02上分别添加新硬盘（/dev/sdb），并且使用DRBD进行数据分区（/dev/sdb1）的主备同步。
三、相关配置：
注：
此实验建立在上篇--heartbeat环境下，路由及防火墙的添加不再重述。
主从配置相同，以下仅提供主盘配置情况，从盘配置略。
对新添加的物理磁盘进行分区：
[root@MySQL01 tools]# fdisk -l # 查看物理硬盘
Disk /dev/sda: 8589 MB, 8589934592 bytes # 第一块硬盘，系统盘
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000ff67
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 91 524288 82 Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 91 1045 7658496 83 Linux
Disk /dev/sdb: 2147 MB, 2147483648 bytes # 新加的硬盘，用作DRBD数据盘，分配了2G的磁盘大小
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@MySQL01 tools]# fdisk /dev/sdb # 使用fdisk对新加的硬盘sdb进行分区
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xff42f02f.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n # 新建分区
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1 # 1号分区sdb1--用作DRBD数据盘
First cylinder (1-261, default 1): # 起始扇区使用默认值--1
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-261, default 261): +384M # 该分区使用磁盘空间大小--384M
Command (m for help): n # 新建分区
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2 # 2号分区sdb2--用作DRBD Meta盘
First cylinder (51-261, default 51): # 起始扇区使用默认值--51
Using default value 51
Last cylinder, +cylinders or +size{K,M,G} (51-261, default 261): +size ^H^H^[[D
Unsupported suffix: 'si'.
Supported: 10^N: KB (KiloByte), MB (MegaByte), GB (GigaByte)
2^N: K (KibiByte), M (MebiByte), G (GibiByte)
Last cylinder, +cylinders or +size{K,M,G} (51-261, default 261): +150M # 该分区使用磁盘空间大小--150M
Command (m for help): p # 显示分区规划信息
Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xff42f02f
Device Boot Start End Blocks Id System
/dev/sdb1 1 50 401593+ 83 Linux
/dev/sdb2 51 70 160650 83 Linux
Command (m for help): w # 保存分区信息
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@MySQL01 tools]# partprobe # 创建了分区后在不重启机器的情况下让系统识别改分区表
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.
[root@MySQL01 tools]# mkfs.ext4 /dev/sdb1 # 在新分区上创建文件系统
mke2fs 1.41.12 (17-May-2010)
warning: 183 blocks unused.
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
100744 inodes, 401409 blocks
20079 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
49 block groups
8192 blocks per group, 8192 fragments per group
2056 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@MySQL01 tools]# tune2fs -c -1 /dev/sdb1 # tune2fs为调整和查看ext4文件系统的文件系统参数；-c为设置强制自建的挂载次数，-1表示用不进行强制自检
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
安装DRBD应用程序：
[root@MySQL01 tools]# wget http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz [root@MySQL01 tools]# tar zxf drbd-8.4.4.tar.gz [root@MySQL01 tools]# cd drbd-8.4.4 [root@MySQL01 drbd-8.4.4]# export LC_ALL=C [root@MySQL01 drbd-8.4.4]# ./configure --prefix=/application/drbd8.4.4 --with-km --with-heartbeat --sysconfdir=/etc/ # km为内核模块，sysconfdir为配置文件路径 [root@MySQL01 drbd-8.4.4]# echo $? [root@MySQL01 drbd-8.4.4]# ls -ld /usr/src/kernels/$(uname -r)/ [root@MySQL01 drbd-8.4.4]# make KDIR=/usr/src/kernels/$(uname -r)/ [root@MySQL01 drbd-8.4.4]# make install [root@MySQL01 drbd-8.4.4]# modprobe drbd [root@MySQL01 drbd-8.4.4]# lsmod |grep drbd drbd 340615 0 libcrc32c 1246 1 drbd [root@MySQL01 drbd-8.4.4]# echo "modprobe drbd" >>/etc/rc.local
修改DRBD配置文件（/etc/drbd.conf）：
global {
usage-count no;
}
common {
syncer {
rate 100M;
verify-alg crc32c;
}
}
# primary for drbd1
resource data {
protocol C; # 协议C--同步复制协议
disk {
on-io-error detach;
}
on MySQL01.stephenzhou.net {
device /dev/drbd0; # 逻辑盘
disk /dev/sdb1; # 物理数据盘
address 192.168.5.23:7788; # 心跳IP地址及传输端口
meta-disk /dev/sdb2[0]; # 物理meta盘
}
on MySQL02.stephenzhou.net {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.5.24:7788;
meta-disk /dev/sdb2[0];
}
}
修改/etc/hosts列表：
[root@MySQL01 etc]# vi /etc/hosts 127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost MySQL01 ::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost MySQL01 192.168.5.23 MySQL01.stpehenzhou.net 192.168.5.24 MySQL02.stpehenzhou.net 192.168.3.23 MySQL01.stpehenzhou.net 192.168.3.24 MySQL02.stpehenzhou.net
启动drbd服务：
[root@MySQL01 etc]# drbdadm create-md data # 启动drbd服务，读取data节点 Writing meta data... initializing activity log NOT initializing bitmap New drbd meta data block successfully created. [root@MySQL01 etc]# drbdadm up data # 开启同步模式 /application/drbd8.4.4/var/run/drbd: No such file or directory /application/drbd8.4.4/var/run/drbd: No such file or directory [root@MySQL01 etc]# mkdir /application/drbd8.4.4/var/run/drbd -p # CentOS6.x系统必须手工创建此目录 [root@MySQL01 etc]# cat /proc/drbd # 查看drbd运行状况，默认均以Secondary模式开启 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:401596 [root@MySQL01 etc]# /etc/init.d/iptables stop [root@MySQL01 etc]# cat /proc/drbd # 关闭防火墙并且MySQL02同样开启后，DRBD处于备/备模式 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:401596 [root@MySQL01 etc]# drbdadm -- --overwrite-data-of-peer primary data # 设置MySQL01为主动模式 [root@MySQL01 etc]# cat /proc/drbd # 开始进行同步 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:49152 nr:0 dw:0 dr:49811 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:352444 [=>..................] sync'ed: 13.2% (352444/401596)K finish: 0:00:07 speed: 49,152 (49,152) K/sec [root@MySQL01 etc]# cat /proc/drbd # 完成同步，显示为主/备模式。 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:401593 nr:0 dw:0 dr:402252 al:0 bm:25 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 [root@MySQL01 etc]# mkdir /data # 建立/data文件夹 [root@MySQL01 etc]# mount /dev/drbd0 /data # 将DRBD逻辑盘挂载到/data上 [root@MySQL01 etc]# df -h # 显示磁盘使用情况 Filesystem Size Used Avail Use% Mounted on /dev/sda3 7.2G 1.9G 5.0G 28% / tmpfs 242M 0 242M 0% /dev/shm /dev/sda1 194M 57M 128M 31% /boot /dev/drbd0 380M 11M 350M 3% /data [root@MySQL01 etc]# cd /data [root@MySQL01 data]# ll total 12 drwx------. 2 root root 12288 Jul 8 20:28 lost+found [root@MySQL01 data]# touch `seq 10` # 创建0~9文件 [root@MySQL01 data]# ls 1 10 2 3 4 5 6 7 8 9 lost+found [root@MySQL01 data]# cat /proc/drbd # 同步完成 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:401608 nr:0 dw:15 dr:402592 al:2 bm:25 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
从盘查看同步情况（默认不能读写，必须关闭DRBD并卸载、挂载后才能看到磁盘文件）：
[root@MySQL02 etc]# mount /dev/drbd0 /mnt # 默认是无法挂载DRBD数据盘的 mount: you must specify the filesystem type [root@MySQL02 etc]# mount /dev/sdb1 /mnt # 同样物理盘也无法挂载，因为DRBD在使用它 mount: /dev/sdb1 already mounted or /mnt busy [root@MySQL02 etc]# drbdadm down data # 关闭同步服务 [root@MySQL02 etc]# mount /dev/sdb1 /mnt/ # 挂载物理盘 [root@MySQL02 etc]# df -h # 查看磁盘使用情况，可以看到此处sdb1与MySQL01使用情况完全一致 Filesystem Size Used Avail Use% Mounted on /dev/sda3 7.2G 1.9G 5.0G 28% / tmpfs 242M 0 242M 0% /dev/shm /dev/sda1 194M 57M 128M 31% /boot /dev/sdb1 380M 11M 350M 3% /mnt [root@MySQL02 etc]# drbdadm up data # 无法直接启动同步服务，因为物理盘被挂载到了/mnt上了 0: Failure: (104) Can not open backing device. Command 'drbdsetup attach 0 /dev/sdb1 /dev/sdb2 0 --on-io-error=detach --resync-rate=100M' terminated with exit code 10 [root@MySQL02 etc]# umount /mnt/ # 卸载物理盘 [root@MySQL02 etc]# drbdadm up data # 开启DRBD同步模式 [root@MySQL02 etc]# cat /proc/drbd # 查看同步情况，恢复到主/备模式 version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by [email protected], 2014-07-08 20:52:23 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
扩容主/备物理数据盘：
[root@MySQL02 ~]# drbdadm down data [root@MySQL02 ~]# mount /dev/sdb1 /mnt/ [root@MySQL02 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 7.2G 1.9G 5.0G 28% / tmpfs 242M 0 242M 0% /dev/shm /dev/sda1 194M 57M 128M 31% /boot /dev/sdb1 380M 11M 350M 3% /mnt # 可以看到备节点磁盘利用率只有380M [root@MySQL02 ~]# parted /dev/sdb p Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 2147MB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 32.3kB 814MB 814MB primary ext4 # 然而备节点真实大小为814M 2 814MB 979MB 165MB primary [root@MySQL02 ~]# umount /mnt/ [root@MySQL02 ~]# e2fsck -f /dev/sdb1 # 查看分区大小 e2fsck 1.41.12 (17-May-2010) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 38/100744 files (0.0% non-contiguous), 22971/401409 blocks [root@MySQL02 ~]# resize2fs /dev/sdb1 # 重新分配分区（扩容） resize2fs 1.41.12 (17-May-2010) Resizing the filesystem on /dev/sdb1 to 795184 (1k) blocks. The filesystem on /dev/sdb1 is now 795184 blocks long. [root@MySQL02 ~]# mount /dev/sdb1 /mnt/ [root@MySQL02 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 7.2G 1.9G 5.0G 28% / tmpfs 242M 0 242M 0% /dev/shm /dev/sda1 194M 57M 128M 31% /boot /dev/sdb1 752M 11M 703M 2% /mnt # 备节点磁盘空间已增加至最大 [root@MySQL02 ~]# ls /mnt/ 1 10 11 12 13 14 15 16 17 18 19 2 20 3 4 5 6 7 8 9 a b c d e f g lost+found # 原文件健在
四、参考博文：
partprobe 小命令大作用
linux tune2fs命令详解