linux下安装drbd详解

关于drbd理论方面的知识本文中不做介绍，本文章主要讲述如何搭建drbd。

 

一、系统环境

 

系统平台: Centos6.3(内核2.6.32-431.29.2.el6.x86_64)

节点一主机名: Cenos-HA1     IP:192.168.1.202

节点一主机名: Cenos-HA2     IP:192.168.1.202

 DRBD版本：DRBD-8.4.3

 

二、安装

 

1.准备

(1).关闭iptables和SELINUX，避免安装过程中报错。

节点1和节点2

#/etc/init.d/iptables stop

#/etc/init.d/ip6tables stop

#chkconfig iptables off

#chkconfig ip6tables off

# vi /etc/sysconfig/selinux

---------------

SELINUX=disabled

---------------

 

(2).设置 hosts文件(节点1和节点2)

 

# vi /etc/hosts

-----------------

192.168.1.202 Centos-HA1
192.168.1.203 Centos-HA2

 

-----------------

 

节点1和节点2(重启）

(3).在两台虚拟机分别添加一块2G硬盘sdb作为DRBD，分别分区为sdb1，大小2G，并在本地系统创建/data目录，不做挂载操作。(节点1和节点2)

节点1

[root@Centos-HA1 ~]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd59786d2.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 

 

[root@Centos-HA1 ~]#mkdir /data

 

节点2

[root@Centos-HA2 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x9adc5d39.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 

[root@Centos-HA2 ~]#mkdir /data

 

(4).时间同步

# ntpdate -u asia.pool.ntp.org

2、DRBD的安装配置

 

(1).安装依赖包：( 节点1和节点2)

 

#yum install gcc gcc-c++ make glibc flex kernel kernel-devel kernel-headers

 

(2)安装DRBD: (节点1和节点2)

# wget http://oss.linbit.com/drbd/8.4/drbd-8.4.3.tar.gz

# tar -zxvf drbd-8.4.3.tar.gz

# cd drbd-8.4.3

# ./configure --prefix=/usr/local/drbd --with-km

# make KDIR=/usr/src/kernels/2.6.32-431.29.2.el6.x86_64/

# make install

# mkdir -p /usr/local/drbd/var/run/drbd

# cp /usr/local/drbd/etc/rc.d/init.d/drbd /etc/init.d/

# chkconfig --add drbd 

# chkconfig --level 35 drbd on

 

(3)加载DRBD模块(节点1和节点2)

#modprobe drbd

 

如果出现如下错误

[root@Centos-HA1 drbd-8.4.3]# modprobe drbd
FATAL: Module drbd not found.

解决办法：

http://88fly.blog.163.com/blog/static/1226803902013111345347479

(注意：不要使用yum install kernel* 因为我做实验然后重启系统进不去了

可以使用yum install kernel kernel-devel kernel-headers)

 

查看DRBD模块是否加载到内核： (节点1和节点2)

# lsmod |grep drbd

 

节点1

 

节点2

 

 

3.参数配置：(节点1和节点2)

# vi /usr/local/drbd/etc/drbd.conf

清空文件内容，并添加如下配置：

---------------

resource r0{

protocol C;

 

startup { wfc-timeout 0; degr-wfc-timeout 120;}

disk { on-io-error detach;}

net{

  timeout 60;

  connect-int 10;

  ping-int 10;

  max-buffers 2048;

  max-epoch-size 2048;

}

syncer { rate 30M;}

 

on Centos-HA1{

  device /dev/drbd0;

  disk   /dev/sdb1;

  address 192.168.1.202:7788;

  meta-disk internal;

}

on Centos-HA2{

  device /dev/drbd0;

  disk   /dev/sdb1;

  address 192.168.1.203:7788;

  meta-disk internal;

}

}

---------------

 

(注:至于参数详解请参考  http://www.cnblogs.com/Elliot-wang/p/3994362.html )

(4).创建DRBD设备并激活ro资源：(节点1和节点2)

# mknod /dev/drbd0 b 147 0

# drbdadm create-md r0

 

 

等待片刻，显示success表示drbd块创建成功

----------------

Writing meta data...

initializing activity log

NOT initializing bitmap

New drbd meta data block successfully created.

 

               --== Creating metadata ==--

As with nodes, we count the total number of devices mirrored by DRBD

at http://usage.drbd.org.

 

The counter works anonymously. It creates a random number to identify

the device and sends that random number, along with the kernel and

DRBD version, to usage.drbd.org.

 

http://usage.drbd.org/cgi-bin/insert_usage.pl?

 

nu=716310175600466686&ru=15741444353112217792&rs=1085704704

 

* If you wish to opt out entirely, simply enter 'no'.

* To continue, just press [RETURN]

 

success

----------------

 

再次输入该命令：

# drbdadm create-md r0

成功激活r0

----------------

[need to type 'yes' to confirm] yes

 

Writing meta data...

initializing activity log

NOT initializing bitmap

New drbd meta data block successfully created.

----------------

 

 

5.启动DRBD服务：(节点1,节点2)

# service drbd start

 

6.查看状态：(node1,node2)

 

#service drbd status

# cat /proc/drbd

 

 

 

 

(7).将Centos-HA1主机配置为主节点：(节点1)

[root@Centos-HA1 ~]# drbdsetup /dev/drbd0 primary --force

 

查看主机状态

节点1

[root@Centos-HA1 ~]#service drbd status

 

 

节点2

[root@Centos-HA1 ~]#service drbd status

 

 

 

ro在主从服务器上分别显示 Primary/Secondary和Secondary/Primary

ds显示UpToDate/UpToDate

表示主从配置成功。

 

(8).挂载DRBD：(节点1)

 

从刚才的状态上看到mounted和fstype参数为空，所以我们这步开始挂载DRBD到系统目录

[root@Centos-HA1 ~]# mkfs.ext4 /dev/drbd0

[root@Centos-HA1 ~]# mount /dev/drbd0 /data     

注：Secondary节点上不允许对DRBD设备进行任何操作，包括只读，所有的读写操作只能在Primary节点上进行，只有当Primary节点挂掉时，Secondary节点才能提升为Primary节点继续工作。

(9).模拟DRBD1故障，DRBD2接管并提升为Primary

(节点1)

[root@Centos-HA1 ~]#cd /data

[root@Centos-HA1 ~]# touch a.txt b.txt c.txt d.txt e.txt

[root@Centos-HA1 ~]#cd ..

[root@Centos-HA1 ~]# umount /data

[root@Centos-HA1 ~]# drbdsetup /dev/drbd0 secondary

 

注：这里实际生产环境若DRBD1宕机，在DRBD2状态信息中ro的值会显示为Secondary/Unknown,只需要进行DRBD提权操作即可。

 

(节点2)

 

[root@Centos-HA2 ~]# drbdsetup /dev/drbd0 primary

 

[root@Centos-HA2 ~]#  mount  /dev/drbd0 /data

[root@Centos-HA2 ~]# cd /data

[root@Centos-HA2 ~]#  touch f.txt g.txt h.txt j.txt k.txt

[root@Centos-HA2 ~]# ls

--------------

a.txt  b.txt  c.txt  d.txt  e.txt  f.txt  g.txt  h.txt  j.txt  k.txt  lost+found

--------------

 

 

到此DRBD已经安装完成。

 

不过如何保证DRBD主从结构的智能切换，实现高可用，这里就需要Heartbeat来实现了。

Heartbeat会在DRBD主端挂掉的情况下，自动切换从端为主端并自动挂载/data分区。

 

 

注：（摘自酒哥的构建高可用LINUX服务器第2版）

假设你把Primary的eth0挡掉，然后直接在Secondary上进行主Primary主机的提升，并且mount上，你可能会发现在Primary上测试考入的文件确实同步过来了，之后把Primary的eth0恢复后，看看有没有自动恢复 主从关系。经过查看，发现DRBD检测出了Split-Brain的状况，也就是两个节点都处于standalone状态，故障描述如下：Split-Brain detected,dropping connection! 这就是传说中的“脑裂”。

 

 

下面本人做个测试

 

首先将刚才的主备环境切换回来，此步我不再叙说，安照第9步反过来操作即可

 

下面我来模拟测试

节点1

[root@Centos-HA1 ~]# cd /data
[root@Centos-HA1 ~]# ls

a.txt  b.txt  c.txt  d.txt  e.txt  f.txt  g.txt  h.txt  j.txt  k.txt  lost+found

[root@Centos-HA1 ~]# touch aa.txt bb.txt cc.txt

[root@Centos-HA1 ~]# cd ..

[root@Centos-HA1 ~]# umount data

[root@Centos-HA1 ~]# ifdown eth0

 

 

节点2

[root@Centos-HA2 ~]#drbdsetup /dev/drbd0 primary

[root@Centos-HA2 ~]# cd /data

[root@Centos-HA2 ~]#mount /dev/drbd0 /data

[root@Centos-HA2 ~]#cd /data

[root@Centos-HA2 ~]#ls

 

aa.txt  bb.txt  cc.txt  d.txt  f.txt  h.txt  k.txt
a.txt   b.txt   c.txt   e.txt  g.txt  j.txt  lost+found

 

 

 

 

节点1

[root@Centos-HA1 ~]# ifup eth0

 

 

[root@Centos-HA1 ~]#drbdadm secondary r0

 

[root@Centos-HA1 ~]#service drbd status

 

 

 

节点2

[root@Centos-HA2 ~]# cd /data

 

[root@Centos-HA2 ~]# touch  aaa.txt  bbb.txt ccc.txt

[root@Centos-HA2 ~]#ls

aaa.txt  a.txt    bb.txt  ccc.txt  c.txt  e.txt  g.txt  j.txt  lost+found
aa.txt   bbb.txt  b.txt   cc.txt   d.txt  f.txt  h.txt  k.txt

 

[root@Centos-HA2 ~]# cd ..

[root@Centos-HA2 ~]# umount /data

[root@Centos-HA2 ~]# drbdadm secondary r0

 

 

节点1

[root@Centos-HA1 ~]# drbdsetup /dev/drbd0 primary

 

 

[root@Centos-HA1 ~]#mount /dev/drbd0 /data

[root@Centos-HA1 ~]# cd /data/
[root@Centos-HA1 ~]# ls
aa.txt  bb.txt  cc.txt  d.txt  f.txt  h.txt  k.txt
a.txt   b.txt   c.txt   e.txt  g.txt  j.txt  lost+found

 

(注:文件没有同步过来)

解决办法

 

(先恢复系统到开始前的状态，就是 eth0 down 之后)

节点1

[root@Centos-HA1 ~]# umount /data
[root@Centos-HA1 ~]# drbdadm secondary r0

 

 

 

节点2

[root@Centos-HA2 ~]# drbdsetup /dev/drbd0 primary

 

[root@Centos-HA2 ~]# mount /dev/drbd0 /data

 

[root@Centos-HA2 ~]# cd /data
[root@Centos-HA2 ~]# ls
aaa.txt  a.txt    bb.txt  ccc.txt  c.txt  e.txt  g.txt  j.txt  lost+found
aa.txt   bbb.txt  b.txt   cc.txt   d.txt  f.txt  h.txt  k.txt

 

 

已经恢复到故障开始的状态，下面开始解决

 

节点1

 

 

[root@Centos-HA1 ~]# drbdadm secondary r0

[root@Centos-HA1 ~]# drbdadm disconnect all

[root@Centos-HA1 ~]# drbdadm --discard-my-data connect r0

切换回来

[root@Centos-HA2 ~]# drbdadm secondary r0

[root@Centos-HA1 ~]# drbdsetup /dev/drbd0 primary

[root@Centos-HA1 ~]# mount /dev/drbd0 /data

[root@Centos-HA1 ~]# cd /data
[root@Centos-HA1 ~]# ls
aaa.txt  a.txt    bb.txt  ccc.txt  c.txt  e.txt  g.txt  j.txt  lost+found
aa.txt   bbb.txt  b.txt   cc.txt   d.txt  f.txt  h.txt  k.txt

(注:本次eth0 宕机实验，如果/data 目录不卸载的话直接 ifdown eth0是不能成功的)

 

完成