keepalived+ceph rbd配置nfs的高可用
原理比较简单，利用ceph的rbd导出nfs，然后配置keepalived，在不同主机映射的rbd块间进行vip漂移。
1. 创建并映射rbd块设备
试验机器为ceph中的osd1和osd2主机，ip分别为192.168.32.3、192.168.32.4，vip为192.168.32.5。
先创建一个rbd块设备，然后在两台机器上导出此相同的块：
[root@osd1 keepalived]# rbd ls test test.img [root@osd1 keepalived]# rbd showmapped id pool image snap device 0 test test.img - /dev/rbd0 [root@osd2 keepalived]# rbd showmapped id pool image snap device 0 test test.img - /dev/rbd0
然后将/dev/rbd0格式化后进行挂载，如/mnt目录。
2. 配置keepalived
从http://www.keepalived.org/download.html下载最新的keepalived 1.2.15版本，安装比较简单，直接按照INSTALL的说明进行默认安装即可。为了方便管理keepalived服务，需进行以下操作：
cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/ cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ mkdir /etc/keepalived cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/ cp /usr/local/sbin/keepalived /usr/sbin/ chkconfig --add keepalived chkconfig keepalived on chkconfig --list keepalived配置/etc/keepalived.conf。osd1的配置如下：
[root@osd1 keepalived]# cat keepalived.conf
global_defs
{
notification_email
{
}
router_id osd1
}
vrrp_instance VI_1 {
state MASTER
interface em1
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.32.5/24
}
} osd2的配置如下：
[root@osd2 keepalived]# cat keepalived.conf
global_defs
{
notification_email
{
# [email protected]
}
#notification_email_from [email protected]
#smtp_server 127.0.0.1
#stmp_connect_timeout 30
router_id osd2
}
vrrp_instance VI_1 {
state BACKUP
interface em1
virtual_router_id 100
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
notify_master "/etc/keepalived/ChangeToMaster.sh"
notify_backup "/etc/keepalived/ChangeToBackup.sh"
virtual_ipaddress {
192.168.32.5/24
}
} 在osd2写了两个控制脚本，用于osd2的keepalived状态改变时执行。ChangeToMaster.sh：
[root@osd2 keepalived]# cat ChangeToMaster.sh #!/bin/bash service nfs start ssh lm "umount -f /mnt" ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"ChangeToBackup.sh:
[root@osd2 keepalived]# cat ChangeToBackup.sh #!/bin/bash ssh lm "umount -f /mnt" ssh osd1 "service nfs stop" ssh osd1 "umount /mnt" ssh osd1 "rbd unmap /dev/rbd0" ssh osd1 "rbd map test/test.img" ssh osd1 "mount /dev/rbd0 /mnt" ssh osd1 "service nfs start" ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt" service nfs stop umount /mnt rbd unmap /dev/rbd0 rbd map test/test.img mount /dev/rbd0
3. 配置nfs
在ceph的一个节点利用rbd map一个块设备，然后格式化并挂载在一个目录，如/mnt。在此节点上安装nfs的rpm包：
yum -y install nfs-utils
设置挂载目录：
cat /etc/exports /mnt 192.168.101.157(rw,async,no_subtree_check,no_root_squash) /mnt 192.168.108.4(rw,async,no_subtree_check,no_root_squash)
启动并导出：
service nfs start chkconfig nfs on exportfs -r
客户端查看一下：
showmount -e mon0 Export list for mon0: /mnt 192.168.108.4,192.168.101.157然后挂载：
mount -t nfs mon0:/mnt /mnt
需要注意的是，NFS默认是用UDP协议，如果网络不稳定，换成TCP协议即可：
mount -t nfs mon0:/mnt /mnt -o proto=tcp -o nolock
4. 测试
关闭osd1的网卡em1查看结果：
[root@osd1 keepalived]# ifdown em1
[root@osd1 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff 查看osd2的网卡：
[root@osd2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff
inet 192.168.32.4/24 brd 192.168.32.255 scope global em1
valid_lft forever preferred_lft forever
inet 192.168.32.5/24 scope global secondary em1
valid_lft forever preferred_lft forever
inet6 fe80::ca1f:66ff:fef7:615d/64 scope link
valid_lft forever preferred_lft forever vip已经漂移到osd2，查看客户端挂载情况：
[root@lm /]# df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 454G 79G 353G 19% / tmpfs tmpfs 1.7G 4.6M 1.7G 1% /dev/shm 192.168.32.5:/mnt nfs 100G 21G 80G 21% /mnt打开osd1的网卡em1：
[root@osd1 keepalived]# ifup em1
Determining if ip address 192.168.32.3 is already in use for device em1...
[root@osd1 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.3/24 brd 192.168.32.255 scope global em1
valid_lft forever preferred_lft forever
inet 192.168.32.5/24 scope global secondary em1
valid_lft forever preferred_lft forever
inet6 fe80::ca1f:66ff:fede:5e65/64 scope link
valid_lft forever preferred_lft forever osd2的em1：
[root@osd2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff
inet 192.168.32.4/24 brd 192.168.32.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::ca1f:66ff:fef7:615d/64 scope link
valid_lft forever preferred_lft forever 现在的客户端：
[root@lm /]# df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 454G 79G 353G 19% / tmpfs tmpfs 1.7G 4.6M 1.7G 1% /dev/shm 192.168.32.5:/mnt nfs 100G 21G 80G 21% /mnt [root@lm /]# ls /mnt 31.txt a.txt b.txt c.txt etc linux-3.17.4 linux-3.17.4.tar m2.txt test.img test.img2
也可以为ceph rbd的iscsi配置vip，后面会进行记录。