初探OCFS2-配置服务
2016/11/28
目的：研究如何使用 iscsi 存储。
一、认识
据zstack的童鞋们介绍：OCFS2是主流对接 SAN 云存储的方式
存储的选择，通常有以下方式：
1、NAS
1）NFS
简单；自建有可靠性问题，商业的 NFS 产品更靠谱；
2）分布式文件系统，以 GlusterFS 为例；
易懂；建议使用 glusterfs 的接口来访问，但也支持通过 nfs 来访问；
2、IP SAN（iscsi）
1）使用iscsiadm 发现一下 target 即可
2）研究 ovirt 是如何使用 iscsi 存储的
看到一个讨论ovirt通过lvm来使用iscsi的方式
http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi
按照字面意思，我的理解（未实验）是：
a、每个 host 挂载 iscsi 存储
b、通过 lvm 划分出 vm disk
c、在运行了 vm 的 host 上 通过 lvchange 来激活指定的 vm disk
lvchange -ay vol_group/logical_vol
3）通过集群文件系统（cluster filesystem: OCFS2,GFS,etc. ）来控制读写 iscsi 存储
这就是本文的初衷，接下来展开记录 OCFS2 是如何使用的。
二、配置
1、准备安装 oracle uek 内核（因为新版本的 ocfs2 内核模块被 uek 整合了）
[root@n149 ~]# wget http://public-yum.oracle.com/public-yum-ol6.repo -O /etc/yum.repos.d/public-yum-ol6.repo
[root@n149 ~]# rpm --import http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
[root@n149 ~]# yum install yum-plugin-downloadonly -y
[root@n149 ~]# mkdir /tmp/ocfs2 && cd /tmp/ocfs2/
[root@n149 ocfs2]# yum install --downloadonly --downloaddir=/tmp/ocfs2/ kernel-uek ocfs2-tools -y
2、安装 uek
[root@n149 ocfs2]# yum localinstall kernel-uek*
使用新的内核，编辑 grub 调整启动顺序：
[root@n149 ocfs2]# vim /etc/grub.conf
重启系统后查看：
[root@n149 ocfs2]# uname -r
2.6.39-400.290.1.el6uek.x86_64
查看 ocfs2 模块是否加载：
[root@n149 ocfs2]# modprobe ocfs2
[root@n149 ocfs2]# lsmod |grep oc
ocfs2 1094439 0
ocfs2_nodemanager 223253 1 ocfs2
configfs 29308 2 ocfs2_nodemanager
ocfs2_stackglue 11896 1 ocfs2
jbd2 93114 2 ocfs2,ext4
3、安装 ocfs2-tools
[root@n149 ocfs2]# yum localinstall ocfs2-tools* -y
4、配置 ocfs2 服务
[root@n149 ~]# vim /etc/hosts
10.50.200.149 n149.ocfs2.test
10.50.200.150 n150.ocfs2.test
[root@n149 ~]# service o2cb load
checking debugfs...
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
[root@n149 ~]# service o2cb online
checking debugfs...
Setting cluster stack "o2cb": OK
Checking O2CB cluster configuration : Failed
符合预期，继续配置集群：
[root@n149 ~]# o2cb_ctl -C -n prdcluster -t cluster -a name=prdcluster
[root@n149 ~]# o2cb_ctl -C -n n149.ocfs2.test -t node -a number=0 -a ip_address=10.50.200.149 -a ip_port=11111 -a cluster=prdcluster
[root@n149 ~]# o2cb_ctl -C -n n150.ocfs2.test -t node -a number=1 -a ip_address=10.50.200.150 -a ip_port=11111 -a cluster=prdcluster
[root@n149 ~]# cat /etc/ocfs2/cluster.conf
node:
name = n149.ocfs2.test
cluster = prdcluster
number = 0
ip_address = 10.50.200.149
ip_port = 11111
node:
name = n150.ocfs2.test
cluster = prdcluster
number = 1
ip_address = 10.50.200.150
ip_port = 11111
cluster:
name = prdcluster
heartbeat_mode = local
node_count = 2
[root@n149 ~]# service o2cb configure
[root@n149 ~]# service o2cb online
上述操作，在每个节点上执行。
其他可选操作：
[root@n149 ~]# service o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster "prdcluster": Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Heartbeat mode: Local
Checking O2CB heartbeat: Not active
Debug file system at /sys/kernel/debug: mounted
另外：确认防火墙放行了对应的端口，本例是 tcp port 11111
5、配置 iscsi 存储
[root@n149 ocfs2]# iscsiadm -m discovery -t st -p 10.50.200.41
[root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.101
[root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.102
[root@n149 ocfs2]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea --login
[root@n149 ocfs2]# yum install device-mapper-multipath -y
[root@n149 ocfs2]# modprobe dm-multipath
[root@n149 ocfs2]# mpathconf
[root@n149 ocfs2]# cat /etc/multipath.conf
defaults {
path_grouping_policy multibus
path_checker readsector0
rr_min_io_rq 10
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_names yes
}
multipaths {
multipath {
wwid 3600a098000a64d8b000002a4582305f8
alias test_block_a
}
}
[root@n149 ocfs2]# service multipathd start
[root@n149 ocfs2]# chkconfig multipathd on
[root@n149 ocfs2]# multipath -ll
test_block_a (3600a098000a64d8b000002a4582305f8) dm-0 DELL,MD38xxi
size=100G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 4:0:0:0 sda 8:0 active ready running
`- 5:0:0:0 sdb 8:16 active ready running
6、使用 iscsi 存储
挂载参数请参考：man mkfs.ocfs2
[root@n149 ~]# mkfs.ocfs2 -b 4K -C 64K -L testblocka -T datafiles -F /dev/mapper/test_block_a
mkfs.ocfs2 1.8.6
Cluster stack: classic o2cb
Overwriting existing ocfs2 partition.
WARNING: Cluster check disabled.
Proceed (y/N): y
Filesystem Type of datafiles
Label: testblocka
Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bg
Block size: 4096 (12 bits)
Cluster size: 65536 (16 bits)
Volume size: 107374182400 (1638400 clusters) (26214400 blocks)
Cluster groups: 51 (tail covers 25600 clusters, rest cover 32256 clusters)
Extent allocator size: 41943040 (10 groups)
Journal size: 33554432
Node slots: 8
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 4 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful
[root@n149 ~]# cat <<_EOF >>/etc/fstab
/dev/mapper/test_block_a /mnt/tba ocfs2 defaults 0 0
_EOF
[root@n149 ~]# mkdir /mnt/tba
[root@n149 ~]# mount -a
[root@n149 ~]# df -h /mnt/tba
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/test_block_a 100G 586M 100G 1% /mnt/tba
在另一个节点上执行：
[root@n150 ~]# cat <<_EOF >>/etc/fstab
/dev/mapper/test_block_a /mnt/tba ocfs2 defaults 0 0
_EOF
[root@n150 ~]# mkdir /mnt/tba
[root@n150 ~]# mount -a
[root@n150 ~]# df -h /mnt/tba
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/test_block_a 100G 586M 100G 1% /mnt/tba
查看对应的日志：
Nov 16 13:46:01 n149 kernel: o2hb: Heartbeat mode set to local
Nov 16 13:46:01 n149 o2hbmonitor: Starting
Nov 16 13:47:42 n149 kernel: OCFS2 1.8.0
Nov 16 13:47:43 n149 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 1 ) 1 nodes
Nov 16 13:47:43 n149 kernel: ocfs2: Mounting device (251,0) on (node 1, slot 0) with ordered data mode.
Nov 16 13:48:16 n149 kernel: o2net: Connection to node n150.ocfs2.test (num 0) at 10.50.200.150:11111 shutdown, state 7
Nov 16 13:48:18 n149 kernel: o2net: Connected to node n150.ocfs2.test (num 0) at 10.50.200.150:11111
Nov 16 13:48:22 n149 kernel: o2dlm: Node 0 joins domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodes
Nov 16 13:45:43 n150 kernel: o2hb: Heartbeat mode set to local
Nov 16 13:45:43 n150 o2hbmonitor: Starting
Nov 16 13:48:18 n150 kernel: o2net: Accepted connection from node n149.ocfs2.test (num 1) at 10.50.200.149:11111
Nov 16 13:48:22 n150 kernel: OCFS2 1.8.0
Nov 16 13:48:22 n150 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodes
Nov 16 13:48:22 n150 kernel: ocfs2: Mounting device (251,0) on (node 0, slot 1) with ordered data mode.
2个节点的 heartbeat 的状态也都变成了 Active
[root@n149 ~]# service o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster "prdcluster": Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Heartbeat mode: Local
Checking O2CB heartbeat: Active
Debug file system at /sys/kernel/debug: mounted
7、测试文件读写
【2个节点同时写一个文件】
[root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done
[root@n150 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done
[root@n149 ~]# cat /mnt/tba/1.log
n149.ocfs2.test -> [Wed Nov 16 13:55:20 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016]
n150.ocfs2.test -> [Wed Nov 16 13:55:29 CST 2016]
结论：符合预期。
【一个节点写，另一个节点读】
[root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done
[root@n150 ~]# tail -f /mnt/tba/1.log
n149.ocfs2.test -> [Wed Nov 16 13:57:42 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:43 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:44 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:45 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:46 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:47 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:48 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:49 CST 2016]
n149.ocfs2.test -> [Wed Nov 16 13:57:50 CST 2016]
^C
结论：符合预期。
三、问题解释
1、cluster 处于 Offline 的状态
[root@n150 ~]# mount -a
mount.ocfs2: Internal logic failure while trying to join the group
[root@n150 ~]# service o2cb status
Checking O2CB cluster "prdcluster": Offline
则请检查各节点的域名是否能正常解析，ocfs2 要用到 hostname 来提供服务。有一种情况是：
上述节点的 hostname 为：
n149.test.com
n150.test.com
而配置 ocfs2 是在 hosts 中写死了：
10.50.200.149 n149.ocfs2.test
10.50.200.150 n150.ocfs2.test
而实际上，ocfs2 使用的却是 hostname，因而节点之间的连接异常。
2、heartbeat 处于 Not active 状态
挂载卷后 heartbeat 才会启动，因此，先操作后再观察。
ZYXW、参考
1、[ovirt-users] shared storage with iscsi
http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi
2、Linux : Oracle unbreakable kernel and OCFS2 on CentOS/RHEL6
http://www.itechlounge.net/2014/08/linux-oracle-unbreakable-kernel-and-ocfs2-on-centosrhel6/
3、Oracle Linux 6: Create an OCFS2 Cluster and Filesystem
https://www.itfromallangles.com/2013/03/oracle-linux-6-create-an-ocfs2-cluster-and-filesystem/
4、Installing And Configuring An OCFS2 Clustered File System
http://www.shapeblue.com/installing-and-configuring-an-ocfs2-clustered-file-system/
5、解决o2cb status时状态始终为offline(o2cb_ctl: Internal logic failure failed while adding RAC1)
http://www.blogjava.net/KingKong/archive/2011/06/06/351826.html
6、O2CB heartbeat: Not active
https://oss.oracle.com/pipermail/ocfs2-users/2009-April/003477.html