1.配置环境 (root用户)
(1) 在两台机器上分别配置/etc/hosts
202.117.10.37 amao991
202.117.10.50 amao992
(2) 配置无密码登录(在amao991上)
[root@amao991 ~]# ssh-keygen -t rsa
[root@amao991 ~]# cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
[root@amao991 ~]# scp ~/.ssh/* root@amao992:~/.ssh/
(3) 机器角色分配
amao991作为metadata server
amao991和amao992同时作为I/O server
amao991和amao992同时作为client
2. 安装pvfs-2.8.2 (root用户在amao991和amao992上分配编译安装)
(0)预备
在/root下创建 program/pvfs2-install目录,作为安装目录
soft目录(放置pvfs-2.8.2.tar.gz)
(1) 解压
[root@amao991 soft]# tar -zxvf pvfs-2.8.2.tar.gz
(2) 编译安装
[root@amao991 pvfs-2.8.2]# ./configure -prefix=/root/program/pvfs2-install/
-with-kernel=/usr/src/kernels/2.6.18-164.el5-i686/
-with-mpi=/root/program/mpich2-install/
[root@amao991 pvfs-2.8.2]# make
[root@amao991 pvfs-2.8.2]# make install
3. 服务器端配置
(1)amao991作metadata server,amao991和amao992作I/O节点
(2) 在amao991上生成配置文件
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-genconfig /etc/pvfs2-fs.conf
********************************************************************** Welcome to the PVFS2 Configuration Generator: This interactive script will generate configuration files suitable for use with a new PVFS2 file system. Please see the PVFS2 quickstart guide for details. ********************************************************************** You must first select the network protocol that your file system will use. The only currently supported options are "tcp", "gm", "mx", "ib", and "portals". (For multi-homed configurations, use e.g. "ib,tcp".) * Enter protocol type [Default is tcp]: Choose a TCP/IP port for the servers to listen on. Note that this script assumes that all servers will use the same port number. * Enter port number [Default is 3334]: Choose a directory for each server to store data in. * Enter directory name: [Default is /pvfs2-storage-space]: Choose a file for each server to write log messages to. * Enter log file location [Default is /tmp/pvfs2-server.log]: Next you must list the hostnames of the machines that will act as I/O servers. Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}". * Enter hostnames [Default is localhost]: node15, node16 Use same servers for metadata? (recommended) * Enter yes or no [Default is yes]: no Now list the hostnames of the machines that will act as Metadata servers. This list may or may not overlap with the I/O server list. * Enter hostnames [Default is localhost]: node15 Configured a total of 2 servers: 2 of them are I/O servers. 1 of them are Metadata servers. * Would you like to verify server list (y/n) [Default is n]? y ****** I/O servers: node15 node16 ****** Metadata servers: node15 * Does this look ok (y/n) [Default is y]? y Writing fs config file... done(3)把配置文件从amao991上拷贝到amao992上,放在相同的路径下
[root@amao991 ~]# scp /etc/pvfs2-fs.conf amao992:/etc/
(4)在amao991和amao992上分别设置服务,并开机启动
[root@amao991 ~]# cp ~/soft/pvfs-2.8.2/examples/pvfs2-server.rc /etc/rc.d/init.d/pvfs2-server
[root@amao991 ~]# chmod a+x /etc/rc.d/init.d/pvfs2-server
[root@amao991 ~]# chkconfig pvfs2-server on
在amao992上的执行同上,也可以在amao991上对amao992执行操作,如下:
[root@amao991 ~]# scp ~/soft/pvfs-2.8.2/examples/pvfs2-server.rc amao992:/etc/rc.d/init.d/pvfs2-server
[root@amao991 ~]# ssh amao992 /bin/chmod a+x /etc/rc.d/init.d/pvfs2-server
[root@amao991 ~]# ssh amao992 /sbin/chkconfig pvfs2-server on
(5)创建存储空间(分别在两个IO Server上)
[root@amao991 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf -f
[root@amao991 ~]# ll / 会发现多了/pvfs2-storage-space
同样在amao992上执行
[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf -f
[root@amao992 ~]# ll / 会发现多了/pvfs2-storage-space
(6) 手动启动metadata server和I/O server(即就是在amao991和amao992上执行)
[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf
[S 12/23 18:09] PVFS2 Server on node amao992 version 2.8.2 starting...
可以用netstat -nplt查看该服务已经启动了,占用3334端口
同样在amao992上执行
[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf
4. 客户端配置
也amao991作client为例(任何其他机器都可以作client),创建挂在点,并设置开机挂在文件pvfs2tab
[root@amao991 ~]# mkdir /mnt/pvfs2
[root@amao991 ~]# touch /etc/pvfs2tab
[root@amao991 ~]# chmod a+r /etc/pvfs2tab
[root@amao991 ~]# vi /etc/pvfs2tab
[root@amao991 ~]# cat /etc/pvfs2tab
tcp://amao992:3334/pvfs2-fs /mnt/pvfs2 pvfs2 defaults,noauto 0 0 #noauto on mount -a
说明:
(1)其实把上面这一行放入/etc/fstab也是可以的,就没必要创建/etc/pvfs2tab文件了
(2)本例子中存储节点有两个amao991和amao992,当client端挂在server时,因为server是平等身份的,所以,随便挂在哪一个都可以,本例子中挂在了tcp://amao992:3334/pvfs2-fs
(3)当有多个client时,不同的Client尽量不挂在同一个server,这样做可以分散input/output负载
(4)设置完成后,重启client,就会自动mount了,然后就可以测试了
5. 测试(在client端测试借助pvfs2-ping, pvfs-cp, pvfs-ls命令来测试)
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ping -m /mnt/pvfs2/
(1) Parsing tab file... (2) Initializing system interface... (3) Initializing each file system found in tab file: /etc/pvfs2tab... PVFS2 servers: tcp://amao992:3334 Storage name: pvfs2-fs Local mount point: /mnt/pvfs2 /mnt/pvfs2: Ok (4) Searching for /mnt/pvfs2/ in pvfstab... PVFS2 servers: tcp://amao992:3334 Storage name: pvfs2-fs Local mount point: /mnt/pvfs2 meta servers: tcp://amao991:3334 data servers: tcp://amao991:3334 tcp://amao992:3334 (5) Verifying that all servers are responding... meta servers: tcp://amao991:3334 Ok data servers: tcp://amao991:3334 Ok tcp://amao992:3334 Ok (6) Verifying that fsid 1249973606 is acceptable to all servers... Ok; all servers understand fs_id 1249973606 (7) Verifying that root handle is owned by one server... Root handle: 1048576 Ok; root handle is owned by exactly one server. ============================================================= The PVFS2 filesystem at /mnt/pvfs2/ appears to be correctly configured.
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ls /mnt/pvfs2
lost+found //第一次ls,没任何东西
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-cp -t /usr/lib/libc.a /mnt/pvfs2/testfile
Wrote 2817820 bytes in 1.041003 seconds. 2.581436 MB/seconds //cp了一些东西放进去
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ls -alh /mnt/pvfs2
drwxrwxrwx 1 root root 4.0K 2011-12-23 18:37 . //果真有东西了
drwxrwxrwx 1 root root 4.0K 2011-12-23 18:37 .. (faked)
-rw-r--r-- 1 root root 2.7M 2011-12-23 18:37 testfile
drwxrwxrwx 1 root root 4.0K 2011-12-23 18:06 lost+found
[root@amao991 /]# ~/program/pvfs2-install/bin/pvfs2-cp -t /mnt/pvfs2/testfile /tmp/testfile-out
Wrote 2817820 bytes in 0.157310 seconds. 17.082721 MB/seconds //把放进去的东西再拷出来,放在/tmp下面
[root@amao991 /]# diff /tmp/testfile-out /usr/lib/libc.a //把考出来的文件和原来的文件比较,发现一模一样
6.总结
(1)client把I/O servers(不管I/O server有多少个节点)之一挂在到本地/mnt/pvfs2目录后,就可以像访问本地目录一样对其进行读/写/操作了
(2)唯一不同的是,不是用linux自带的读写命令(如ls, cp),而要用到pvfs2的命令(pvfs2-ls, pvfs2-cp),故在client上也应该把这些命令文件放置好
(3)为了在client上执行命令时,不用绝对路径或者相对路径,可以把防止pvfs2命令的目录加入到环境变量文件中
(4)通过client给/mnt/pvfs2写入一些文件后,分别在amao991和amao992两个节点上通过linux自带的ll查看本地存储目录/pvfs2-storage-space/
du /pvfs2-storage-space/6b24f650/bstreams/
(a) 当写入的文件非常小(例如8KB),查看后,发现该文件并没有切分,而是整个只放在一个data server上。
(b)当写入的文件有点大(例如4.2MB),查看后,发现该文件被一分为二,在每个data server节点上有一个块。