Ubuntu下安装GlusterFS
1 安装
1.1 配置要求
1 硬件要求
GlusterFS支持大多数的符合商业标准的x86-64bit硬件平台。存储可以是直接附加的存储，RAID盘，以SATA/SAS/FC盘做后端的FC/Infiniband/iSCSI SAN盘。
2 网路要求
GlusterFS支持千兆网，百兆网和Infiniband（光纤）。
3 操作系统要求
GlusterFS可以支持很多POSIX兼容的系统，如GNU/Linux, FreeBSD, OpenSolaris, Solaris, 和Mac OS X。Gluster建议采用Ext3, Ext4或者ZFS作为子系统磁盘的格式。Gluster对GNU/linux, Mac OS X, FreeBSD和OpenSolaris有内嵌的支持。
1.2 Ubuntu下安装
Ubuntu 14: install software-properties-common:
sudo apt-get install software-properties-common
Then add the community GlusterFS PPA:
sudo add-apt-repository ppa:gluster/glusterfs-3.8
sudo apt-get update
Finally, install the packages:
sudo apt-get install glusterfs-server1.3 gluster安装包下载
https://download.gluster.org/pub/gluster/glusterfs/
1.4 源码安装
1.4.1 安装
- ./configure
- make
- make install
1.4.2 问题
- # ./configure
- configure: error: no acceptable C compiler found in $PATH
解决：apt-get install build-essential
- configure: error: Flex or lex required to build glusterfs.
- configure: error: GNU Bison required to build glusterfs.
- # ./configure
- configure: error: OpenSSL crypto library is required to build glusterfs
解决：apt-get install libssl-dev
- error: Support for POSIX ACLs is required
解决：apt-get install libacl1 libacl1-dev
- checking for SQLITE... configure: error: pass --disable-tiering to build without sqlite
- configure: error: liburcu-bp not found
- # /usr/local/sbin/gluster
- /usr/local/sbin/gluster: error while loading shared libraries: libglusterfs.so.0: cannot open shared object file: No such file or directory
1.4.3 卸载
执行make命令：
- make uninstall-recursive
- make uninstall
- make mostlyclean-recursive
- make clean-recursive
- make distclean-recursive
- # for i in `find / -name "gluster*"`; do rm -rf $i; done
2 安装
2.1 部署拓扑
3台存储服务器: 192.168.4.132、192.168.4.133、192.168.4.134- root@block2:/glusterfs# cat /etc/hosts
- 127.0.0.1 localhost
- 192.168.4.130 controller
- 192.168.4.131 network
- 192.168.4.132 compute
- 192.168.4.133 block1
- 192.168.4.134 block2
2.2 安装
在所有存储服务器上执行安装。
2.2.1 如果出现错误：
- W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/binary-amd64/Packages Hash Sum mismatch
- W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/binary-i386/Packages Hash Sum mismatch
- W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/i18n/Translation-en Hash Sum mismatch
- E: Some index files failed to download. They have been ignored, or old ones used instead.
- root@block1:~# ls /var/lib/apt/lists/ppa*
- /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_InRelease
- /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_binary-amd64_Packages
- /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_binary-i386_Packages
- /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_i18n_Translation-en
解决方法：
将/var/lib/apt/lists/partial/下的所有文件删除，但是这可以解决大部分问题，个别源不能解决，
注：经过长时间的实践，这个解决方法不是万能的。
2.2.2 安装glusterfs-server时，会安装glusterfs-client
- root@block2:/glusterfs# apt-get install glusterfs-server
- Reading package lists... Done
- Building dependency tree
- Reading state information... Done
- The following extra packages will be installed:
- attr glusterfs-client glusterfs-common liblvm2app2.2 liburcu1
- The following NEW packages will be installed:
- attr glusterfs-client glusterfs-common glusterfs-server liblvm2app2.2
- liburcu1
- 0 upgraded, 6 newly installed, 0 to remove and 4 not upgraded.
- Need to get 3,389 kB of archives.
- After this operation, 16.0 MB of additional disk space will be used.
- Do you want to continue? [Y/n]
安装成功后，gluster会运行。查看版本：
- root@block2:/glusterfs# glusterfs --version
- glusterfs 3.8.10
- root@block2:/glusterfs# ps -ef | grep gluster
- root 17920 1 0 17:58 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
2.3 查看运行端口
- root@compute: # netstat -pan | grep glus
- tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 5323/glusterd
- tcp 0 0 192.168.4.132:24007 192.168.4.133:1023 ESTABLISHED 5323/glusterd
- tcp 0 0 192.168.4.132:1023 192.168.4.134:24007 ESTABLISHED 5323/glusterd
- tcp 0 0 192.168.4.132:24007 192.168.4.134:1020 ESTABLISHED 5323/glusterd
- tcp 0 0 192.168.4.132:1021 192.168.4.133:24007 ESTABLISHED 5323/glusterd
2.4 服务启停
使用命令：service glusterfs-server start/stop
- root@compute:~# ps -ef | grep gluster
- root 10929 1 0 Mar16 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
- root 12215 1650 0 10:22 pts/0 00:00:00 grep --color=auto gluster
- root@compute:~# ls /etc/init.d/glusterfs*
- /etc/init.d/glusterfs-server
- root@compute:~# service glusterfs-server stop
- glusterfs-server stop/waiting
- root@compute:~# ps -ef | grep gluster
- root 12225 1650 0 10:23 pts/0 00:00:00 grep --color=auto gluster
- root@compute:~# service glusterfs-server start
- glusterfs-server start/running, process 12233
- root@compute:~# ps -ef | grep gluster
- root 12233 1 0 10:23 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
- root 12467 1650 0 10:23 pts/0 00:00:00 grep --color=auto gluster
3 集群关联
3.1创建GlusterFS集群
$ gluster peer probe SERVER SERVER表示存储服务器的hostname。在服务器192.168.4.132上执行：
- root@compute:~# gluster peer probe
- Usage: peer probe <HOSTNAME>
- root@compute:~# gluster peer probe block1
- peer probe: success
- root@compute:~# gluster peer probe block2
- peer probe: success
注意：在构建
GlusterFS
集群时，需要在其中任意一台存储服务器上依次将其他存储服务器添加到集群中。
3.2 查看集群信息
指令：gluster peer status
- root@compute:/etc/glusterfs# gluster peer status
- Number of Peers: 2
- Hostname: block1
- Port: 24007
- Uuid: ac468b50-3d66-4290-aa8d-cb01279e9bf2
- State: Peer in Cluster (Connected)
- Hostname: block2
- Port: 24007
- Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
- State: Peer in Cluster (Connected)
- root@block1:/etc/glusterfs# gluster peer status
- Number of Peers: 2
- Hostname: 192.168.4.132
- Port: 24007
- Uuid: 9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d
- State: Peer in Cluster (Connected)
- Hostname: block2
- Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
- State: Peer in Cluster (Connected)
- root@block1:/etc/glusterfs# gluster volume info
- No volumes present
Connected,说明集群节点添加成功，但其中compute是以192.168.4.132这个ip加入集群的，如果需要换成hostname，登陆集群中的集群中的另一个节点，执行：
- root@block1:~# gluster peer detach 192.168.4.132
- peer detach: success
- root@block1:~#
- root@block1:~# gluster peer status
- Number of Peers: 1
- Hostname: block2
- Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
- State: Peer in Cluster (Connected)
- root@block1:~#
- root@block1:~# gluster peer probe compute
- peer probe: success.
- root@block1:~#
- root@block1:~# gluster peer status
- Number of Peers: 2
- Hostname: block2
- Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
- State: Peer in Cluster (Connected)
- Hostname: compute
- Uuid: 9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d
- State: Peer in Cluster (Connected)
4 Volume卷操作
4.1．创建volume
a. 单磁盘，调试环境推荐
sudo gluster volume create vol_name 172.17.0.2:/d/disk0
b. 多磁盘，无raid，试验、测试环境推荐。
sudo gluster volume create vol_name 172.17.0.2:/d/disk0 172.17.0.3:/d/disk0 172.17.0.4:/d/disk0 172.17.0.5:/d/disk0
c. 多磁盘，有raid1。线上高并发环境推荐。
- Usage: volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
实验命令是在根目录下创建目录/glusterfs，要在所有的3台服务器上都创建这个目录，否则会出错。
- root@compute:~# gluster volume create zhai-volume replica 2 192.168.4.132:/glusterfs/brick1 block1:/glusterfs/brick1 block2:/glusterfs/brick1
- number of bricks is not a multiple of replica count
- Usage: volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
- root@compute:~#
- root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2
- volume create: zhai-volume: failed: The brick compute:/glusterfs/brick1 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior.
- root@compute:~#
- root@compute:~# ls /glusterfs/
- root@compute:~#
- root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2 force
- volume create: zhai-volume: failed: Staging failed on block2. Error: Failed to create brick directory for brick block2:/glusterfs/brick1. Reason : No such file or directory
- Staging failed on block1. Error: Failed to create brick directory for brick block1:/glusterfs/brick1. Reason : No such file or directory
- root@compute:~#
- root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2 force
- volume create: zhai-volume: success: please start the volume to access data
- root@compute:~#
- root@compute:~# gluster volume info
- Volume Name: zhai-volume
- Type: Distributed-Replicate
- Volume ID: 779a7440-e127-43c6-a13f-7420ab8baec8
- Status: Created
- Snapshot Count: 0
- Number of Bricks: 3 x 2 = 6
- Transport-type: tcp
- Bricks:
- Brick1: compute:/glusterfs/brick1
- Brick2: compute:/glusterfs/brick2
- Brick3: block1:/glusterfs/brick1
- Brick4: block1:/glusterfs/brick2
- Brick5: block2:/glusterfs/brick1
- Brick6: block2:/glusterfs/brick2
- Options Reconfigured:
- transport.address-family: inet
Note : The number of bricks should be a multiple of the replica count for a distributed replicated volume. Also, the order in which bricks are specified has a great effect on data protection. Each replica_count consecutive bricks in the list you give will form a replica set, with all replica sets combined into a volume-wide distribute set.
volume创建成功后，会在每台服务器的/glusterfs目录下生成brick1和brick2目录。
查看日志：
- root@compute:~# tail -20 /var/log/glusterfs/cli.log
- [2017-03-17 03:01:38.518626] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.10
- [2017-03-17 03:01:38.608896] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
- [2017-03-17 03:01:38.608958] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
- [2017-03-17 03:01:38.619870] I [cli-rpc-ops.c:1117:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
- [2017-03-17 03:01:38.619953] I [input.c:31:cli_batch] 0-: Exiting with: -1
- [2017-03-17 03:02:34.746102] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.10
- [2017-03-17 03:02:34.837934] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
- [2017-03-17 03:02:34.838008] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
- [2017-03-17 03:02:36.111778] I [cli-rpc-ops.c:1117:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
- [2017-03-17 03:02:36.111888] I [input.c:31:cli_batch] 0-: Exiting with: 0
4.2．启动volume
刚创建好的volume还没有运行，需要执行运行命令方可使用。 在其中一台服务器上执行。sudo gluster volume start vol_name
- root@compute:~# ps -ef | grep volume
- root 12663 1650 0 11:16 pts/0 00:00:00 grep --color=auto volume
- root@compute:~# gluster volume start zhai-volume
- volume start: zhai-volume: success
- root@compute:~# ps -ef | grep volume
- root 12683 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s compute --volfile-id zhai-volume.compute.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/compute-glusterfs-brick1.pid -S /var/run/gluster/233b16b60a6dacf67c1b711bb1e7e89c.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
- root 12702 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s compute --volfile-id zhai-volume.compute.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/compute-glusterfs-brick2.pid -S /var/run/gluster/55ba583ae215e8215759c8d1aca9548c.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153
- root 12729 1650 0 11:16 pts/0 00:00:00 grep --color=auto volume
- root@block1:~# ps -ef | grep zhai-volume
- root 26213 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s block1 --volfile-id zhai-volume.block1.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/block1-glusterfs-brick1.pid -S /var/run/gluster/98664ae15cd31af2fd3394f1a239d85b.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=ac468b50-3d66-4290-aa8d-cb01279e9bf2 --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
- root 26232 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s block1 --volfile-id zhai-volume.block1.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/block1-glusterfs-brick2.pid -S /var/run/gluster/e4bf4eb272d50def1259ad279e81bc61.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=ac468b50-3d66-4290-aa8d-cb01279e9bf2 --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153
- root@block2:~# ps -ef | grep zhai-volume
- root 24570 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s block2 --volfile-id zhai-volume.block2.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/block2-glusterfs-brick1.pid -S /var/run/gluster/b5c757abb3485edb13b25f9aaac3c387.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=06288da0-1b18-42fb-a7b8-cd9ca24d5f26 --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
- root 24589 1 0 11:16 ? 00:00:00 /usr/sbin/glusterfsd -s block2 --volfile-id zhai-volume.block2.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/block2-glusterfs-brick2.pid -S /var/run/gluster/0acc5e5ccc951203695b9654ef4dbd67.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=06288da0-1b18-42fb-a7b8-cd9ca24d5f26 --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153
4.3．挂载volume
4.3.1 在3台服务器中任一服务器上挂载都可以。
For this step, we will use one of the servers to mount the volume. Typically, you would do this from an external machine, known as a "client". Since using this method would require additional packages to be installed on the client machine, we willuse one of the servers as a simple place to test first, as if it were that "client".
# 挂载任一节点即可(推荐)
- root@compute:~# mount -t glusterfs 192.168.4.132:/zhai-volume /mnt
- root@compute:~# df -h
- Filesystem Size Used Avail Use% Mounted on
- udev 3.9G 12K 3.9G 1% /dev
- tmpfs 788M 660K 787M 1% /run
- /dev/dm-0 909G 2.4G 860G 1% /
- none 4.0K 0 4.0K 0% /sys/fs/cgroup
- none 5.0M 0 5.0M 0% /run/lock
- none 3.9G 0 3.9G 0% /run/shm
- none 100M 0 100M 0% /run/user
- /dev/sda2 237M 91M 134M 41% /boot
- /dev/sda1 511M 3.4M 508M 1% /boot/efi
- 192.168.4.132:/zhai-volume 2.7T 6.0G 2.6T 1% /mnt
4.3.2 在3台服务器之外的服务器上挂载
需要安装glusterfs-client包
否则，出错：
- root@network:~# mount -t glusterfs 192.168.4.132:/zhai-volume /mnt
- mount: unknown filesystem type 'glusterfs'
安装
- sudo apt-get install glusterfs-client
安装后，执行挂载
- # mount -t glusterfs 10.10.30.4:/zhai-volume /mnt
- Mount failed. Please check the log file for more details.
4.3.3 开机挂载
# 若需要开机自动挂载，请更新/etc/fstab
4.4．使用GlusterFS
a. 挂载了GlusterFS的某个卷后，就可以将其当做本地文件访问，代码中只需使用原生的文件api即可。这种方式使用不一定需要root权限，只要拥有对应目录或文件的权限即可。b. 直接API方式，这种方式需要root权限才能使用，并且java、python、ruby的api包装目前都不够完整，一般情况不推荐使用。
到
测试1：
到/mnt目录下创建文件：
- root@compute:/mnt# uname -a >info
- root@compute:/mnt# ls
- info
- root@compute:/mnt# cat info
- Linux compute 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
- root@block2:~# ls /glusterfs/brick1/
- info
- root@block2:~# ls /glusterfs/brick2
- info
- root@block2:~# cat /glusterfs/brick1/info
- Linux compute 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
测试2：
mount -t glusterfs server1:/gv0 /mnt
for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done
First, check the mount point:
ls -lA /mnt | wc -l
You should see 100 files returned. Next, check the GlusterFS mount points on each server:
ls -lA /data/brick1/gv0
You should see 100 files on each server using the method we listed here. Without replication, in a distribute only volume (not detailed here), you should see about 50 files on each one.
4.5．卸载volume
卸载与挂载操作是一对。虽然没有卸载也可以停止volume，但是这样做是会出问题，如果集群较大，可能导致后面volume启动失败。sudo umount /mnt
4.6．停止volume
停止与启动操作是一对。停止前最好先卸载所有客户端。sudo gluster volume stop vol_name
4.7．删除volume
删除与创建操作是一对。删除前需要先停止volume。在生产上一般不会删除volumesudo gluster volume delete vol_name
5 使用独立存储
使用独立/dev/sdb
5.1 分区
fdisk /dev/sdb //创建分区，输入,n,p,w，创建新主分区并写入分区表
5.2 Format and mount the bricks
(on both nodes): Note: These examples are going to assume the brick is going to reside on /dev/sdb1.
mkfs.xfs -i size=512 /dev/sdb1
mkdir -p /data/brick1
echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
mount -a && mount
You should now see sdb1 mounted at /data/brick1
- root@block1:~# mkfs.xfs -i size=512 /dev/sdb1
- root@compute:~# echo '/dev/sdb1 /glusterfs xfs defaults 1 2' >> /etc/fstab
- root@compute:~# mount -a && mount
- /dev/mapper/compute--vg-root on / type ext4 (rw,errors=remount-ro)
- proc on /proc type proc (rw,noexec,nosuid,nodev)
- sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
- none on /sys/fs/cgroup type tmpfs (rw)
- none on /sys/fs/fuse/connections type fusectl (rw)
- none on /sys/kernel/debug type debugfs (rw)
- none on /sys/kernel/security type securityfs (rw)
- none on /sys/firmware/efi/efivars type efivarfs (rw)
- udev on /dev type devtmpfs (rw,mode=0755)
- devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
- tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
- none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
- none on /run/shm type tmpfs (rw,nosuid,nodev)
- none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
- none on /sys/fs/pstore type pstore (rw)
- tracefs on /var/lib/ureadahead/debugfs/tracing type tracefs (rw,relatime)
- /dev/sda2 on /boot type ext2 (rw)
- /dev/sda1 on /boot/efi type vfat (rw)
- systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
- rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
- 192.168.4.132:/zhai-volume on /mnt type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
- /dev/sdb1 on /glusterfs type xfs (rw)
- root@compute:~# df -h
- Filesystem Size Used Avail Use% Mounted on
- udev 3.9G 12K 3.9G 1% /dev
- tmpfs 788M 660K 787M 1% /run
- /dev/dm-0 909G 2.4G 860G 1% /
- none 4.0K 0 4.0K 0% /sys/fs/cgroup
- none 5.0M 0 5.0M 0% /run/lock
- none 3.9G 0 3.9G 0% /run/shm
- none 100M 0 100M 0% /run/user
- /dev/sda2 237M 91M 134M 41% /boot
- /dev/sda1 511M 3.4M 508M 1% /boot/efi
- /dev/sdb1 932G 34M 932G 1% /glusterfs
mount之后再安装glusterfs
5.3 其它命令
- #apt-get install xfsprogs //安装xfs文件系统（如果没有的话），更安全可靠
- #lshw -short //查看硬件信息，主要是查看硬盘信息，以决定哪块硬盘作为gluster来使用
- 或使用lsblk来查看块设备
- #lsblk
6 相关问题
1 Q: 创建/启用/停用/删除volume最终都干了些什么？
(1) 创建volume能看得到的结果是: glusterd在工作目录(默认为/var/lib/glusterd)的vols目录下，创建以volume名称为名字的目录，并在这个目录下创建相关文件记录volume的相关信息；看不到的结果是glusterd程序中也记录了volume的相关信息。
(2) 启用volume的结果是在指定brick的节点上启动glusterfsd和glusterfs进程(注意：如果多个brick在同一节点下，会有多个glusterfsd进程，但只有一个glusterfs进程)，同时在日志文件中更新记录的volume相关信息。
(3) 停用volume的结果是结束glusterfsd和glusterfs进程。
(4)删除volume则是将创建volume的相关文件信息全部清除。
2 错误
- root@compute:~# gluster volume create volume-test block1:/glusterfs/repbrick block2:/glusterfs/repbrick
- volume create: volume-test: failed: Staging failed on block2. Error: Brick: block2:/glusterfs/repbrick not available. Brick may be containing or be contained by an existing brick
- Staging failed on block1. Error: Brick: block1:/glusterfs/repbrick not available. Brick may be containing or be contained by an existing brick
这是因为创建volume时，在brick对应的存储目录上，增加了"trusted.glusterfs.volume-id"和"trusted.gfid"两个扩展属性，在删除volume时并未移除这两个属性，再次创建时，对brick的存储目录进行校验，发现已经有了扩展属性，因此会有brick已经是volume的一部分的提示。
通过getfattr命令可以查看这些存储路径的扩展属性
- root@block2:~# getfattr -d -m ".*" /glusterfs/repbrick
- getfattr: Removing leading '/' from absolute path names
- # file: glusterfs/repbrick
- trusted.glusterfs.volume-id=0sS29pmfEgRDyMNCX3YAbwsQ==
设置属性
- setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/volume-down-fault/info | cut -d= -f2 | sed 's/-//g') /glusterfs/brick2
3 可执行程序
glusterfs安装后，会有gluster，glusterd，glusterfs，glusterfsd这么几个可执行程序，其作用分别为：
gluster：Glusterfs控制台管理程序（Gluster Console Manager），可以以命令形式或者交互式形式对glusterfs中的volume，brick，集群节点等信息进行查看及操作（增，删，改）。
glusterd：软链接指向glusterfsd，Glusterfs的管理进程，负责处理来自gluster的命令。
glusterfs：软链接指向glusterfsd，Glusterfs自带的客户端
glusterfsd：Glusterfs服务端程序
注：glusterd,glusterfs,glusterfsd为最终是运行同一个程序(glusterfsd)，程序内部会根据可执行程序名称加以区别。
- staticuint8_t gf_get_process_mode (char*exec_name)
- {
- char*dup_execname = NULL, *base = NULL;
- uint8_t ret = 0;
- dup_execname = gf_strdup (exec_name);
- base = basename (dup_execname);
- if(!strncmp (base, "glusterfsd", 10)) {
- ret = GF_SERVER_PROCESS;
- }else if (!strncmp(base, "glusterd", 8)) {
- ret = GF_GLUSTERD_PROCESS;
- }else {
- ret = GF_CLIENT_PROCESS;
- }
- GF_FREE (dup_execname);
- returnret;
- }
4 客户端依靠nfs协议（或gluster native协议）动态的从服务端获取卷的信息。
5 如果建立的是复制卷，gluster客户端会并发的向几个服务器写入\读取数据，而不是通过服务器之间进行数据的传输。