1、为什么使用分布式文件系统glusterfs?
答:使用持久化存储nfs,但是使用nfs这个共享存储特别简单,但是后期在扩容和管理方面都特别的不方便,在生产中一般都是使用分布式文件系统,这里使用的是分布式文件系统glusterfs。
2、什么是分布式文件系统glusterfs?
答:分布式文件系统glusterfs是一个开源分布式文件系统,具有强大的横向扩展能力,可以支持数PB存储容器和数千客户端,通过网络互联成一个并行的网络文件系统。具有可扩展性、高性能、高可用性等特点。
3、安装分布式文件系统glusterfs,需要所有节点都需要进行安装的,如下所示:
首先在三台节点上都安装glusterfs的yum源,如下所示:
1 [root@k8s-master ~]# yum install centos-release-gluster -y 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * extras: mirrors.bfsu.edu.cn 9 * updates: mirrors.bfsu.edu.cn 10 base | 3.6 kB 00:00:00 11 extras | 2.9 kB 00:00:00 12 updates | 2.9 kB 00:00:00 13 Resolving Dependencies 14 --> Running transaction check 15 ---> Package centos-release-gluster7.noarch 0:1.0-2.el7.centos will be installed 16 --> Processing Dependency: centos-release-storage-common for package: centos-release-gluster7-1.0-2.el7.centos.noarch 17 --> Running transaction check 18 ---> Package centos-release-storage-common.noarch 0:2-2.el7.centos will be installed 19 --> Finished Dependency Resolution 20 21 Dependencies Resolved 22 23 ================================================================================================================================================================================================================= 24 Package Arch Version Repository Size 25 ================================================================================================================================================================================================================= 26 Installing: 27 centos-release-gluster7 noarch 1.0-2.el7.centos extras 5.2 k 28 Installing for dependencies: 29 centos-release-storage-common noarch 2-2.el7.centos extras 5.1 k 30 31 Transaction Summary 32 ================================================================================================================================================================================================================= 33 Install 1 Package (+1 Dependent package) 34 35 Total download size: 10 k 36 Installed size: 2.4 k 37 Downloading packages: 38 (1/2): centos-release-gluster7-1.0-2.el7.centos.noarch.rpm | 5.2 kB 00:00:00 39 (2/2): centos-release-storage-common-2-2.el7.centos.noarch.rpm | 5.1 kB 00:00:00 40 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 41 Total 13 kB/s | 10 kB 00:00:00 42 Running transaction check 43 Running transaction test 44 Transaction test succeeded 45 Running transaction 46 Installing : centos-release-storage-common-2-2.el7.centos.noarch 1/2 47 Installing : centos-release-gluster7-1.0-2.el7.centos.noarch 2/2 48 Verifying : centos-release-gluster7-1.0-2.el7.centos.noarch 1/2 49 Verifying : centos-release-storage-common-2-2.el7.centos.noarch 2/2 50 51 Installed: 52 centos-release-gluster7.noarch 0:1.0-2.el7.centos 53 54 Dependency Installed: 55 centos-release-storage-common.noarch 0:2-2.el7.centos 56 57 Complete! 58 [root@k8s-master ~]#
可以看到yum源里面多了一些文件,默认安装的是CentOS-Gluster-7.repo这个7.x版本的,如下所示:
1 [root@k8s-master ~]# ls /etc/yum.repos.d/ 2 CentOS-Base.repo CentOS-CR.repo CentOS-Debuginfo.repo CentOS-fasttrack.repo CentOS-Gluster-7.repo CentOS-Media.repo CentOS-Sources.repo CentOS-Storage-common.repo CentOS-Vault.repo 3 [root@k8s-master ~]#
然后开始在三台节点上安装glusterfs-server包安装上,最主要的就是安装这个包的,如下所示:
1 [root@k8s-master ~]# yum install install glusterfs-server -y 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * centos-gluster7: mirrors.tuna.tsinghua.edu.cn 9 * extras: mirrors.bfsu.edu.cn 10 * updates: mirrors.bfsu.edu.cn 11 centos-gluster7 | 3.0 kB 00:00:00 12 centos-gluster7/7/x86_64/primary_db | 69 kB 00:00:00 13 No package install available. 14 Resolving Dependencies 15 --> Running transaction check 16 ---> Package glusterfs-server.x86_64 0:7.6-1.el7 will be installed 17 --> Processing Dependency: glusterfs = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 18 --> Processing Dependency: glusterfs-api = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 19 --> Processing Dependency: glusterfs-cli = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 20 --> Processing Dependency: glusterfs-client-xlators = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 21 --> Processing Dependency: glusterfs-fuse = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 22 --> Processing Dependency: glusterfs-libs = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 23 --> Processing Dependency: libgfapi.so.0(GFAPI_6.0)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 24 --> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_6.0)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 25 --> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_6.1)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 26 --> Processing Dependency: liburcu-bp.so.6()(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 27 --> Processing Dependency: liburcu-cds.so.6()(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 28 --> Running transaction check 29 ---> Package glusterfs.x86_64 0:3.12.2-18.el7 will be updated 30 ---> Package glusterfs.x86_64 0:7.6-1.el7 will be an update 31 ---> Package glusterfs-api.x86_64 0:3.12.2-18.el7 will be updated 32 ---> Package glusterfs-api.x86_64 0:7.6-1.el7 will be an update 33 ---> Package glusterfs-cli.x86_64 0:3.12.2-18.el7 will be updated 34 ---> Package glusterfs-cli.x86_64 0:7.6-1.el7 will be an update 35 ---> Package glusterfs-client-xlators.x86_64 0:3.12.2-18.el7 will be updated 36 ---> Package glusterfs-client-xlators.x86_64 0:7.6-1.el7 will be an update 37 ---> Package glusterfs-fuse.x86_64 0:7.6-1.el7 will be installed 38 ---> Package glusterfs-libs.x86_64 0:3.12.2-18.el7 will be updated 39 ---> Package glusterfs-libs.x86_64 0:7.6-1.el7 will be an update 40 ---> Package userspace-rcu.x86_64 0:0.10.0-3.el7 will be installed 41 --> Finished Dependency Resolution 42 43 Dependencies Resolved 44 45 ================================================================================================================================================================================================================= 46 Package Arch Version Repository Size 47 ================================================================================================================================================================================================================= 48 Installing: 49 glusterfs-server x86_64 7.6-1.el7 centos-gluster7 1.3 M 50 Installing for dependencies: 51 glusterfs-fuse x86_64 7.6-1.el7 centos-gluster7 156 k 52 userspace-rcu x86_64 0.10.0-3.el7 centos-gluster7 93 k 53 Updating for dependencies: 54 glusterfs x86_64 7.6-1.el7 centos-gluster7 640 k 55 glusterfs-api x86_64 7.6-1.el7 centos-gluster7 114 k 56 glusterfs-cli x86_64 7.6-1.el7 centos-gluster7 198 k 57 glusterfs-client-xlators x86_64 7.6-1.el7 centos-gluster7 850 k 58 glusterfs-libs x86_64 7.6-1.el7 centos-gluster7 425 k 59 60 Transaction Summary 61 ================================================================================================================================================================================================================= 62 Install 1 Package (+2 Dependent packages) 63 Upgrade ( 5 Dependent packages) 64 65 Total download size: 3.7 M 66 Downloading packages: 67 No Presto metadata available for centos-gluster7 68 warning: /var/cache/yum/x86_64/7/centos-gluster7/packages/glusterfs-cli-7.6-1.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e451e5b5: NOKEY ] 0.0 B/s | 0 B --:--:-- ETA 69 Public key for glusterfs-cli-7.6-1.el7.x86_64.rpm is not installed 70 (1/8): glusterfs-cli-7.6-1.el7.x86_64.rpm | 198 kB 00:00:00 71 (2/8): glusterfs-api-7.6-1.el7.x86_64.rpm | 114 kB 00:00:00 72 (3/8): glusterfs-libs-7.6-1.el7.x86_64.rpm | 425 kB 00:00:00 73 (4/8): glusterfs-fuse-7.6-1.el7.x86_64.rpm | 156 kB 00:00:01 74 (5/8): glusterfs-client-xlators-7.6-1.el7.x86_64.rpm | 850 kB 00:00:03 75 (6/8): glusterfs-7.6-1.el7.x86_64.rpm | 640 kB 00:00:03 76 (7/8): glusterfs-server-7.6-1.el7.x86_64.rpm | 1.3 MB 00:00:04 77 (8/8): userspace-rcu-0.10.0-3.el7.x86_64.rpm | 93 kB 00:00:13 78 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 79 Total 242 kB/s | 3.7 MB 00:00:15 80 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage 81 Importing GPG key 0xE451E5B5: 82 Userid : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/Storage)" 83 Fingerprint: 7412 9c0b 173b 071a 3775 951a d4a2 e50b e451 e5b5 84 Package : centos-release-storage-common-2-2.el7.centos.noarch (@extras) 85 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage 86 Running transaction check 87 Running transaction test 88 Transaction test succeeded 89 Running transaction 90 Updating : glusterfs-libs-7.6-1.el7.x86_64 1/13 91 Updating : glusterfs-client-xlators-7.6-1.el7.x86_64 2/13 92 Updating : glusterfs-7.6-1.el7.x86_64 3/13 93 Updating : glusterfs-api-7.6-1.el7.x86_64 4/13 94 Installing : glusterfs-fuse-7.6-1.el7.x86_64 5/13 95 Updating : glusterfs-cli-7.6-1.el7.x86_64 6/13 96 Installing : userspace-rcu-0.10.0-3.el7.x86_64 7/13 97 Installing : glusterfs-server-7.6-1.el7.x86_64 8/13 98 Cleanup : glusterfs-api-3.12.2-18.el7.x86_64 9/13 99 Cleanup : glusterfs-3.12.2-18.el7.x86_64 10/13 100 Cleanup : glusterfs-client-xlators-3.12.2-18.el7.x86_64 11/13 101 Cleanup : glusterfs-cli-3.12.2-18.el7.x86_64 12/13 102 Cleanup : glusterfs-libs-3.12.2-18.el7.x86_64 13/13 103 Verifying : glusterfs-libs-7.6-1.el7.x86_64 1/13 104 Verifying : glusterfs-api-7.6-1.el7.x86_64 2/13 105 Verifying : glusterfs-cli-7.6-1.el7.x86_64 3/13 106 Verifying : glusterfs-fuse-7.6-1.el7.x86_64 4/13 107 Verifying : glusterfs-client-xlators-7.6-1.el7.x86_64 5/13 108 Verifying : glusterfs-server-7.6-1.el7.x86_64 6/13 109 Verifying : glusterfs-7.6-1.el7.x86_64 7/13 110 Verifying : userspace-rcu-0.10.0-3.el7.x86_64 8/13 111 Verifying : glusterfs-3.12.2-18.el7.x86_64 9/13 112 Verifying : glusterfs-cli-3.12.2-18.el7.x86_64 10/13 113 Verifying : glusterfs-client-xlators-3.12.2-18.el7.x86_64 11/13 114 Verifying : glusterfs-libs-3.12.2-18.el7.x86_64 12/13 115 Verifying : glusterfs-api-3.12.2-18.el7.x86_64 13/13 116 117 Installed: 118 glusterfs-server.x86_64 0:7.6-1.el7 119 120 Dependency Installed: 121 glusterfs-fuse.x86_64 0:7.6-1.el7 userspace-rcu.x86_64 0:0.10.0-3.el7 122 123 Dependency Updated: 124 glusterfs.x86_64 0:7.6-1.el7 glusterfs-api.x86_64 0:7.6-1.el7 glusterfs-cli.x86_64 0:7.6-1.el7 glusterfs-client-xlators.x86_64 0:7.6-1.el7 glusterfs-libs.x86_64 0:7.6-1.el7 125 126 Complete! 127 [root@k8s-master ~]#
然后开始启动glusterfs,并设置开机自启动,如下所示:
1 [root@k8s-master ~]# systemctl start glusterd.service 2 [root@k8s-master ~]# systemctl enable glusterd.service 3 [root@k8s-master ~]#
生产环境可以增加一个/dev/sdb或者/dev/sdc用于挂载,将它挂载到某一个目录下面去使用的,默认只有/dev/sda。这里直接增加目录来使用了,在三台机器上都执行下面的创建目录的命令,如下所示:
1 [root@k8s-master ~]# mkdir -p /gfs/test1 2 [root@k8s-master ~]# mkdir -p /gfs/test2 3 [root@k8s-master ~]#
添加存储资源池,即将节点都组装起来,在master主节点进行操作,可以查看gluster的资源池(如果gluster没有提示,可以先退出xshell,再重新登录即可),如下所示:
1 [root@k8s-master ~]# gluster pool list 2 UUID Hostname State 3 d2ea56bf-6402-49c5-93a2-0d784110b231 localhost Connected 4 [root@k8s-master ~]# gluster pool 5 unrecognized command 6 7 Usage: gluster [options]8 Options: 9 --help Shows the help information 10 --version Shows the version 11 --print-logdir Shows the log directory 12 --print-statedumpdir Shows the state dump directory 13 14 [root@k8s-master ~]#
资源池默认只能识别出自己本身,这里需要将另外两个节点加入到资源池中,根据hostname加入到资源池,需要做host解析的,如下所示:
1 [root@k8s-master ~]# gluster peer 2 detach probe status 3 [root@k8s-master ~]# gluster peer probe k8s-node2 4 peer probe: success. 5 [root@k8s-master ~]# gluster peer probe k8s-node3 6 peer probe: success. 7 [root@k8s-master ~]#
此时,再次查看资源池,发现已经有了三个节点了,如下所示:
1 [root@k8s-master ~]# gluster pool list 2 UUID Hostname State 3 977fb392-500c-4353-b03c-0ee9e53cee29 k8s-node2 Connected 4 3b7c328f-2eff-4098-b3ac-ebc730975d22 k8s-node3 Connected 5 d2ea56bf-6402-49c5-93a2-0d784110b231 localhost Connected 6 [root@k8s-master ~]#
资源池添加完毕之后,开始进行glusterfs卷管理,创建分布式复制卷,如下所示:
可以在资源池里面使用资源池创建一个卷,在生产环境中glusterfs支持七种模式的卷,使用最多的是分布式复制卷,其他有的卷不稳定肯能会丢失书数据,最稳定的一种就是分布式复制卷,分布式复制卷要求至少四个存储单元,就是刚才创建的目录,正常情况下,一个目录就要挂载一个空的硬盘,一个硬盘就是一个存储单元,或者说一个目录就是一个存储单元,glusterfs这个分布式复制卷,最少复制的副本是2,再加上是分布式,至少要有四个目录作为存储单元。
1 [root@k8s-master ~]# gluster volume create biehl replica 2 k8s-master:/gfs/test1 k8s-master:/gfs/test2 k8s-node2:/gfs/test1 k8s-node2:/gfs/test2 force 2 volume create: biehl: success: please start the volume to access data 3 [root@k8s-master ~]#
卷已经创建成功了,现在需要启动一下才可以访问里面的数据,启动卷并进行查看卷,如下所示:
1 [root@k8s-master ~]# gluster volume start biehl 2 volume start: biehl: success 3 [root@k8s-master ~]# gluster volume info biehl 4 5 Volume Name: biehl 6 Type: Distributed-Replicate 7 Volume ID: 5729192c-627f-4444-92eb-32a245086f43 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 2 x 2 = 4 11 Transport-type: tcp 12 Bricks: 13 Brick1: k8s-master:/gfs/test1 14 Brick2: k8s-master:/gfs/test2 15 Brick3: k8s-node2:/gfs/test1 16 Brick4: k8s-node2:/gfs/test2 17 Options Reconfigured: 18 transport.address-family: inet 19 storage.fips-mode-rchecksum: on 20 nfs.disable: on 21 performance.client-io-threads: off 22 [root@k8s-master ~]#
下面开始挂载卷,可以在三台机器的任意一个节点上进行挂载,如下所示:
1 [root@k8s-master ~]# mount -t glusterfs 192.168.110.133:/biehl /mnt 2 [root@k8s-master ~]#
含义就是将192.168.110.133的biehl卷,挂载到mnt目录下面,现在可以进行查看,如下所示:
1 [root@k8s-master ~]# df -h 2 Filesystem Size Used Avail Use% Mounted on 3 /dev/mapper/centos-root 18G 14G 3.9G 79% / 4 devtmpfs 1.2G 0 1.2G 0% /dev 5 tmpfs 1.2G 0 1.2G 0% /dev/shm 6 tmpfs 1.2G 40M 1.1G 4% /run 7 tmpfs 1.2G 0 1.2G 0% /sys/fs/cgroup 8 /dev/sda1 197M 157M 41M 80% /boot 9 tmpfs 229M 12K 229M 1% /run/user/42 10 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/fbf4133aed53d4e17a706be46fadcbea9fe6eb1e615379bfaa157570f0de7bc6/merged 11 shm 64M 0 64M 0% /var/lib/docker/containers/5e72b0961647003c757fd41ffe824027c857d1f3938b9a53535645f4f372a40a/shm 12 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/80967f34f76b4afe42411f1b1dc6514b2b19d997644dfa4687a3a0924df06a08/merged 13 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/e085a4fba182d19139db32c664fcc1f08895ca603db3b1f9855183fc071c1adc/merged 14 shm 64M 0 64M 0% /var/lib/docker/containers/276a98c9d5ccd61f42a0a1ef55c30f76beb2977483ed2d79281dfcec79922029/shm 15 shm 64M 0 64M 0% /var/lib/docker/containers/0f89914f64c09c0fbd777e637d8f987414420861628d7b78f8a1bafc9054df59/shm 16 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/23afbc15307f6e5b397db4b1fd0e124c943bdb95312234e8d41e4f80a104b0cd/merged 17 shm 64M 0 64M 0% /var/lib/docker/containers/f686976b5743236b368b8b70e222d230b737612acadc716ca4cbabcd9a187011/shm 18 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/15696b3573a066b5d3fd7aa202086b2bcc4145d9d0bb46e54c7d8d7787d1cc73/merged 19 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/6d9b6978705654a815dd1e72d5501408cead42493c25bdd33e4faa0c9decdcd0/merged 20 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/534837088c8ba6021c80d2ebb4ddb87a075106a784c499cc9ffca541d2e86dce/merged 21 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/8cdf1249e1823f24caf1d14ca9c05caa4e8e569d81292d06c642e618bc90b82b/merged 22 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/3ee0d48c04a32a2f0938af17898697042423d7d147df3ee8c1178bfda6e5fd9a/merged 23 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/31f5db66511cc7fd15cf1e2a23b7178d0eafbf1d7ddbdef1e0f82a254b6f6fc5/merged 24 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/16288e087e06b2c1fc369d19528eeb933cde44b8aa785f35b6756ec4da3a0f60/merged 25 tmpfs 229M 0 229M 0% /run/user/0 26 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 27 [root@k8s-master ~]# df -h /mnt/ 28 Filesystem Size Used Avail Use% Mounted on 29 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 30 [root@k8s-master ~]#
分布式复制卷扩容,可以先使用df -h命令进行查看扩容前的容量,然后扩容后查看容量,在运行状态也可以进行扩容的哦,如下所示:
1 [root@k8s-master ~]# df -h /mnt/ 2 Filesystem Size Used Avail Use% Mounted on 3 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 4 [root@k8s-master ~]# gluster volume add-brick biehl k8s-node3:/gfs/test1 k8s-node3:/gfs/test2 force 5 volume add-brick: success 6 [root@k8s-master ~]# df -h /mnt/ 7 Filesystem Size Used Avail Use% Mounted on 8 192.168.110.133:/biehl 23G 15G 8.8G 62% /mnt 9 [root@k8s-master ~]#
接下来,可以向/mnt目录里面上传一些文件进行观察,如下所示:
1 [root@k8s-master ~]# cd /mnt/ 2 [root@k8s-master mnt]# rz -E 3 rz waiting to receive. 4 [root@k8s-master mnt]# ls 5 book-master6.0.war 6 [root@k8s-master mnt]# unzip book-master6.0.war
解压之后显示一堆文件,这些文件会分别在放到不同的存储单元中,首先查看master主节点的目录单元,如下所示:
1 [root@k8s-master mnt]# yum install tree 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * centos-gluster7: mirrors.tuna.tsinghua.edu.cn 9 * extras: mirrors.bfsu.edu.cn 10 * updates: mirrors.bfsu.edu.cn 11 Resolving Dependencies 12 --> Running transaction check 13 ---> Package tree.x86_64 0:1.6.0-10.el7 will be installed 14 --> Finished Dependency Resolution 15 16 Dependencies Resolved 17 18 ================================================================================================================================================================================================================= 19 Package Arch Version Repository Size 20 ================================================================================================================================================================================================================= 21 Installing: 22 tree x86_64 1.6.0-10.el7 base 46 k 23 24 Transaction Summary 25 ================================================================================================================================================================================================================= 26 Install 1 Package 27 28 Total download size: 46 k 29 Installed size: 87 k 30 Is this ok [y/d/N]: y 31 Downloading packages: 32 tree-1.6.0-10.el7.x86_64.rpm | 46 kB 00:00:05 33 Running transaction check 34 Running transaction test 35 Transaction test succeeded 36 Running transaction 37 Installing : tree-1.6.0-10.el7.x86_64 1/1 38 Verifying : tree-1.6.0-10.el7.x86_64 1/1 39 40 Installed: 41 tree.x86_64 0:1.6.0-10.el7 42 43 Complete! 44 [root@k8s-master mnt]# tree /gfs/
由于我的文件太多,就不进行展示了,使用命令tree /gfs就可以进行查看了,可以在三台机器都进行查看观察的。
4、glusterfs作为k8s的后端存储,可以使用命令查看k8s支持的后端存储类型,如下所示:
1 [root@k8s-master test1]# kubectl explain pv.spec 2 RESOURCE: spec
可以查看详细的glusterfs配置需要什么,endpoints就是svc里面的一个资源,endpoints就是(clusterIP + 端口号),path就是glusterfs卷的一个路径,readOnly只读模式,如下所示:
1 [root@k8s-master test1]# kubectl explain pv.spec.glusterfs 2 RESOURCE: glusterfs3 4 DESCRIPTION: 5 Glusterfs represents a Glusterfs volume that is attached to a host and 6 exposed to the pod. Provisioned by an admin. More info: 7 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md 8 9 Represents a Glusterfs mount that lasts the lifetime of a pod. Glusterfs volumes do not support ownership management or SELinux relabeling. 10 11 FIELDS: 12 endpoints <string> -required- 13 EndpointsName is the endpoint name that details Glusterfs topology. More 14 info: 15 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 16 17 path <string> -required- 18 Path is the Glusterfs volume path. More info: 19 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 20 21 readOnly 22 ReadOnly here will force the Glusterfs volume to be mounted with read-only 23 permissions. Defaults to false. More info: 24 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 25 26 27 [root@k8s-master test1]#
开始创建endpoints的glusterfs-ep.yaml配置文件,如下所示:
1 [root@k8s-master k8s]# ls 2 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume 3 [root@k8s-master k8s]# mkdir glusterfs 4 [root@k8s-master k8s]# cd glusterfs/ 5 [root@k8s-master glusterfs]# ls 6 [root@k8s-master glusterfs]# vim glusterfs-ep.yaml 7 [root@k8s-master glusterfs]#
具体内容,如下所示:
1 apiVersion: v1 2 kind: Endpoints 3 metadata: 4 name: glusterfs 5 namespace: default 6 subsets: 7 # k8s连接glusterfs的ip地址和端口号 8 - addresses: 9 - ip: 192.168.110.133 10 - ip: 192.168.110.134 11 - ip: 192.168.110.135 12 ports: 13 - port: 49152 14 protocol: TCP 15
每个节点启动了glusterfs就存在了49152端口号,如下所示:
1 [root@k8s-node3 test1]# netstat -lntup 49152 2 Active Internet connections (only servers) 3 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 4 tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 47623/glusterd 5 tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 14515/kubelet 6 tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 6779/kube-proxy 7 tcp 0 0 0.0.0.0:40842 0.0.0.0:* LISTEN - 8 tcp 0 0 192.168.110.135:10250 0.0.0.0:* LISTEN 14515/kubelet 9 tcp 0 0 192.168.110.135:10255 0.0.0.0:* LISTEN 14515/kubelet 10 tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd 11 tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 90097/rpc.mountd 12 tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 7122/X 13 tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 7304/dnsmasq 14 tcp 0 0 0.0.0.0:55318 0.0.0.0:* LISTEN 52362/rpc.statd 15 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6776/sshd 16 tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 6778/cupsd 17 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7094/master 18 tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 71495/glusterfsd
开始创建这个endpoint,如下所示:
1 [root@k8s-master glusterfs]# kubectl create -f glusterfs-ep.yaml 2 endpoints "glusterfs" created 3 [root@k8s-master glusterfs]#
创建成功可以查看一下,如下所示:
1 [root@k8s-master glusterfs]# kubectl get ep 2 NAME ENDPOINTS AGE 3 glusterfs 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 34s 4 kubernetes 192.168.110.133:6443 25d 5 mysql 172.16.66.6:3306 5h 6 myweb 172.16.66.5:8080 6h 7 [root@k8s-master glusterfs]# kubectl get endpoints 8 NAME ENDPOINTS AGE 9 glusterfs 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 1m 10 kubernetes 192.168.110.133:6443 25d 11 mysql 172.16.66.6:3306 5h 12 myweb 172.16.66.5:8080 6h 13 [root@k8s-master glusterfs]#
接下来,给这个endpoint创建一个Servic,需要注意的是Service和endpoint是靠名称来进行关联的,他们两个的名称必须一致的,如下所示:
1 [root@k8s-master glusterfs]# vim glusterfs-svc.yaml 2 [root@k8s-master glusterfs]#
配置内容,如下所示:
1 apiVersion: v1 2 kind: Service 3 metadata: 4 name: glusterfs 5 namespace: default 6 spec: 7 ports: 8 - port: 49152 9 protocol: TCP 10 targetPort: 49152 11 sessionAffinity: None 12 type: ClusterIP 13
开始进行创建Service,并进行查看,如下所示:
1 [root@k8s-master glusterfs]# vim glusterfs-svc.yaml 2 [root@k8s-master glusterfs]# kubectl create -f glusterfs-svc.yaml 3 service "glusterfs" created 4 [root@k8s-master glusterfs]# kubectl get svc 5 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE 6 glusterfs 10.254.182.4149152/TCP 10s 7 kubernetes 10.254.0.1 443/TCP 25d 8 mysql 10.254.126.11 3306/TCP 5h 9 myweb 10.254.188.155 8080:30008/TCP 6h 10 [root@k8s-master glusterfs]#
可以使用查看详情,看是否已经关联上了,如下所示:
1 [root@k8s-master glusterfs]# kubectl describe svc glusterfs 2 Name: glusterfs 3 Namespace: default 4 Labels:5 Selector: 6 Type: ClusterIP 7 IP: 10.254.182.41 8 Port: 49152/TCP 9 Endpoints: 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 10 Session Affinity: None 11 No events. 12 [root@k8s-master glusterfs]#
接下来,开始创建glusterfs类型的PV,k8s对接各种后端存储的时候最关键的地方就是创建PV,如下所示:
1 [root@k8s-master glusterfs]# vim glusterfs-pv.yaml 2 [root@k8s-master glusterfs]#
配置内容,如下所示:
1 apiVersion: v1 2 kind: PersistentVolume 3 metadata: 4 name: gluster 5 labels: 6 type: glusterfs 7 spec: 8 capacity: 9 storage: 50Gi 10 accessModes: 11 - ReadWriteMany 12 glusterfs: 13 endpoints: "glusterfs" 14 path: "biehl" 15 readOnly: false
创建并查看PV,如下所示:
1 [root@k8s-master glusterfs]# gluster volume list 2 biehl 3 [root@k8s-master glusterfs]# kubectl create -f glusterfs-pv.yaml 4 persistentvolume "gluster" created 5 [root@k8s-master glusterfs]# kubectl get pv 6 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 7 gluster 50Gi RWX Retain Available 9s 8 mysql 5Gi RWX Recycle Bound default/mysql 5h 9 [root@k8s-master glusterfs]#
开始创建PVC,才可以使用PV,开始创建PVC,如下所示:
1 [root@k8s-master glusterfs]# cp ../tomcat_demo/mysql-pvc.yaml . 2 [root@k8s-master glusterfs]# ls 3 glusterfs-ep.yaml glusterfs-pv.yaml glusterfs-svc.yaml mysql-pvc.yaml 4 [root@k8s-master glusterfs]# mv mysql-pvc.yaml glusterfs-pvc.yaml 5 [root@k8s-master glusterfs]# ls 6 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml 7 [root@k8s-master glusterfs]#
配置内容,如下所示:
1 kind: PersistentVolumeClaim 2 apiVersion: v1 3 metadata: 4 name: gluster 5 spec: 6 accessModes: 7 - ReadWriteMany 8 resources: 9 requests: 10 storage: 5Gi
创建并查看PVC、PV的内容,如下所示:
1 [root@k8s-master glusterfs]# kubectl create -f glusterfs-pvc.yaml 2 persistentvolumeclaim "gluster" created 3 [root@k8s-master glusterfs]# kubectl get pvc 4 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE 5 gluster Bound gluster 50Gi RWX 4s 6 mysql Bound mysql 5Gi RWX 6h 7 [root@k8s-master glusterfs]# kubectl get pv 8 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 9 gluster 50Gi RWX Retain Bound default/gluster 5m 10 mysql 5Gi RWX Recycle Bound default/mysql 6h 11 [root@k8s-master glusterfs]#
PVC绑定PV之后就可以使用了,可以对pod进行使用,如下所示:
1 [root@k8s-master glusterfs]# ls 2 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml 3 [root@k8s-master glusterfs]# cp ../pod/nginx_pod.yaml . 4 [root@k8s-master glusterfs]# ls 5 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml nginx_pod.yaml 6 [root@k8s-master glusterfs]# vim nginx_pod.yaml 7 [root@k8s-master glusterfs]#
将volumeMounts、volumes都配置好,如下所示:
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: nginx 5 labels: 6 app: web 7 env: nginx 8 spec: 9 containers: 10 - name: nginx 11 image: 192.168.110.133:5000/nginx:1.13 12 ports: 13 - containerPort: 80 14 volumeMounts: 15 - name: nfs-vol2 16 mountPath: /usr/share/nginx/html 17 volumes: 18 - name: nfs-vol2 19 persistentVolumeClaim: 20 claimName: gluster
创建nginx这个pod,并进行查看,发现已经正常运行起来了,如下所示:
1 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 2 pod "nginx" created 3 [root@k8s-master glusterfs]# kubectl get pods -o wide 4 NAME READY STATUS RESTARTS AGE IP NODE 5 mysql-wldks 1/1 Running 0 5h 172.16.66.6 k8s-node3 6 myweb-c8sf6 1/1 Running 1 7h 172.16.66.5 k8s-node3 7 nginx 1/1 Running 0 3s 172.16.66.3 k8s-node3 8 [root@k8s-master glusterfs]#
然后可以访问一下这个Pod,如下所示:
1 [root@k8s-master glusterfs]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 5h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 7h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 3s 172.16.66.3 k8s-node3 6 [root@k8s-master glusterfs]# curl 172.16.66.3
我们的容器除了SVC里面通过反向代理和端口映射的方式访问之外,还可以让Pod不经过我们的SVC进行访问,可以在Pod里面,新增一个hostPort进行访问。
将之前的nginx的pod删除掉,再重新创建一个,如下所示:
1 [root@k8s-master glusterfs]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 2h 172.16.66.3 k8s-node3 6 [root@k8s-master glusterfs]# kubectl delete pod nginx 7 pod "nginx" deleted 8 [root@k8s-master glusterfs]# kubectl get pods -o wide 9 NAME READY STATUS RESTARTS AGE IP NODE 10 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 11 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 12 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 13 pod "nginx" created 14 [root@k8s-master glusterfs]# kubectl get pods -o wide 15 NAME READY STATUS RESTARTS AGE IP NODE 16 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 17 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 18 nginx 1/1 Running 0 2s 172.16.66.3 k8s-node3 19 [root@k8s-master glusterfs]#
可以看到新创建的nginx在k8s-node3这个节点上了,但是此时我发现无论curl访问还是浏览器访问都是403错误,这不都是很郁闷了。
1 [root@k8s-master glusterfs]# curl -I 172.16.66.3 2 HTTP/1.1 403 Forbidden 3 Server: nginx/1.15.12 4 Date: Tue, 30 Jun 2020 14:03:56 GMT 5 Content-Type: text/html 6 Content-Length: 154 7 Connection: keep-alive
后来经过仔细地分析和查找,发现是确实访问不到页面才出现地错误,究其原因,主要就是/mnt目录下面要有可以直接访问地界面才可以地。
1 [root@k8s-master glusterfs]# cd /mnt/ 2 [root@k8s-master mnt]# ls 3 css img index.html js
删除之前地nginx地pod,创建新的pod,如下所示:
1 [root@k8s-master glusterfs]# kubectl get pod -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 6m 172.16.66.3 k8s-node3 6 nginx2 1/1 Running 0 17m 172.16.74.5 k8s-node2 7 [root@k8s-master glusterfs]# kubectl delete pod nginx 8 pod "nginx" deleted 9 [root@k8s-master glusterfs]# kubectl get pod -o wide 10 NAME READY STATUS RESTARTS AGE IP NODE 11 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 12 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 13 nginx2 1/1 Running 0 17m 172.16.74.5 k8s-node2 14 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 15 pod "nginx" created 16 [root@k8s-master glusterfs]# kubectl get pod -o wide 17 NAME READY STATUS RESTARTS AGE IP NODE 18 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 19 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 20 nginx 1/1 Running 0 2s 172.16.66.3 k8s-node3 21 nginx2 1/1 Running 0 18m 172.16.74.5 k8s-node2 22 [root@k8s-master glusterfs]# curl 172.16.66.3 23 24 "en"> 25 26 "UTF-8"> 27 "viewport" content="width=device-width, initial-scale=1.0"> 28 "X-UA-Compatible" content="ie=edge"> 29HTML5ȫı·ɻСԎϷ - Դë֮¼м/title> 30 31 65 66 67 68 69 70 71class="canvasbig"> 7275 76 77 80 81 [root@k8s-master glusterfs]#class="canvasdiv">73 74
访问http://192.168.110.135:8889/,在分析地过程中,我将nginx_pod.yaml地hostPort: 8889改成了8889端口号哦。
游戏是可以玩的,功能也算是搞完了。