Kerbernetes的StatefulSet与Operator

            Kerbernetes的StatefulSet与Operator

                                     作者:尹正杰

版权声明:原创作品,谢绝转载!否则将追究法律责任。 

 

 

一.StatefulSet概述

1>.StatefulSet与Operator的关系

  管理一组有状态pod服务的部署(deployment)和扩展(scaling),并保证这些pod的顺序和唯一性。
    与部署一样,StatefulSet管理基于相同容器规范的pod。
    与部署不同,statefulSet为每个pod维护一个粘性标识。这些pod是从同一个规范创建的,但不能互换:每个pod都有一个peristent标识符,可以在任何重新调度中维护。

  事实上,使用StatefulSet可以很轻松的帮咱们完成部署,但是扩展的话需要我们自己编写封装扩容集群或者缩容集群脚本代码的程序,这个程序我们统称为Operator。而且我们编写的脚本还需要进行大量测试,否则出现任意一个bug,将会在线上带来不堪设想的后果。
 
  于是,有一家公司叫CoreOS的公司对StatefulSet进行二次封装开源了Operator SDK(Software Development Kit)接口,用户可以借助于Operator SDK的API比较轻松定制出Operator控制器。

  这意味着第三方的程序员再去开发云原生应用不是针对Kubernetes云原生API,而是针对CoreOS公司的Operator SDK。换句话说,Kubernetes的大本营被人开了一道后门,从后方侵入快速拿下一半的云原生领土,而这个领土的所属方叫CoreOS。而CoreOS已经是Redhat旗下的产品,而Redhat又是IBM旗下的产品。

  在国外,微软(Microsoft),亚马逊(Amazon)已经是云计算的领头羊了,而IBM作为一个老牌的基础设施公司虽说也有自己的云计算产品但影响力却远远不如前两者公司。因为大公司通常是很难接收革新的,它们有自己成熟的产品也有充分的市场份额,有一个其它产品时竞争对手时,它们通常把竞争对手视为眼中钉肉中刺,而不是拥抱和开发它们,IBM就错失了很多良机。IBM之所以愿意买RedHat,原因之一可能是因为红帽公司旗下的CoreOS公司。

  IBM公司地位发展也给咱们运维人员一定的启发,这就好比你已经轻车熟路的使用K8S,突然间又出现了一个颠覆性的产品的苗头之后,你一定不是第一个愿意接收它的人,会有一种抵触心理,此时若很多人抱着开放的心胸去学习了该产品,恍然之间觉得自己的技能要被淘汰了不得不去学习的时候,那些先行者早已在这个行业称为巨头了。

  目前大部分应用也有自己的Operator开源项目,比较火热的应用在Github有对应的开源地址。博主推荐阅读:https://github.com/operator-framework/awesome-operators。

2>.StatefulSet的应用场景

  statefulset对于需要以下一项或多项的应用程序很有价值:
    (1)有唯一固定的网络标识符;
    (2)固定唯一的持久存储;
    (3)有序优雅的部署和扩展;
    (4)有序右眼的删除和终止;
    (5)有序自动的执行滚动更新;

  StatefulSet的限制条件(Limitations):
    (1)各Pod用到的存储卷必须使用由StorageClass动态供给给或由管理事先创建好的PV;
    (2)删除StatefulSet或缩减其规模导致Pod被删除时不应该自动删除(这个需要我们定义pvc指定其回收策略为"Retain")其存储卷以确保数据安全;
    (3)StatefulSet控制器依赖于一个事先存在的Headless Serivce对象实现Pod对象的持久,唯一的标识符配置;次Headless Service需要由用户手动配置。

3>.StatefulSet的Pod 标识符(Identity)

  有序索引(Ordinal Index):
    对于一个有N个副本的StatefulSet,StatefulSet中的每个Pod将被分配一个整数序数,从0到N-1,在这个集合上是唯一的。

  稳定网络ID(Stabel Network ID):
    StatefulSet中的每个Pod都从StatefulSet的名称和Pod的序号派生其主机名。
    StatefulSet可以使用无头服务(Headless Service)来控制其Pods的域。此服务管理的域的任务格式为:"$(service name).$(namespace).svc.cluster.local",其中“cluster.local”是群集域。
    创建每个Pod时,它将获得一个匹配的DNS子域,其格式为:"$(podname).$(governing service domain)",其中governing service由StatefulSet上的serviceName字段定义。

4>.Pod管理策略(Management Policies)

  在Kubernetes 1.7及更高版本中,StatefulSet允许您放宽其排序保证,同时通过其.spec.podManagementPolicay字段保持其唯一性和标识保证。

  有序(OrderedReady)Pod管理(Management):
    OrderedReady pod management是statefulset的默认设置。换句话说,就是当一个Pod创建成功后再回去创建下一个Pod,即有序创建Pod。
  并行的(Parallel)Pod管理(Management):     并行Pod管理告诉StatefulSet控制器并行启动或终止所有Pod,不要等到Pod运行并准备就绪或完全终止后再启动或终止另一个Pod。

5>.更新策略(Update Strategies)

  在Kubernetes 1.7及更高版本中,StatefulSet的.spec.updateStrategy字段允许您配置和禁用StatefulSet中Pods的容器、标签、资源请求/限制和注释的自动滚动更新。

  ON Delete:
    由用户手动删除现有的Pod对象从而触发其更新过程。

  Rolling Updates:
    默认策略,它通过自动更新机制完成更新过程,启动更新过程时,它自动删除每个Pod对象并以新配置进行重建,更新顺序同删除StatefulSet时的逆向操作机制,一次删除并更新一个Pod对象。

  可以通过指定.spec.update strategy.RollingUpdate.partition对RollingUpdate更新策略进行分区。

  如果指定了分区,则当StatefulSet的.pec.template被更新时,序号大于或等于分区的所有pod都将被更新。

  序号小于分区的所有pod都将不会更新,即使它们被删除,也将在以前的版本中重新创建。

 

二.StatefulSet实战案例

1>.构建NFS网络文件系统

[[email protected] ~]# yum -y install nfs-utils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.bit.edu.cn
 * extras: mirrors.tuna.tsinghua.edu.cn
 * updates: mirrors.tuna.tsinghua.edu.cn
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
Installing for dependencies:
 gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
 keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
 libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
 libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
 libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
 libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
 libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
 libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
 libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
 libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
 quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
 quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
 rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
 tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+14 Dependent packages)

Total download size: 1.3 M
Installed size: 3.6 M
Downloading packages:
(1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
(2/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
(3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
(4/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
(5/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
(6/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
(7/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
(8/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
(9/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
(10/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
(11/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
(12/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
(13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
(14/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
(15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                         2.9 MB/s | 1.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
  Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
  Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
  Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
  Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
  Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
  Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
  Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
  Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
  Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
  Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
  Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
  Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
  Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
  Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
  Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
  Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
  Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
  Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
  Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
  Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
  Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
  Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
  Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
  Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
  Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
  Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
  Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 

Installed:
  nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           

Dependency Installed:
  gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
  libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      

Complete!
[[email protected] ~]# 
[[email protected] ~]# yum -y install nfs-utils
[[email protected] ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
mkdir: created directory ‘/yinzhengjie/data’
mkdir: created directory ‘/yinzhengjie/data/volume1’
mkdir: created directory ‘/yinzhengjie/data/volume2’
mkdir: created directory ‘/yinzhengjie/data/volume3’
mkdir: created directory ‘/yinzhengjie/data/volume4’
mkdir: created directory ‘/yinzhengjie/data/volume5’
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# ll /yinzhengjie/data/ -R
/yinzhengjie/data/:
total 0
drwxr-xr-x 2 root root 6 Feb 10 06:04 volume1
drwxr-xr-x 2 root root 6 Feb 10 06:04 volume2
drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3
drwxr-xr-x 2 root root 6 Feb 10 06:04 volume4
drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5

/yinzhengjie/data/volume1:
total 0

/yinzhengjie/data/volume2:
total 0

/yinzhengjie/data/volume3:
total 0

/yinzhengjie/data/volume4:
total 0

/yinzhengjie/data/volume5:
total 0
[[email protected] ~]# 
[[email protected] ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
[[email protected] ~]# vim /etc/exports
[[email protected] ~]# 
[[email protected] ~]# cat /etc/exports
/yinzhengjie/data/volume1    172.200.0.0/21(rw,no_root_squash)
/yinzhengjie/data/volume2    172.200.0.0/21(rw,no_root_squash)
/yinzhengjie/data/volume3    172.200.0.0/21(rw,no_root_squash)
/yinzhengjie/data/volume4    172.200.0.0/21(rw,no_root_squash)
/yinzhengjie/data/volume5    172.200.0.0/21(rw,no_root_squash)
[[email protected] ~]# 
[[email protected] ~]# ll /yinzhengjie/data/
total 0
drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume2
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume3
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume4
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume5
[[email protected] ~]# 
[[email protected] ~]# exportfs -rav
exporting 172.200.0.0/21:/yinzhengjie/data/volume5
exporting 172.200.0.0/21:/yinzhengjie/data/volume4
exporting 172.200.0.0/21:/yinzhengjie/data/volume3
exporting 172.200.0.0/21:/yinzhengjie/data/volume2
exporting 172.200.0.0/21:/yinzhengjie/data/volume1
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# vim /etc/exports
[[email protected] ~]# systemctl status nfs
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[[email protected] ~]# 
[[email protected] ~]# systemctl start nfs
[[email protected] ~]# 
[[email protected] ~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[[email protected] ~]# 
[[email protected] ~]# systemctl status nfs
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Mon 2020-02-10 06:08:52 CST; 6s ago
 Main PID: 6116 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Starting NFS server and services...
Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Started NFS server and services.
[[email protected] ~]# 
[[email protected] ~]# ss -ntl | grep 2049
LISTEN     0      64           *:2049                     *:*                  
LISTEN     0      64          :::2049                    :::*                  
[[email protected] ~]# 
[[email protected] ~]# iptables -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[[email protected] ~]# 
[[email protected] ~]# systemctl start nfs

2>.在K8S集群的每一个节点安装nfs客户端驱动

[[email protected] ~]# yum -y install nfs-utils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.bit.edu.cn
 * extras: mirror.bit.edu.cn
 * updates: mirror.bit.edu.cn
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
Installing for dependencies:
 gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
 keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
 libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
 libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
 libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
 libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
 libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
 libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
 libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
 libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
 quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
 quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
 rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
 tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+14 Dependent packages)

Total download size: 1.3 M
Installed size: 3.6 M
Downloading packages:
(1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
(2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
(3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
(4/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
(5/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
(6/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
(7/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
(8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
(9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
(10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
(11/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                            |  50 kB  00:00:00     
(12/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
(13/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
(14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
(15/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                         1.4 MB/s | 1.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
  Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
  Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
  Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
  Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
  Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
  Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
  Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
  Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
  Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
  Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
  Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
  Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
  Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
  Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
  Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
  Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
  Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
  Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
  Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
  Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
  Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
  Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
  Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
  Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
  Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
  Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
  Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 

Installed:
  nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           

Dependency Installed:
  gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
  libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      

Complete!
[[email protected] ~]# 
[[email protected] ~]# yum -y install nfs-utils
[[email protected] ~]#  yum -y install nfs-utils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirror.bit.edu.cn
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
Installing for dependencies:
 gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
 keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
 libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
 libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
 libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
 libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
 libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
 libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
 libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
 libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
 quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
 quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
 rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
 tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+14 Dependent packages)

Total download size: 1.3 M
Installed size: 3.6 M
Downloading packages:
(1/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
(2/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
(3/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
(4/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
(5/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
(6/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
(7/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
(8/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
(9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
(10/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                         |  28 kB  00:00:00     
(11/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
(12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
(13/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
(14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
(15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                         2.2 MB/s | 1.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
  Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
  Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
  Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
  Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
  Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
  Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
  Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
  Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
  Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
  Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
  Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
  Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
  Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
  Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
  Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
  Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
  Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
  Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
  Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
  Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
  Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
  Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
  Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
  Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
  Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
  Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
  Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 

Installed:
  nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           

Dependency Installed:
  gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
  libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      

Complete!
[[email protected] ~]# 
[[email protected] ~]# yum -y install nfs-utils
[[email protected] ~]# yum -y install nfs-utils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.bit.edu.cn
 * extras: mirror.bit.edu.cn
 * updates: mirror.bit.edu.cn
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
Installing for dependencies:
 gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
 keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
 libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
 libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
 libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
 libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
 libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
 libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
 libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
 libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
 quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
 quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
 rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
 tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+14 Dependent packages)

Total download size: 1.3 M
Installed size: 3.6 M
Downloading packages:
(1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
(2/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
(3/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
(4/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
(5/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
(6/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
(7/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
(8/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
(9/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
(10/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
(11/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                            |  89 kB  00:00:00     
(12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
(13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
(14/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
(15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                         1.9 MB/s | 1.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
  Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
  Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
  Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
  Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
  Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
  Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
  Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
  Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
  Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
  Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
  Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
  Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
  Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
  Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
  Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
  Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
  Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
  Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
  Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
  Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
  Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
  Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
  Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
  Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
  Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
  Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
  Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 

Installed:
  nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           

Dependency Installed:
  gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
  libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      

Complete!
[[email protected] ~]# 
[[email protected] ~]# yum -y install nfs-utils
[[email protected] ~]# yum -y install nfs-utils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirror.bit.edu.cn
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
Installing for dependencies:
 gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
 keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
 libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
 libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
 libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
 libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
 libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
 libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
 libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
 libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
 quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
 quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
 rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
 tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+14 Dependent packages)

Total download size: 1.3 M
Installed size: 3.6 M
Downloading packages:
(1/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
(2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
(3/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
(4/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
(5/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
(6/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
(7/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
(8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
(9/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                               |  90 kB  00:00:00     
(10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
(11/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
(12/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                      | 8.9 kB  00:00:00     
(13/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
(14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
(15/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                         2.6 MB/s | 1.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
  Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
  Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
  Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
  Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
  Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
  Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
  Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
  Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
  Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
  Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
  Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
  Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
  Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
  Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
  Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
  Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
  Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
  Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
  Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
  Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
  Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
  Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
  Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
  Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
  Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
  Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
  Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 

Installed:
  nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           

Dependency Installed:
  gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
  libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      

Complete!
[[email protected] ~]# 
[[email protected] ~]# yum -y install nfs-utils
[[email protected] ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载
[[email protected] ~]# 
[[email protected] ~]# mount | grep mnt
172.200.1.110:/yinzhengjie/data/volume1 on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.200.1.200,local_lock=none,addr=172.200.1.110)
[[email protected] ~]# 
[[email protected] ~]# df -h | grep mnt
172.200.1.110:/yinzhengjie/data/volume1  1.6T  416M  1.6T   1% /mnt
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载,即验证可用性。

3>.创建nfs对应的pv

[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v3
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v4
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume4
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v5
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume5
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
persistentvolume/pv-nfs-v3 created
persistentvolume/pv-nfs-v4 created
persistentvolume/pv-nfs-v5 created
[[email protected] ~]# 
[[email protected] ~]# kubectl  get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   12s
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   11s
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   11s
[[email protected] ~]# 
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
[[email protected] ~]# kubectl  get pv -o wide
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE   VOLUMEMODE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   16s   Filesystem
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl get pv -o wide

4>.基于statefulset部署Pod

[[email protected] ~]# kubectl  get pv -o wide
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE   VOLUMEMODE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   16s   Filesystem
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
[[email protected] ~]# 
[[email protected] ~]# kubectl get pv -o wide
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: yinzhengjie-sts
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-sts-svc
  namespace: yinzhengjie-sts
  labels:
    app: myapp
    controller: mystatefulset
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: myapp-pod
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-demo
  namespace: yinzhengjie-sts
spec:
  selector:
    matchLabels:
      app: myapp-pod
      controller: mystatefulset
  serviceName: "myapp-sts-svc"
  replicas: 2
  template:
    metadata:
      namespace: yinzhengjie-sts
      labels:
        app: myapp-pod 
        controller: mystatefulset
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: myapp-pvc
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: myapp-pvc
      namespace: yinzhengjie-sts
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi
[[email protected] ~]# 
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
namespace/yinzhengjie-sts created
service/myapp-sts-svc created
statefulset.apps/statefulset-demo created
[[email protected] ~]# 
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml
[[email protected] ~]# kubectl get all -n yinzhengjie-sts
NAME                     READY   STATUS    RESTARTS   AGE
pod/statefulset-demo-0   1/1     Running   0          56s
pod/statefulset-demo-1   1/1     Running   0          55s

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/myapp-sts-svc   ClusterIP   None                 80/TCP    56s

NAME                                READY   AGE
statefulset.apps/statefulset-demo   2/2     56s
[[email protected] ~]# 
[[email protected] ~]# kubectl get all -n yinzhengjie-sts
[[email protected] ~]# kubectl get pvc -n yinzhengjie-sts
NAME                           STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myapp-pvc-statefulset-demo-0   Bound    pv-nfs-v5   5Gi        RWO,ROX,RWX                   73s
myapp-pvc-statefulset-demo-1   Bound    pv-nfs-v3   5Gi        RWO,ROX,RWX                   72s
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl get pvc -n yinzhengjie-sts
[[email protected] ~]# kubectl get pv -o wide
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE     VOLUMEMODE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           6m34s   Filesystem
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                                                          6m33s   Filesystem
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           6m33s   Filesystem
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl get pv -o wide
[[email protected] ~]# kubectl get svc -n yinzhengjie-sts
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
myapp-sts-svc   ClusterIP   None                 80/TCP    4m40s
[[email protected] ~]# 
[[email protected] ~]# kubectl get svc -n yinzhengjie-sts
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts -o wide
NAME                 READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
statefulset-demo-0   1/1     Running   0          5m21s   10.244.1.35   node201.yinzhengjie.org.cn              
statefulset-demo-1   1/1     Running   0          5m20s   10.244.3.25   node203.yinzhengjie.org.cn              
[[email protected] ~]# 
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts -o wide

5>.扩容Pod的副本数

[[email protected] ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           18m
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                                                          18m
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           18m
[[email protected] ~]# 
[[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts 
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
pod/statefulset-demo-0   1/1     Running   0          13m   10.244.1.35   node201.yinzhengjie.org.cn              
pod/statefulset-demo-1   1/1     Running   0          13m   10.244.3.25   node203.yinzhengjie.org.cn              

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/myapp-sts-svc   ClusterIP   None                 80/TCP    13m   app=myapp-pod

NAME                                READY   AGE   CONTAINERS   IMAGES
statefulset.apps/statefulset-demo   2/2     13m   myapp        ikubernetes/myapp:v1
[[email protected] ~]# 
[[email protected] ~]# kubectl scale sts statefulset-demo --replicas=3 -n yinzhengjie-sts
statefulset.apps/statefulset-demo scaled
[[email protected] ~]# 
[[email protected] ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           19m
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           19m
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           19m
[[email protected] ~]# 
[[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts 
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
pod/statefulset-demo-0   1/1     Running   0          15m   10.244.1.35   node201.yinzhengjie.org.cn              
pod/statefulset-demo-1   1/1     Running   0          14m   10.244.3.25   node203.yinzhengjie.org.cn              
pod/statefulset-demo-2   1/1     Running   0          14s   10.244.2.30   node202.yinzhengjie.org.cn              

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/myapp-sts-svc   ClusterIP   None                 80/TCP    15m   app=myapp-pod

NAME                                READY   AGE   CONTAINERS   IMAGES
statefulset.apps/statefulset-demo   3/3     15m   myapp        ikubernetes/myapp:v1
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl scale sts statefulset-demo --replicas=3 -n yinzhengjie-sts

Kerbernetes的StatefulSet与Operator_第1张图片

6>.升级Pod版本

[[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts 
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
pod/statefulset-demo-0   1/1     Running   0          19m     10.244.1.35   node201.yinzhengjie.org.cn              
pod/statefulset-demo-1   1/1     Running   0          19m     10.244.3.25   node203.yinzhengjie.org.cn              
pod/statefulset-demo-2   1/1     Running   0          4m29s   10.244.2.30   node202.yinzhengjie.org.cn              

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/myapp-sts-svc   ClusterIP   None                 80/TCP    19m   app=myapp-pod

NAME                                READY   AGE   CONTAINERS   IMAGES
statefulset.apps/statefulset-demo   3/3     19m   myapp        ikubernetes/myapp:v1
[[email protected] ~]# 
[[email protected] ~]# kubectl set image sts statefulset-demo myapp=ikubernetes/myapp:v2 -n yinzhengjie-sts
statefulset.apps/statefulset-demo image updated
[[email protected] ~]# 
[[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts 
NAME                     READY   STATUS    RESTARTS   AGE    IP            NODE                         NOMINATED NODE   READINESS GATES
pod/statefulset-demo-0   1/1     Running   0          69s    10.244.1.36   node201.yinzhengjie.org.cn              
pod/statefulset-demo-1   1/1     Running   0          90s    10.244.3.26   node203.yinzhengjie.org.cn              
pod/statefulset-demo-2   1/1     Running   0          103s   10.244.2.31   node202.yinzhengjie.org.cn              

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/myapp-sts-svc   ClusterIP   None                 80/TCP    22m   app=myapp-pod

NAME                                READY   AGE   CONTAINERS   IMAGES
statefulset.apps/statefulset-demo   3/3     22m   myapp        ikubernetes/myapp:v2
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl set image sts statefulset-demo myapp=ikubernetes/myapp:v2 -n yinzhengjie-sts
[[email protected] ~]# kubectl describe sts statefulset-demo -n yinzhengjie-sts
Name:               statefulset-demo
Namespace:          yinzhengjie-sts
CreationTimestamp:  Wed, 12 Feb 2020 10:40:46 +0800
Selector:           app=myapp-pod,controller=mystatefulset
Labels:             
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"statefulset-demo","namespace":"yinzhengjie-sts"},"spec":...
Replicas:           3 desired | 3 total
Update Strategy:    RollingUpdate
  Partition:        824644603528
Pods Status:        3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=myapp-pod
           controller=mystatefulset
  Containers:
   myapp:
    Image:        ikubernetes/myapp:v2
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  
    Mounts:
      /usr/share/nginx/html from myapp-pvc (rw)
  Volumes:  
Volume Claims:
  Name:          myapp-pvc
  StorageClass:  
  Labels:        
  Annotations:   
  Capacity:      2Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age                  From                    Message
  ----    ------            ----                 ----                    -------
  Normal  SuccessfulCreate  25m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-0 Pod statefulset-demo-0 in StatefulSet statefulset-demo success
  Normal  SuccessfulCreate  25m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-1 Pod statefulset-demo-1 in StatefulSet statefulset-demo success
  Normal  SuccessfulCreate  10m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-2 Pod statefulset-demo-2 in StatefulSet statefulset-demo success
  Normal  SuccessfulDelete  4m45s                statefulset-controller  delete Pod statefulset-demo-2 in StatefulSet statefulset-demo successful
  Normal  SuccessfulCreate  4m33s (x2 over 10m)  statefulset-controller  create Pod statefulset-demo-2 in StatefulSet statefulset-demo successful
  Normal  SuccessfulDelete  4m23s                statefulset-controller  delete Pod statefulset-demo-1 in StatefulSet statefulset-demo successful
  Normal  SuccessfulCreate  4m20s (x2 over 25m)  statefulset-controller  create Pod statefulset-demo-1 in StatefulSet statefulset-demo successful
  Normal  SuccessfulDelete  4m10s                statefulset-controller  delete Pod statefulset-demo-0 in StatefulSet statefulset-demo successful
  Normal  SuccessfulCreate  3m59s (x2 over 25m)  statefulset-controller  create Pod statefulset-demo-0 in StatefulSet statefulset-demo successful
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl describe sts statefulset-demo -n yinzhengjie-sts

Kerbernetes的StatefulSet与Operator_第2张图片

7>.缩容Pod的副本数(缩容时pvc对应pv数据并不会被删除)

[[email protected] ~]# kubectl get pods -n yinzhengjie-sts
NAME                 READY   STATUS    RESTARTS   AGE
statefulset-demo-0   1/1     Running   0          19m
statefulset-demo-1   1/1     Running   0          19m
statefulset-demo-2   1/1     Running   0          5s
[[email protected] ~]# 
[[email protected] ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh
/ # 
/ # cd /usr/share/nginx/html/        #进入到存储卷挂载路径
/usr/share/nginx/html # 
/usr/share/nginx/html # ls
/usr/share/nginx/html # 
/usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt      #在存储卷路径写入测试数据。
/usr/share/nginx/html # 
/usr/share/nginx/html # ls -l
total 4
-rw-r--r--    1 root     root            37 Feb 12  2020 blog.txt
/usr/share/nginx/html # 
/usr/share/nginx/html # exit
[[email protected] ~]# 
[[email protected] ~]# 
/usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt      #在存储卷路径写入测试数据。
[[email protected] ~]# 
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts
NAME                 READY   STATUS    RESTARTS   AGE
statefulset-demo-0   1/1     Running   0          19m
statefulset-demo-1   1/1     Running   0          19m
statefulset-demo-2   1/1     Running   0          5s
[[email protected] ~]# 
[[email protected] ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh
/ # 
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # 
/usr/share/nginx/html # ls
/usr/share/nginx/html # 
/usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt
/usr/share/nginx/html # 
/usr/share/nginx/html # ls -l
total 4
-rw-r--r--    1 root     root            37 Feb 12  2020 blog.txt
/usr/share/nginx/html # 
/usr/share/nginx/html # exit
[[email protected] ~]# 
[[email protected] ~]# kubectl scale sts statefulset-demo --replicas=2 -n yinzhengjie-sts
statefulset.apps/statefulset-demo scaled
[[email protected] ~]# 
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts
NAME                 READY   STATUS    RESTARTS   AGE
statefulset-demo-0   1/1     Running   0          24m
statefulset-demo-1   1/1     Running   0          24m
[[email protected] ~]# 
[[email protected] ~]# kubectl scale sts statefulset-demo --replicas=2 -n yinzhengjie-sts      #对Pod的副本进行缩容

Kerbernetes的StatefulSet与Operator_第3张图片

8>.删除statefulset

[[email protected] ~]# kubectl get sts -n yinzhengjie-sts
NAME               READY   AGE
statefulset-demo   3/3     55m
[[email protected] ~]# 
[[email protected] ~]# kubectl delete sts statefulset-demo  -n yinzhengjie-sts
statefulset.apps "statefulset-demo" deleted
[[email protected] ~]# 
[[email protected] ~]# kubectl get sts -n yinzhengjie-sts
No resources found in yinzhengjie-sts namespace.
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl delete sts statefulset-demo -n yinzhengjie-sts

 

三.使用StatefulSet部署etcd服务

  StatefulSet(简称sts)可以做到简单的部署,扩容等功能,但是要涉及到比较复杂的功能比如数据备份等功能建议大家使用Operator。

1>.创建etcd的service资源

[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
apiVersion: v1
kind: Service
metadata:
  name: etcd
  labels:
    app: etcd
  annotations:
    # Create endpoints also if the related pod isn't ready
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  ports:
  - port: 2379
    name: client
  - port: 2380
    name: peer
  clusterIP: None
  selector:
    app: etcd-member
---
apiVersion: v1
kind: Service
metadata:
  name: etcd-client
  labels:
    app: etcd
spec:
  ports:
  - name: etcd-client
    port: 2379
    protocol: TCP
    targetPort: 2379
  selector:
    app: etcd-member
  type: NodePort
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
service/etcd created
service/etcd-client created
[[email protected] ~]# 
[[email protected] ~]# kubectl get svc 
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
etcd          ClusterIP   None                     2379/TCP,2380/TCP   14s
etcd-client   NodePort    10.107.187.160           2379:32360/TCP      14s
kubernetes    ClusterIP   10.96.0.1                443/TCP             7d16h
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml
[[email protected] ~]# 
[[email protected] ~]# kubectl get svc -o wide
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE     SELECTOR
etcd          ClusterIP   None                     2379/TCP,2380/TCP   31s     app=etcd-member
etcd-client   NodePort    10.107.187.160           2379:32360/TCP      31s     app=etcd-member
kubernetes    ClusterIP   10.96.0.1                443/TCP             7d16h   
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl get svc -o wide

2>.确认nfs有足够的空间及挂载目录并创建pv

[[email protected] ~]# vim /etc/exports
[[email protected] ~]# 
[[email protected] ~]# cat /etc/exports
/yinzhengjie/data/volume1    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume2    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume3    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume4    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume5    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume6    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume7    172.200.0.0/21(rw,sync,no_root_squash)
/yinzhengjie/data/volume8    172.200.0.0/21(rw,sync,no_root_squash)
[[email protected] ~]# 
[[email protected] ~]# vim /etc/exports
[[email protected] ~]# mkdir /yinzhengjie/data/volume{6,7,8}
[[email protected] ~]# 
[[email protected] ~]# ll /yinzhengjie/data/
total 0
drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1
drwxr-xr-x 2 polkitd root  6 Feb 10 20:04 volume2
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume3
drwxr-xr-x 2 root    root 22 Feb 13 03:11 volume4
drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume5
drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume6
drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume7
drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume8
[[email protected] ~]# 
[[email protected] ~]# exportfs -rav
exporting 172.200.0.0/21:/yinzhengjie/data/volume8
exporting 172.200.0.0/21:/yinzhengjie/data/volume7
exporting 172.200.0.0/21:/yinzhengjie/data/volume6
exporting 172.200.0.0/21:/yinzhengjie/data/volume5
exporting 172.200.0.0/21:/yinzhengjie/data/volume4
exporting 172.200.0.0/21:/yinzhengjie/data/volume3
exporting 172.200.0.0/21:/yinzhengjie/data/volume2
exporting 172.200.0.0/21:/yinzhengjie/data/volume1
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# exportfs -rav
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v3
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v4
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume4
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v5
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume5
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v6
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume6
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v7
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume7
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-v8
  labels:
    storsys: nfs
spec:
  accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.200.1.110
    path: /yinzhengjie/data/volume8
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
[[email protected] ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           23m
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           23m
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           23m
[[email protected] ~]# 
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
persistentvolume/pv-nfs-v3 unchanged
persistentvolume/pv-nfs-v4 unchanged
persistentvolume/pv-nfs-v5 unchanged
persistentvolume/pv-nfs-v6 created
persistentvolume/pv-nfs-v7 created
persistentvolume/pv-nfs-v8 created
[[email protected] ~]# 
[[email protected] ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE
pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           3h52m
pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           3h52m
pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           3h52m
pv-nfs-v6   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
pv-nfs-v7   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
pv-nfs-v8   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml

3>.创建etcd的实例StatefulSet配置文件

[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-statefulset.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
  labels:
    app: etcd
spec:
  serviceName: etcd
  # changing replicas value will require a manual etcdctl member remove/add
  #   # command (remove before decreasing and add after increasing)
  replicas: 3
  selector:
    matchLabels:
      app: etcd-member
  template:
    metadata:
      name: etcd
      labels:
        app: etcd-member
    spec:
      containers:
      - name: etcd
        image: "quay.io/coreos/etcd:v3.2.16"
        ports:
        - containerPort: 2379
          name: client
        - containerPort: 2380
          name: peer
        env:
        - name: CLUSTER_SIZE
          value: "3"
        - name: SET_NAME
          value: "etcd"
        volumeMounts:
        - name: data
          mountPath: /var/run/etcd
        command:
          - "/bin/sh"
          - "-ecx"
          - |
            IP=$(hostname -i)
            PEERS=""
            for i in $(seq 0 $((${CLUSTER_SIZE} - 1))); do
                PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}:2380"
            done
            # start etcd. If cluster is already initialized the `--initial-*` options will be ignored.
            exec etcd --name ${HOSTNAME} \
              --listen-peer-urls http://${IP}:2380 \
              --listen-client-urls http://${IP}:2379,http://127.0.0.1:2379 \
              --advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 \
              --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}:2380 \
              --initial-cluster-token etcd-cluster-1 \
              --initial-cluster ${PEERS} \
              --initial-cluster-state new \
              --data-dir /var/run/etcd/default.etcd
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 1Gi
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-statefulset.yaml

 

四.使用StatefulSet部署zookeeper服务参考配置文件

[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/zk-sts.yaml 
apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk-hs
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        image: gcr.io/google-containers/kubernetes-zookeeper:1.0-3.4.10
        resources:
          requests:
            memory: "1Gi"
            cpu: "0.5"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=3 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: data
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: gluster-dynamic
      resources:
        requests:
          storage: 5Gi
[[email protected] ~]# 
[[email protected] ~]# 
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/zk-sts.yaml

 

五.基于Operator部署etcd

  博主推荐阅读:
    https://github.com/coreos/etcd-operator
    https://github.com/coreos/etcd-operator/blob/master/example/deployment.yaml
    https://github.com/coreos/etcd-operator/blob/master/example/example-etcd-cluster.yaml

Kerbernetes的StatefulSet与Operator_第4张图片

 

你可能感兴趣的:(Kerbernetes的StatefulSet与Operator)