Kerbernetes的StatefulSet与Operator
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.StatefulSet概述
1>.StatefulSet与Operator的关系
管理一组有状态pod服务的部署(deployment)和扩展(scaling)，并保证这些pod的顺序和唯一性。 与部署一样，StatefulSet管理基于相同容器规范的pod。 与部署不同，statefulSet为每个pod维护一个粘性标识。这些pod是从同一个规范创建的，但不能互换：每个pod都有一个peristent标识符，可以在任何重新调度中维护。 事实上，使用StatefulSet可以很轻松的帮咱们完成部署，但是扩展的话需要我们自己编写封装扩容集群或者缩容集群脚本代码的程序,这个程序我们统称为Operator。而且我们编写的脚本还需要进行大量测试，否则出现任意一个bug，将会在线上带来不堪设想的后果。 于是,有一家公司叫CoreOS的公司对StatefulSet进行二次封装开源了Operator SDK(Software Development Kit)接口，用户可以借助于Operator SDK的API比较轻松定制出Operator控制器。 这意味着第三方的程序员再去开发云原生应用不是针对Kubernetes云原生API,而是针对CoreOS公司的Operator SDK。换句话说，Kubernetes的大本营被人开了一道后门,从后方侵入快速拿下一半的云原生领土，而这个领土的所属方叫CoreOS。而CoreOS已经是Redhat旗下的产品，而Redhat又是IBM旗下的产品。 在国外,微软(Microsoft)，亚马逊(Amazon)已经是云计算的领头羊了，而IBM作为一个老牌的基础设施公司虽说也有自己的云计算产品但影响力却远远不如前两者公司。因为大公司通常是很难接收革新的,它们有自己成熟的产品也有充分的市场份额，有一个其它产品时竞争对手时，它们通常把竞争对手视为眼中钉肉中刺，而不是拥抱和开发它们，IBM就错失了很多良机。IBM之所以愿意买RedHat，原因之一可能是因为红帽公司旗下的CoreOS公司。 IBM公司地位发展也给咱们运维人员一定的启发，这就好比你已经轻车熟路的使用K8S，突然间又出现了一个颠覆性的产品的苗头之后,你一定不是第一个愿意接收它的人，会有一种抵触心理，此时若很多人抱着开放的心胸去学习了该产品，恍然之间觉得自己的技能要被淘汰了不得不去学习的时候，那些先行者早已在这个行业称为巨头了。 目前大部分应用也有自己的Operator开源项目，比较火热的应用在Github有对应的开源地址。博主推荐阅读:https://github.com/operator-framework/awesome-operators。
2>.StatefulSet的应用场景
statefulset对于需要以下一项或多项的应用程序很有价值: (1)有唯一固定的网络标识符; (2)固定唯一的持久存储; (3)有序优雅的部署和扩展; (4)有序右眼的删除和终止; (5)有序自动的执行滚动更新； StatefulSet的限制条件(Limitations): (1)各Pod用到的存储卷必须使用由StorageClass动态供给给或由管理事先创建好的PV; (2)删除StatefulSet或缩减其规模导致Pod被删除时不应该自动删除(这个需要我们定义pvc指定其回收策略为"Retain")其存储卷以确保数据安全; (3)StatefulSet控制器依赖于一个事先存在的Headless Serivce对象实现Pod对象的持久，唯一的标识符配置;次Headless Service需要由用户手动配置。
3>.StatefulSet的Pod 标识符(Identity)
有序索引(Ordinal Index): 对于一个有N个副本的StatefulSet，StatefulSet中的每个Pod将被分配一个整数序数，从0到N-1，在这个集合上是唯一的。 稳定网络ID(Stabel Network ID): StatefulSet中的每个Pod都从StatefulSet的名称和Pod的序号派生其主机名。 StatefulSet可以使用无头服务(Headless Service)来控制其Pods的域。此服务管理的域的任务格式为:"$(service name).$(namespace).svc.cluster.local",其中“cluster.local”是群集域。 创建每个Pod时，它将获得一个匹配的DNS子域，其格式为:"$(podname).$(governing service domain)",其中governing service由StatefulSet上的serviceName字段定义。
4>.Pod管理策略(Management Policies)
在Kubernetes 1.7及更高版本中，StatefulSet允许您放宽其排序保证，同时通过其.spec.podManagementPolicay字段保持其唯一性和标识保证。 有序(OrderedReady)Pod管理(Management): OrderedReady pod management是statefulset的默认设置。换句话说,就是当一个Pod创建成功后再回去创建下一个Pod，即有序创建Pod。
并行的(Parallel)Pod管理(Management): 并行Pod管理告诉StatefulSet控制器并行启动或终止所有Pod，不要等到Pod运行并准备就绪或完全终止后再启动或终止另一个Pod。
5>.更新策略(Update Strategies)
在Kubernetes 1.7及更高版本中，StatefulSet的.spec.updateStrategy字段允许您配置和禁用StatefulSet中Pods的容器、标签、资源请求/限制和注释的自动滚动更新。 ON Delete: 由用户手动删除现有的Pod对象从而触发其更新过程。 Rolling Updates: 默认策略,它通过自动更新机制完成更新过程，启动更新过程时,它自动删除每个Pod对象并以新配置进行重建,更新顺序同删除StatefulSet时的逆向操作机制，一次删除并更新一个Pod对象。
可以通过指定.spec.update strategy.RollingUpdate.partition对RollingUpdate更新策略进行分区。
如果指定了分区，则当StatefulSet的.pec.template被更新时，序号大于或等于分区的所有pod都将被更新。
序号小于分区的所有pod都将不会更新，即使它们被删除，也将在以前的版本中重新创建。
二.StatefulSet实战案例
1>.构建NFS网络文件系统
[[email protected] ~]# yum -y install nfs-utils Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 updates/7/x86_64/primary_db | 6.7 MB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Running transaction check ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed ---> Package quota.x86_64 1:4.01-19.el7 will be installed --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64 --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64 ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed --> Running transaction check ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64 --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64 ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed --> Running transaction check ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: nfs-utils x86_64 1:1.3.0-0.65.el7 base 412 k Installing for dependencies: gssproxy x86_64 0.7.0-26.el7 base 110 k keyutils x86_64 1.5.8-3.el7 base 54 k libbasicobjects x86_64 0.1.1-32.el7 base 26 k libcollection x86_64 0.7.0-32.el7 base 42 k libini_config x86_64 1.3.1-32.el7 base 64 k libnfsidmap x86_64 0.25-19.el7 base 50 k libpath_utils x86_64 0.2.1-32.el7 base 28 k libref_array x86_64 0.1.5-32.el7 base 27 k libtirpc x86_64 0.2.4-0.16.el7 base 89 k libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k quota x86_64 1:4.01-19.el7 base 179 k quota-nls noarch 1:4.01-19.el7 base 90 k rpcbind x86_64 0.2.0-48.el7 base 60 k tcp_wrappers x86_64 7.6-77.el7 base 78 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+14 Dependent packages) Total download size: 1.3 M Installed size: 3.6 M Downloading packages: (1/15): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00 (2/15): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00 (3/15): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00 (4/15): gssproxy-0.7.0-26.el7.x86_64.rpm | 110 kB 00:00:00 (5/15): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00 (6/15): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00 (7/15): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00 (8/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00 (9/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00 (10/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00 (11/15): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00 (12/15): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00 (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00 (14/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm | 412 kB 00:00:00 (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm | 60 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 2.9 MB/s | 1.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libbasicobjects-0.1.1-32.el7.x86_64 1/15 Installing : libref_array-0.1.5-32.el7.x86_64 2/15 Installing : libcollection-0.7.0-32.el7.x86_64 3/15 Installing : libtirpc-0.2.4-0.16.el7.x86_64 4/15 Installing : rpcbind-0.2.0-48.el7.x86_64 5/15 Installing : 1:quota-nls-4.01-19.el7.noarch 6/15 Installing : tcp_wrappers-7.6-77.el7.x86_64 7/15 Installing : 1:quota-4.01-19.el7.x86_64 8/15 Installing : keyutils-1.5.8-3.el7.x86_64 9/15 Installing : libnfsidmap-0.25-19.el7.x86_64 10/15 Installing : libpath_utils-0.2.1-32.el7.x86_64 11/15 Installing : libini_config-1.3.1-32.el7.x86_64 12/15 Installing : libverto-libevent-0.2.5-4.el7.x86_64 13/15 Installing : gssproxy-0.7.0-26.el7.x86_64 14/15 Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64 15/15 Verifying : libtirpc-0.2.4-0.16.el7.x86_64 1/15 Verifying : libverto-libevent-0.2.5-4.el7.x86_64 2/15 Verifying : 1:quota-4.01-19.el7.x86_64 3/15 Verifying : gssproxy-0.7.0-26.el7.x86_64 4/15 Verifying : libpath_utils-0.2.1-32.el7.x86_64 5/15 Verifying : libnfsidmap-0.25-19.el7.x86_64 6/15 Verifying : keyutils-1.5.8-3.el7.x86_64 7/15 Verifying : 1:nfs-utils-1.3.0-0.65.el7.x86_64 8/15 Verifying : tcp_wrappers-7.6-77.el7.x86_64 9/15 Verifying : libcollection-0.7.0-32.el7.x86_64 10/15 Verifying : libref_array-0.1.5-32.el7.x86_64 11/15 Verifying : libbasicobjects-0.1.1-32.el7.x86_64 12/15 Verifying : rpcbind-0.2.0-48.el7.x86_64 13/15 Verifying : libini_config-1.3.1-32.el7.x86_64 14/15 Verifying : 1:quota-nls-4.01-19.el7.noarch 15/15 Installed: nfs-utils.x86_64 1:1.3.0-0.65.el7 Dependency Installed: gssproxy.x86_64 0:0.7.0-26.el7 keyutils.x86_64 0:1.5.8-3.el7 libbasicobjects.x86_64 0:0.1.1-32.el7 libcollection.x86_64 0:0.7.0-32.el7 libini_config.x86_64 0:1.3.1-32.el7 libnfsidmap.x86_64 0:0.25-19.el7 libpath_utils.x86_64 0:0.2.1-32.el7 libref_array.x86_64 0:0.1.5-32.el7 libtirpc.x86_64 0:0.2.4-0.16.el7 libverto-libevent.x86_64 0:0.2.5-4.el7 quota.x86_64 1:4.01-19.el7 quota-nls.noarch 1:4.01-19.el7 rpcbind.x86_64 0:0.2.0-48.el7 tcp_wrappers.x86_64 0:7.6-77.el7 Complete! [[email protected] ~]#
[[email protected] ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5} mkdir: created directory ‘/yinzhengjie/data’ mkdir: created directory ‘/yinzhengjie/data/volume1’ mkdir: created directory ‘/yinzhengjie/data/volume2’ mkdir: created directory ‘/yinzhengjie/data/volume3’ mkdir: created directory ‘/yinzhengjie/data/volume4’ mkdir: created directory ‘/yinzhengjie/data/volume5’ [[email protected] ~]# [[email protected] ~]# [[email protected] ~]# ll /yinzhengjie/data/ -R /yinzhengjie/data/: total 0 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume1 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume2 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume4 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5 /yinzhengjie/data/volume1: total 0 /yinzhengjie/data/volume2: total 0 /yinzhengjie/data/volume3: total 0 /yinzhengjie/data/volume4: total 0 /yinzhengjie/data/volume5: total 0 [[email protected] ~]#
[[email protected] ~]# vim /etc/exports [[email protected] ~]# [[email protected] ~]# cat /etc/exports /yinzhengjie/data/volume1 172.200.0.0/21(rw,no_root_squash) /yinzhengjie/data/volume2 172.200.0.0/21(rw,no_root_squash) /yinzhengjie/data/volume3 172.200.0.0/21(rw,no_root_squash) /yinzhengjie/data/volume4 172.200.0.0/21(rw,no_root_squash) /yinzhengjie/data/volume5 172.200.0.0/21(rw,no_root_squash) [[email protected] ~]# [[email protected] ~]# ll /yinzhengjie/data/ total 0 drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume2 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume4 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5 [[email protected] ~]# [[email protected] ~]# exportfs -rav exporting 172.200.0.0/21:/yinzhengjie/data/volume5 exporting 172.200.0.0/21:/yinzhengjie/data/volume4 exporting 172.200.0.0/21:/yinzhengjie/data/volume3 exporting 172.200.0.0/21:/yinzhengjie/data/volume2 exporting 172.200.0.0/21:/yinzhengjie/data/volume1 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# systemctl status nfs ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: inactive (dead) [[email protected] ~]# [[email protected] ~]# systemctl start nfs [[email protected] ~]# [[email protected] ~]# systemctl enable nfs Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service. [[email protected] ~]# [[email protected] ~]# systemctl status nfs ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) Drop-In: /run/systemd/generator/nfs-server.service.d └─order-with-mounts.conf Active: active (exited) since Mon 2020-02-10 06:08:52 CST; 6s ago Main PID: 6116 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Starting NFS server and services... Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Started NFS server and services. [[email protected] ~]# [[email protected] ~]# ss -ntl | grep 2049 LISTEN 0 64 *:2049 *:* LISTEN 0 64 :::2049 :::* [[email protected] ~]# [[email protected] ~]# iptables -vnL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination [[email protected] ~]# [[email protected] ~]# [[email protected] ~]# iptables -vnL -t nat Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination [[email protected] ~]#
2>.在K8S集群的每一个节点安装nfs客户端驱动
[[email protected] ~]# yum -y install nfs-utils Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirror.bit.edu.cn * updates: mirror.bit.edu.cn base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 updates | 2.9 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Running transaction check ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed ---> Package quota.x86_64 1:4.01-19.el7 will be installed --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64 --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64 ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed --> Running transaction check ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64 --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64 ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed --> Running transaction check ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: nfs-utils x86_64 1:1.3.0-0.65.el7 base 412 k Installing for dependencies: gssproxy x86_64 0.7.0-26.el7 base 110 k keyutils x86_64 1.5.8-3.el7 base 54 k libbasicobjects x86_64 0.1.1-32.el7 base 26 k libcollection x86_64 0.7.0-32.el7 base 42 k libini_config x86_64 1.3.1-32.el7 base 64 k libnfsidmap x86_64 0.25-19.el7 base 50 k libpath_utils x86_64 0.2.1-32.el7 base 28 k libref_array x86_64 0.1.5-32.el7 base 27 k libtirpc x86_64 0.2.4-0.16.el7 base 89 k libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k quota x86_64 1:4.01-19.el7 base 179 k quota-nls noarch 1:4.01-19.el7 base 90 k rpcbind x86_64 0.2.0-48.el7 base 60 k tcp_wrappers x86_64 7.6-77.el7 base 78 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+14 Dependent packages) Total download size: 1.3 M Installed size: 3.6 M Downloading packages: (1/15): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00 (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm | 110 kB 00:00:00 (3/15): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00 (4/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00 (5/15): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00 (6/15): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00 (7/15): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00 (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00 (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00 (10/15): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00 (11/15): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00 (12/15): rpcbind-0.2.0-48.el7.x86_64.rpm | 60 kB 00:00:00 (13/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm | 412 kB 00:00:00 (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00 (15/15): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 1.4 MB/s | 1.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libbasicobjects-0.1.1-32.el7.x86_64 1/15 Installing : libref_array-0.1.5-32.el7.x86_64 2/15 Installing : libcollection-0.7.0-32.el7.x86_64 3/15 Installing : libtirpc-0.2.4-0.16.el7.x86_64 4/15 Installing : rpcbind-0.2.0-48.el7.x86_64 5/15 Installing : 1:quota-nls-4.01-19.el7.noarch 6/15 Installing : tcp_wrappers-7.6-77.el7.x86_64 7/15 Installing : 1:quota-4.01-19.el7.x86_64 8/15 Installing : keyutils-1.5.8-3.el7.x86_64 9/15 Installing : libnfsidmap-0.25-19.el7.x86_64 10/15 Installing : libpath_utils-0.2.1-32.el7.x86_64 11/15 Installing : libini_config-1.3.1-32.el7.x86_64 12/15 Installing : libverto-libevent-0.2.5-4.el7.x86_64 13/15 Installing : gssproxy-0.7.0-26.el7.x86_64 14/15 Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64 15/15 Verifying : libtirpc-0.2.4-0.16.el7.x86_64 1/15 Verifying : libverto-libevent-0.2.5-4.el7.x86_64 2/15 Verifying : 1:quota-4.01-19.el7.x86_64 3/15 Verifying : gssproxy-0.7.0-26.el7.x86_64 4/15 Verifying : libpath_utils-0.2.1-32.el7.x86_64 5/15 Verifying : libnfsidmap-0.25-19.el7.x86_64 6/15 Verifying : keyutils-1.5.8-3.el7.x86_64 7/15 Verifying : 1:nfs-utils-1.3.0-0.65.el7.x86_64 8/15 Verifying : tcp_wrappers-7.6-77.el7.x86_64 9/15 Verifying : libcollection-0.7.0-32.el7.x86_64 10/15 Verifying : libref_array-0.1.5-32.el7.x86_64 11/15 Verifying : libbasicobjects-0.1.1-32.el7.x86_64 12/15 Verifying : rpcbind-0.2.0-48.el7.x86_64 13/15 Verifying : libini_config-1.3.1-32.el7.x86_64 14/15 Verifying : 1:quota-nls-4.01-19.el7.noarch 15/15 Installed: nfs-utils.x86_64 1:1.3.0-0.65.el7 Dependency Installed: gssproxy.x86_64 0:0.7.0-26.el7 keyutils.x86_64 0:1.5.8-3.el7 libbasicobjects.x86_64 0:0.1.1-32.el7 libcollection.x86_64 0:0.7.0-32.el7 libini_config.x86_64 0:1.3.1-32.el7 libnfsidmap.x86_64 0:0.25-19.el7 libpath_utils.x86_64 0:0.2.1-32.el7 libref_array.x86_64 0:0.1.5-32.el7 libtirpc.x86_64 0:0.2.4-0.16.el7 libverto-libevent.x86_64 0:0.2.5-4.el7 quota.x86_64 1:4.01-19.el7 quota-nls.noarch 1:4.01-19.el7 rpcbind.x86_64 0:0.2.0-48.el7 tcp_wrappers.x86_64 0:7.6-77.el7 Complete! [[email protected] ~]#
[[email protected] ~]# yum -y install nfs-utils Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirror.bit.edu.cn base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 updates | 2.9 kB 00:00:00 updates/7/x86_64/primary_db | 6.7 MB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Running transaction check ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed ---> Package quota.x86_64 1:4.01-19.el7 will be installed --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64 --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64 ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed --> Running transaction check ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64 --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64 ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed --> Running transaction check ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: nfs-utils x86_64 1:1.3.0-0.65.el7 base 412 k Installing for dependencies: gssproxy x86_64 0.7.0-26.el7 base 110 k keyutils x86_64 1.5.8-3.el7 base 54 k libbasicobjects x86_64 0.1.1-32.el7 base 26 k libcollection x86_64 0.7.0-32.el7 base 42 k libini_config x86_64 1.3.1-32.el7 base 64 k libnfsidmap x86_64 0.25-19.el7 base 50 k libpath_utils x86_64 0.2.1-32.el7 base 28 k libref_array x86_64 0.1.5-32.el7 base 27 k libtirpc x86_64 0.2.4-0.16.el7 base 89 k libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k quota x86_64 1:4.01-19.el7 base 179 k quota-nls noarch 1:4.01-19.el7 base 90 k rpcbind x86_64 0.2.0-48.el7 base 60 k tcp_wrappers x86_64 7.6-77.el7 base 78 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+14 Dependent packages) Total download size: 1.3 M Installed size: 3.6 M Downloading packages: (1/15): gssproxy-0.7.0-26.el7.x86_64.rpm | 110 kB 00:00:00 (2/15): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00 (3/15): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00 (4/15): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00 (5/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00 (6/15): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00 (7/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00 (8/15): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00 (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00 (10/15): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00 (11/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm | 412 kB 00:00:00 (12/15): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00 (13/15): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00 (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00 (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm | 60 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 2.2 MB/s | 1.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libbasicobjects-0.1.1-32.el7.x86_64 1/15 Installing : libref_array-0.1.5-32.el7.x86_64 2/15 Installing : libcollection-0.7.0-32.el7.x86_64 3/15 Installing : libtirpc-0.2.4-0.16.el7.x86_64 4/15 Installing : rpcbind-0.2.0-48.el7.x86_64 5/15 Installing : 1:quota-nls-4.01-19.el7.noarch 6/15 Installing : tcp_wrappers-7.6-77.el7.x86_64 7/15 Installing : 1:quota-4.01-19.el7.x86_64 8/15 Installing : keyutils-1.5.8-3.el7.x86_64 9/15 Installing : libnfsidmap-0.25-19.el7.x86_64 10/15 Installing : libpath_utils-0.2.1-32.el7.x86_64 11/15 Installing : libini_config-1.3.1-32.el7.x86_64 12/15 Installing : libverto-libevent-0.2.5-4.el7.x86_64 13/15 Installing : gssproxy-0.7.0-26.el7.x86_64 14/15 Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64 15/15 Verifying : libtirpc-0.2.4-0.16.el7.x86_64 1/15 Verifying : libverto-libevent-0.2.5-4.el7.x86_64 2/15 Verifying : 1:quota-4.01-19.el7.x86_64 3/15 Verifying : gssproxy-0.7.0-26.el7.x86_64 4/15 Verifying : libpath_utils-0.2.1-32.el7.x86_64 5/15 Verifying : libnfsidmap-0.25-19.el7.x86_64 6/15 Verifying : keyutils-1.5.8-3.el7.x86_64 7/15 Verifying : 1:nfs-utils-1.3.0-0.65.el7.x86_64 8/15 Verifying : tcp_wrappers-7.6-77.el7.x86_64 9/15 Verifying : libcollection-0.7.0-32.el7.x86_64 10/15 Verifying : libref_array-0.1.5-32.el7.x86_64 11/15 Verifying : libbasicobjects-0.1.1-32.el7.x86_64 12/15 Verifying : rpcbind-0.2.0-48.el7.x86_64 13/15 Verifying : libini_config-1.3.1-32.el7.x86_64 14/15 Verifying : 1:quota-nls-4.01-19.el7.noarch 15/15 Installed: nfs-utils.x86_64 1:1.3.0-0.65.el7 Dependency Installed: gssproxy.x86_64 0:0.7.0-26.el7 keyutils.x86_64 0:1.5.8-3.el7 libbasicobjects.x86_64 0:0.1.1-32.el7 libcollection.x86_64 0:0.7.0-32.el7 libini_config.x86_64 0:1.3.1-32.el7 libnfsidmap.x86_64 0:0.25-19.el7 libpath_utils.x86_64 0:0.2.1-32.el7 libref_array.x86_64 0:0.1.5-32.el7 libtirpc.x86_64 0:0.2.4-0.16.el7 libverto-libevent.x86_64 0:0.2.5-4.el7 quota.x86_64 1:4.01-19.el7 quota-nls.noarch 1:4.01-19.el7 rpcbind.x86_64 0:0.2.0-48.el7 tcp_wrappers.x86_64 0:7.6-77.el7 Complete! [[email protected] ~]#
[[email protected] ~]# yum -y install nfs-utils Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirror.bit.edu.cn * updates: mirror.bit.edu.cn base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 updates | 2.9 kB 00:00:00 updates/7/x86_64/primary_db | 6.7 MB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Running transaction check ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed ---> Package quota.x86_64 1:4.01-19.el7 will be installed --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64 --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64 ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed --> Running transaction check ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64 --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64 ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed --> Running transaction check ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: nfs-utils x86_64 1:1.3.0-0.65.el7 base 412 k Installing for dependencies: gssproxy x86_64 0.7.0-26.el7 base 110 k keyutils x86_64 1.5.8-3.el7 base 54 k libbasicobjects x86_64 0.1.1-32.el7 base 26 k libcollection x86_64 0.7.0-32.el7 base 42 k libini_config x86_64 1.3.1-32.el7 base 64 k libnfsidmap x86_64 0.25-19.el7 base 50 k libpath_utils x86_64 0.2.1-32.el7 base 28 k libref_array x86_64 0.1.5-32.el7 base 27 k libtirpc x86_64 0.2.4-0.16.el7 base 89 k libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k quota x86_64 1:4.01-19.el7 base 179 k quota-nls noarch 1:4.01-19.el7 base 90 k rpcbind x86_64 0.2.0-48.el7 base 60 k tcp_wrappers x86_64 7.6-77.el7 base 78 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+14 Dependent packages) Total download size: 1.3 M Installed size: 3.6 M Downloading packages: (1/15): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00 (2/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00 (3/15): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00 (4/15): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00 (5/15): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00 (6/15): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00 (7/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00 (8/15): gssproxy-0.7.0-26.el7.x86_64.rpm | 110 kB 00:00:00 (9/15): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00 (10/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm | 412 kB 00:00:00 (11/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00 (12/15): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00 (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00 (14/15): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00 (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm | 60 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 1.9 MB/s | 1.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libbasicobjects-0.1.1-32.el7.x86_64 1/15 Installing : libref_array-0.1.5-32.el7.x86_64 2/15 Installing : libcollection-0.7.0-32.el7.x86_64 3/15 Installing : libtirpc-0.2.4-0.16.el7.x86_64 4/15 Installing : rpcbind-0.2.0-48.el7.x86_64 5/15 Installing : 1:quota-nls-4.01-19.el7.noarch 6/15 Installing : tcp_wrappers-7.6-77.el7.x86_64 7/15 Installing : 1:quota-4.01-19.el7.x86_64 8/15 Installing : keyutils-1.5.8-3.el7.x86_64 9/15 Installing : libnfsidmap-0.25-19.el7.x86_64 10/15 Installing : libpath_utils-0.2.1-32.el7.x86_64 11/15 Installing : libini_config-1.3.1-32.el7.x86_64 12/15 Installing : libverto-libevent-0.2.5-4.el7.x86_64 13/15 Installing : gssproxy-0.7.0-26.el7.x86_64 14/15 Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64 15/15 Verifying : libtirpc-0.2.4-0.16.el7.x86_64 1/15 Verifying : libverto-libevent-0.2.5-4.el7.x86_64 2/15 Verifying : 1:quota-4.01-19.el7.x86_64 3/15 Verifying : gssproxy-0.7.0-26.el7.x86_64 4/15 Verifying : libpath_utils-0.2.1-32.el7.x86_64 5/15 Verifying : libnfsidmap-0.25-19.el7.x86_64 6/15 Verifying : keyutils-1.5.8-3.el7.x86_64 7/15 Verifying : 1:nfs-utils-1.3.0-0.65.el7.x86_64 8/15 Verifying : tcp_wrappers-7.6-77.el7.x86_64 9/15 Verifying : libcollection-0.7.0-32.el7.x86_64 10/15 Verifying : libref_array-0.1.5-32.el7.x86_64 11/15 Verifying : libbasicobjects-0.1.1-32.el7.x86_64 12/15 Verifying : rpcbind-0.2.0-48.el7.x86_64 13/15 Verifying : libini_config-1.3.1-32.el7.x86_64 14/15 Verifying : 1:quota-nls-4.01-19.el7.noarch 15/15 Installed: nfs-utils.x86_64 1:1.3.0-0.65.el7 Dependency Installed: gssproxy.x86_64 0:0.7.0-26.el7 keyutils.x86_64 0:1.5.8-3.el7 libbasicobjects.x86_64 0:0.1.1-32.el7 libcollection.x86_64 0:0.7.0-32.el7 libini_config.x86_64 0:1.3.1-32.el7 libnfsidmap.x86_64 0:0.25-19.el7 libpath_utils.x86_64 0:0.2.1-32.el7 libref_array.x86_64 0:0.1.5-32.el7 libtirpc.x86_64 0:0.2.4-0.16.el7 libverto-libevent.x86_64 0:0.2.5-4.el7 quota.x86_64 1:4.01-19.el7 quota-nls.noarch 1:4.01-19.el7 rpcbind.x86_64 0:0.2.0-48.el7 tcp_wrappers.x86_64 0:7.6-77.el7 Complete! [[email protected] ~]#
[[email protected] ~]# yum -y install nfs-utils Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirror.bit.edu.cn base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 updates | 2.9 kB 00:00:00 updates/7/x86_64/primary_db | 6.7 MB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64 --> Running transaction check ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64 ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed ---> Package quota.x86_64 1:4.01-19.el7 will be installed --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64 --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64 ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed --> Running transaction check ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64 --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64 ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed --> Running transaction check ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: nfs-utils x86_64 1:1.3.0-0.65.el7 base 412 k Installing for dependencies: gssproxy x86_64 0.7.0-26.el7 base 110 k keyutils x86_64 1.5.8-3.el7 base 54 k libbasicobjects x86_64 0.1.1-32.el7 base 26 k libcollection x86_64 0.7.0-32.el7 base 42 k libini_config x86_64 1.3.1-32.el7 base 64 k libnfsidmap x86_64 0.25-19.el7 base 50 k libpath_utils x86_64 0.2.1-32.el7 base 28 k libref_array x86_64 0.1.5-32.el7 base 27 k libtirpc x86_64 0.2.4-0.16.el7 base 89 k libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k quota x86_64 1:4.01-19.el7 base 179 k quota-nls noarch 1:4.01-19.el7 base 90 k rpcbind x86_64 0.2.0-48.el7 base 60 k tcp_wrappers x86_64 7.6-77.el7 base 78 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+14 Dependent packages) Total download size: 1.3 M Installed size: 3.6 M Downloading packages: (1/15): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00 (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm | 110 kB 00:00:00 (3/15): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00 (4/15): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00 (5/15): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00 (6/15): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00 (7/15): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00 (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00 (9/15): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00 (10/15): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00 (11/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00 (12/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00 (13/15): rpcbind-0.2.0-48.el7.x86_64.rpm | 60 kB 00:00:00 (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00 (15/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm | 412 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 2.6 MB/s | 1.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libbasicobjects-0.1.1-32.el7.x86_64 1/15 Installing : libref_array-0.1.5-32.el7.x86_64 2/15 Installing : libcollection-0.7.0-32.el7.x86_64 3/15 Installing : libtirpc-0.2.4-0.16.el7.x86_64 4/15 Installing : rpcbind-0.2.0-48.el7.x86_64 5/15 Installing : 1:quota-nls-4.01-19.el7.noarch 6/15 Installing : tcp_wrappers-7.6-77.el7.x86_64 7/15 Installing : 1:quota-4.01-19.el7.x86_64 8/15 Installing : keyutils-1.5.8-3.el7.x86_64 9/15 Installing : libnfsidmap-0.25-19.el7.x86_64 10/15 Installing : libpath_utils-0.2.1-32.el7.x86_64 11/15 Installing : libini_config-1.3.1-32.el7.x86_64 12/15 Installing : libverto-libevent-0.2.5-4.el7.x86_64 13/15 Installing : gssproxy-0.7.0-26.el7.x86_64 14/15 Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64 15/15 Verifying : libtirpc-0.2.4-0.16.el7.x86_64 1/15 Verifying : libverto-libevent-0.2.5-4.el7.x86_64 2/15 Verifying : 1:quota-4.01-19.el7.x86_64 3/15 Verifying : gssproxy-0.7.0-26.el7.x86_64 4/15 Verifying : libpath_utils-0.2.1-32.el7.x86_64 5/15 Verifying : libnfsidmap-0.25-19.el7.x86_64 6/15 Verifying : keyutils-1.5.8-3.el7.x86_64 7/15 Verifying : 1:nfs-utils-1.3.0-0.65.el7.x86_64 8/15 Verifying : tcp_wrappers-7.6-77.el7.x86_64 9/15 Verifying : libcollection-0.7.0-32.el7.x86_64 10/15 Verifying : libref_array-0.1.5-32.el7.x86_64 11/15 Verifying : libbasicobjects-0.1.1-32.el7.x86_64 12/15 Verifying : rpcbind-0.2.0-48.el7.x86_64 13/15 Verifying : libini_config-1.3.1-32.el7.x86_64 14/15 Verifying : 1:quota-nls-4.01-19.el7.noarch 15/15 Installed: nfs-utils.x86_64 1:1.3.0-0.65.el7 Dependency Installed: gssproxy.x86_64 0:0.7.0-26.el7 keyutils.x86_64 0:1.5.8-3.el7 libbasicobjects.x86_64 0:0.1.1-32.el7 libcollection.x86_64 0:0.7.0-32.el7 libini_config.x86_64 0:1.3.1-32.el7 libnfsidmap.x86_64 0:0.25-19.el7 libpath_utils.x86_64 0:0.2.1-32.el7 libref_array.x86_64 0:0.1.5-32.el7 libtirpc.x86_64 0:0.2.4-0.16.el7 libverto-libevent.x86_64 0:0.2.5-4.el7 quota.x86_64 1:4.01-19.el7 quota-nls.noarch 1:4.01-19.el7 rpcbind.x86_64 0:0.2.0-48.el7 tcp_wrappers.x86_64 0:7.6-77.el7 Complete! [[email protected] ~]#
[[email protected] ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt #在K8S node节点测试NFS服务是否可以成功挂载 [[email protected] ~]# [[email protected] ~]# mount | grep mnt 172.200.1.110:/yinzhengjie/data/volume1 on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.200.1.200,local_lock=none,addr=172.200.1.110) [[email protected] ~]# [[email protected] ~]# df -h | grep mnt 172.200.1.110:/yinzhengjie/data/volume1 1.6T 416M 1.6T 1% /mnt [[email protected] ~]# [[email protected] ~]#
3>.创建nfs对应的pv
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml [[email protected] ~]# [[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v3 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume3 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v4 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume4 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v5 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume5 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml persistentvolume/pv-nfs-v3 created persistentvolume/pv-nfs-v4 created persistentvolume/pv-nfs-v5 created [[email protected] ~]# [[email protected] ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Available 12s pv-nfs-v4 5Gi RWO,ROX,RWX Retain Available 11s pv-nfs-v5 5Gi RWO,ROX,RWX Retain Available 11s [[email protected] ~]#
[[email protected] ~]# kubectl get pv -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Available 16s Filesystem pv-nfs-v4 5Gi RWO,ROX,RWX Retain Available 15s Filesystem pv-nfs-v5 5Gi RWO,ROX,RWX Retain Available 15s Filesystem [[email protected] ~]# [[email protected] ~]#
4>.基于statefulset部署Pod
[[email protected] ~]# kubectl get pv -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Available 16s Filesystem pv-nfs-v4 5Gi RWO,ROX,RWX Retain Available 15s Filesystem pv-nfs-v5 5Gi RWO,ROX,RWX Retain Available 15s Filesystem [[email protected] ~]#
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml [[email protected] ~]# [[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml apiVersion: v1 kind: Namespace metadata: name: yinzhengjie-sts --- apiVersion: v1 kind: Service metadata: name: myapp-sts-svc namespace: yinzhengjie-sts labels: app: myapp controller: mystatefulset spec: ports: - port: 80 name: web clusterIP: None selector: app: myapp-pod --- apiVersion: apps/v1 kind: StatefulSet metadata: name: statefulset-demo namespace: yinzhengjie-sts spec: selector: matchLabels: app: myapp-pod controller: mystatefulset serviceName: "myapp-sts-svc" replicas: 2 template: metadata: namespace: yinzhengjie-sts labels: app: myapp-pod controller: mystatefulset spec: terminationGracePeriodSeconds: 10 containers: - name: myapp image: ikubernetes/myapp:v1 ports: - containerPort: 80 name: web volumeMounts: - name: myapp-pvc mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: myapp-pvc namespace: yinzhengjie-sts spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 2Gi [[email protected] ~]#
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml namespace/yinzhengjie-sts created service/myapp-sts-svc created statefulset.apps/statefulset-demo created [[email protected] ~]#
[[email protected] ~]# kubectl get all -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE pod/statefulset-demo-0 1/1 Running 0 56s pod/statefulset-demo-1 1/1 Running 0 55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/myapp-sts-svc ClusterIP None80/TCP 56s NAME READY AGE statefulset.apps/statefulset-demo 2/2 56s [[email protected] ~]#
[[email protected] ~]# kubectl get pvc -n yinzhengjie-sts NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myapp-pvc-statefulset-demo-0 Bound pv-nfs-v5 5Gi RWO,ROX,RWX 73s myapp-pvc-statefulset-demo-1 Bound pv-nfs-v3 5Gi RWO,ROX,RWX 72s [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl get pv -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-1 6m34s Filesystem pv-nfs-v4 5Gi RWO,ROX,RWX Retain Available 6m33s Filesystem pv-nfs-v5 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-0 6m33s Filesystem [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl get svc -n yinzhengjie-sts NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE myapp-sts-svc ClusterIP None80/TCP 4m40s [[email protected] ~]#
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES statefulset-demo-0 1/1 Running 0 5m21s 10.244.1.35 node201.yinzhengjie.org.cnstatefulset-demo-1 1/1 Running 0 5m20s 10.244.3.25 node203.yinzhengjie.org.cn [[email protected] ~]#
5>.扩容Pod的副本数
[[email protected] ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-1 18m pv-nfs-v4 5Gi RWO,ROX,RWX Retain Available 18m pv-nfs-v5 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-0 18m [[email protected] ~]# [[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/statefulset-demo-0 1/1 Running 0 13m 10.244.1.35 node201.yinzhengjie.org.cnpod/statefulset-demo-1 1/1 Running 0 13m 10.244.3.25 node203.yinzhengjie.org.cn NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/myapp-sts-svc ClusterIP None 80/TCP 13m app=myapp-pod NAME READY AGE CONTAINERS IMAGES statefulset.apps/statefulset-demo 2/2 13m myapp ikubernetes/myapp:v1 [[email protected] ~]# [[email protected] ~]# kubectl scale sts statefulset-demo --replicas=3 -n yinzhengjie-sts statefulset.apps/statefulset-demo scaled [[email protected] ~]# [[email protected] ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-1 19m pv-nfs-v4 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-2 19m pv-nfs-v5 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-0 19m [[email protected] ~]# [[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/statefulset-demo-0 1/1 Running 0 15m 10.244.1.35 node201.yinzhengjie.org.cn pod/statefulset-demo-1 1/1 Running 0 14m 10.244.3.25 node203.yinzhengjie.org.cn pod/statefulset-demo-2 1/1 Running 0 14s 10.244.2.30 node202.yinzhengjie.org.cn NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/myapp-sts-svc ClusterIP None 80/TCP 15m app=myapp-pod NAME READY AGE CONTAINERS IMAGES statefulset.apps/statefulset-demo 3/3 15m myapp ikubernetes/myapp:v1 [[email protected] ~]# [[email protected] ~]#
6>.升级Pod版本
[[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/statefulset-demo-0 1/1 Running 0 19m 10.244.1.35 node201.yinzhengjie.org.cnpod/statefulset-demo-1 1/1 Running 0 19m 10.244.3.25 node203.yinzhengjie.org.cn pod/statefulset-demo-2 1/1 Running 0 4m29s 10.244.2.30 node202.yinzhengjie.org.cn NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/myapp-sts-svc ClusterIP None 80/TCP 19m app=myapp-pod NAME READY AGE CONTAINERS IMAGES statefulset.apps/statefulset-demo 3/3 19m myapp ikubernetes/myapp:v1 [[email protected] ~]# [[email protected] ~]# kubectl set image sts statefulset-demo myapp=ikubernetes/myapp:v2 -n yinzhengjie-sts statefulset.apps/statefulset-demo image updated [[email protected] ~]# [[email protected] ~]# kubectl get all -o wide -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/statefulset-demo-0 1/1 Running 0 69s 10.244.1.36 node201.yinzhengjie.org.cn pod/statefulset-demo-1 1/1 Running 0 90s 10.244.3.26 node203.yinzhengjie.org.cn pod/statefulset-demo-2 1/1 Running 0 103s 10.244.2.31 node202.yinzhengjie.org.cn NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/myapp-sts-svc ClusterIP None 80/TCP 22m app=myapp-pod NAME READY AGE CONTAINERS IMAGES statefulset.apps/statefulset-demo 3/3 22m myapp ikubernetes/myapp:v2 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl describe sts statefulset-demo -n yinzhengjie-sts Name: statefulset-demo Namespace: yinzhengjie-sts CreationTimestamp: Wed, 12 Feb 2020 10:40:46 +0800 Selector: app=myapp-pod,controller=mystatefulset Labels:Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"statefulset-demo","namespace":"yinzhengjie-sts"},"spec":... Replicas: 3 desired | 3 total Update Strategy: RollingUpdate Partition: 824644603528 Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=myapp-pod controller=mystatefulset Containers: myapp: Image: ikubernetes/myapp:v2 Port: 80/TCP Host Port: 0/TCP Environment: Mounts: /usr/share/nginx/html from myapp-pvc (rw) Volumes: Volume Claims: Name: myapp-pvc StorageClass: Labels: Annotations: Capacity: 2Gi Access Modes: [ReadWriteOnce] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 25m statefulset-controller create Claim myapp-pvc-statefulset-demo-0 Pod statefulset-demo-0 in StatefulSet statefulset-demo success Normal SuccessfulCreate 25m statefulset-controller create Claim myapp-pvc-statefulset-demo-1 Pod statefulset-demo-1 in StatefulSet statefulset-demo success Normal SuccessfulCreate 10m statefulset-controller create Claim myapp-pvc-statefulset-demo-2 Pod statefulset-demo-2 in StatefulSet statefulset-demo success Normal SuccessfulDelete 4m45s statefulset-controller delete Pod statefulset-demo-2 in StatefulSet statefulset-demo successful Normal SuccessfulCreate 4m33s (x2 over 10m) statefulset-controller create Pod statefulset-demo-2 in StatefulSet statefulset-demo successful Normal SuccessfulDelete 4m23s statefulset-controller delete Pod statefulset-demo-1 in StatefulSet statefulset-demo successful Normal SuccessfulCreate 4m20s (x2 over 25m) statefulset-controller create Pod statefulset-demo-1 in StatefulSet statefulset-demo successful Normal SuccessfulDelete 4m10s statefulset-controller delete Pod statefulset-demo-0 in StatefulSet statefulset-demo successful Normal SuccessfulCreate 3m59s (x2 over 25m) statefulset-controller create Pod statefulset-demo-0 in StatefulSet statefulset-demo successful [[email protected] ~]# [[email protected] ~]#
7>.缩容Pod的副本数(缩容时pvc对应pv数据并不会被删除)
[[email protected] ~]# kubectl get pods -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE statefulset-demo-0 1/1 Running 0 19m statefulset-demo-1 1/1 Running 0 19m statefulset-demo-2 1/1 Running 0 5s [[email protected] ~]# [[email protected] ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh / # / # cd /usr/share/nginx/html/ #进入到存储卷挂载路径 /usr/share/nginx/html # /usr/share/nginx/html # ls /usr/share/nginx/html # /usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt #在存储卷路径写入测试数据。 /usr/share/nginx/html # /usr/share/nginx/html # ls -l total 4 -rw-r--r-- 1 root root 37 Feb 12 2020 blog.txt /usr/share/nginx/html # /usr/share/nginx/html # exit [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# [[email protected] ~]# kubectl get pods -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE statefulset-demo-0 1/1 Running 0 19m statefulset-demo-1 1/1 Running 0 19m statefulset-demo-2 1/1 Running 0 5s [[email protected] ~]# [[email protected] ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh / # / # cd /usr/share/nginx/html/ /usr/share/nginx/html # /usr/share/nginx/html # ls /usr/share/nginx/html # /usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt /usr/share/nginx/html # /usr/share/nginx/html # ls -l total 4 -rw-r--r-- 1 root root 37 Feb 12 2020 blog.txt /usr/share/nginx/html # /usr/share/nginx/html # exit [[email protected] ~]# [[email protected] ~]# kubectl scale sts statefulset-demo --replicas=2 -n yinzhengjie-sts statefulset.apps/statefulset-demo scaled [[email protected] ~]# [[email protected] ~]# kubectl get pods -n yinzhengjie-sts NAME READY STATUS RESTARTS AGE statefulset-demo-0 1/1 Running 0 24m statefulset-demo-1 1/1 Running 0 24m [[email protected] ~]#
8>.删除statefulset
[[email protected] ~]# kubectl get sts -n yinzhengjie-sts NAME READY AGE statefulset-demo 3/3 55m [[email protected] ~]# [[email protected] ~]# kubectl delete sts statefulset-demo -n yinzhengjie-sts statefulset.apps "statefulset-demo" deleted [[email protected] ~]# [[email protected] ~]# kubectl get sts -n yinzhengjie-sts No resources found in yinzhengjie-sts namespace. [[email protected] ~]# [[email protected] ~]#
三.使用StatefulSet部署etcd服务
StatefulSet(简称sts)可以做到简单的部署，扩容等功能，但是要涉及到比较复杂的功能比如数据备份等功能建议大家使用Operator。
1>.创建etcd的service资源
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml [[email protected] ~]# [[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml apiVersion: v1 kind: Service metadata: name: etcd labels: app: etcd annotations: # Create endpoints also if the related pod isn't ready service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" spec: ports: - port: 2379 name: client - port: 2380 name: peer clusterIP: None selector: app: etcd-member --- apiVersion: v1 kind: Service metadata: name: etcd-client labels: app: etcd spec: ports: - name: etcd-client port: 2379 protocol: TCP targetPort: 2379 selector: app: etcd-member type: NodePort [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml service/etcd created service/etcd-client created [[email protected] ~]# [[email protected] ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etcd ClusterIP None2379/TCP,2380/TCP 14s etcd-client NodePort 10.107.187.160 2379:32360/TCP 14s kubernetes ClusterIP 10.96.0.1 443/TCP 7d16h [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# [[email protected] ~]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR etcd ClusterIP None2379/TCP,2380/TCP 31s app=etcd-member etcd-client NodePort 10.107.187.160 2379:32360/TCP 31s app=etcd-member kubernetes ClusterIP 10.96.0.1 443/TCP 7d16h [[email protected] ~]# [[email protected] ~]#
2>.确认nfs有足够的空间及挂载目录并创建pv
[[email protected] ~]# vim /etc/exports [[email protected] ~]# [[email protected] ~]# cat /etc/exports /yinzhengjie/data/volume1 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume2 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume3 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume4 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume5 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume6 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume7 172.200.0.0/21(rw,sync,no_root_squash) /yinzhengjie/data/volume8 172.200.0.0/21(rw,sync,no_root_squash) [[email protected] ~]#
[[email protected] ~]# mkdir /yinzhengjie/data/volume{6,7,8} [[email protected] ~]# [[email protected] ~]# ll /yinzhengjie/data/ total 0 drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1 drwxr-xr-x 2 polkitd root 6 Feb 10 20:04 volume2 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3 drwxr-xr-x 2 root root 22 Feb 13 03:11 volume4 drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5 drwxr-xr-x 2 root root 6 Feb 13 03:42 volume6 drwxr-xr-x 2 root root 6 Feb 13 03:42 volume7 drwxr-xr-x 2 root root 6 Feb 13 03:42 volume8 [[email protected] ~]# [[email protected] ~]# exportfs -rav exporting 172.200.0.0/21:/yinzhengjie/data/volume8 exporting 172.200.0.0/21:/yinzhengjie/data/volume7 exporting 172.200.0.0/21:/yinzhengjie/data/volume6 exporting 172.200.0.0/21:/yinzhengjie/data/volume5 exporting 172.200.0.0/21:/yinzhengjie/data/volume4 exporting 172.200.0.0/21:/yinzhengjie/data/volume3 exporting 172.200.0.0/21:/yinzhengjie/data/volume2 exporting 172.200.0.0/21:/yinzhengjie/data/volume1 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml [[email protected] ~]# [[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v3 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume3 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v4 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume4 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v5 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume5 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v6 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume6 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v7 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume7 --- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs-v8 labels: storsys: nfs spec: accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"] capacity: storage: 5Gi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain nfs: server: 172.200.1.110 path: /yinzhengjie/data/volume8 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-1 23m pv-nfs-v4 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-2 23m pv-nfs-v5 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-0 23m [[email protected] ~]# [[email protected] ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml persistentvolume/pv-nfs-v3 unchanged persistentvolume/pv-nfs-v4 unchanged persistentvolume/pv-nfs-v5 unchanged persistentvolume/pv-nfs-v6 created persistentvolume/pv-nfs-v7 created persistentvolume/pv-nfs-v8 created [[email protected] ~]# [[email protected] ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs-v3 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-1 3h52m pv-nfs-v4 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-2 3h52m pv-nfs-v5 5Gi RWO,ROX,RWX Retain Bound yinzhengjie-sts/myapp-pvc-statefulset-demo-0 3h52m pv-nfs-v6 5Gi RWO,ROX,RWX Retain Available 11s pv-nfs-v7 5Gi RWO,ROX,RWX Retain Available 11s pv-nfs-v8 5Gi RWO,ROX,RWX Retain Available 11s [[email protected] ~]# [[email protected] ~]#
3>.创建etcd的实例StatefulSet配置文件
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: etcd labels: app: etcd spec: serviceName: etcd # changing replicas value will require a manual etcdctl member remove/add # # command (remove before decreasing and add after increasing) replicas: 3 selector: matchLabels: app: etcd-member template: metadata: name: etcd labels: app: etcd-member spec: containers: - name: etcd image: "quay.io/coreos/etcd:v3.2.16" ports: - containerPort: 2379 name: client - containerPort: 2380 name: peer env: - name: CLUSTER_SIZE value: "3" - name: SET_NAME value: "etcd" volumeMounts: - name: data mountPath: /var/run/etcd command: - "/bin/sh" - "-ecx" - | IP=$(hostname -i) PEERS="" for i in $(seq 0 $((${CLUSTER_SIZE} - 1))); do PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}:2380" done # start etcd. If cluster is already initialized the `--initial-*` options will be ignored. exec etcd --name ${HOSTNAME} \ --listen-peer-urls http://${IP}:2380 \ --listen-client-urls http://${IP}:2379,http://127.0.0.1:2379 \ --advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 \ --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}:2380 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster ${PEERS} \ --initial-cluster-state new \ --data-dir /var/run/etcd/default.etcd volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" resources: requests: storage: 1Gi [[email protected] ~]# [[email protected] ~]#
四.使用StatefulSet部署zookeeper服务参考配置文件
[[email protected] ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/zk-sts.yaml apiVersion: v1 kind: Service metadata: name: zk-hs labels: app: zk spec: ports: - port: 2888 name: server - port: 3888 name: leader-election clusterIP: None selector: app: zk --- apiVersion: v1 kind: Service metadata: name: zk-cs labels: app: zk spec: ports: - port: 2181 name: client selector: app: zk --- apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: zk-pdb spec: selector: matchLabels: app: zk maxUnavailable: 1 --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zk spec: selector: matchLabels: app: zk serviceName: zk-hs replicas: 3 updateStrategy: type: RollingUpdate podManagementPolicy: Parallel template: metadata: labels: app: zk spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - zk-hs topologyKey: "kubernetes.io/hostname" containers: - name: kubernetes-zookeeper image: gcr.io/google-containers/kubernetes-zookeeper:1.0-3.4.10 resources: requests: memory: "1Gi" cpu: "0.5" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election command: - sh - -c - "start-zookeeper \ --servers=3 \ --data_dir=/var/lib/zookeeper/data \ --data_log_dir=/var/lib/zookeeper/data/log \ --conf_dir=/opt/zookeeper/conf \ --client_port=2181 \ --election_port=3888 \ --server_port=2888 \ --tick_time=2000 \ --init_limit=10 \ --sync_limit=5 \ --heap=512M \ --max_client_cnxns=60 \ --snap_retain_count=3 \ --purge_interval=12 \ --max_session_timeout=40000 \ --min_session_timeout=4000 \ --log_level=INFO" readinessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 livenessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 volumeMounts: - name: data mountPath: /var/lib/zookeeper securityContext: runAsUser: 1000 fsGroup: 1000 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: gluster-dynamic resources: requests: storage: 5Gi [[email protected] ~]# [[email protected] ~]#
五.基于Operator部署etcd
博主推荐阅读: https://github.com/coreos/etcd-operator https://github.com/coreos/etcd-operator/blob/master/example/deployment.yaml https://github.com/coreos/etcd-operator/blob/master/example/example-etcd-cluster.yaml
