Chiron在K8S中的实践
K8S容器编排
文章目录
-
- 1:k8s集群的安装
-
- 1.1 k8s的架构
- 1.2：修改所有节点IP地址、主机名和host解析
- 1.3：master节点安装etcd
- 1.4：master节点安装kubernetes
- 1.5：所有node节点安装kubernetes
- 6：所有节点配置flannel网络
- 7：配置master为镜像仓库
- 2:什么是k8s,k8s有什么功能?
-
- 2.1 k8s的核心功能
- 2.2 k8s的历史
- 2.3 k8s的安装方式
- 2.4 k8s的应用场景
- 3:k8s常用的资源
-
- 3.1 创建pod资源
- 3.2 ReplicationController资源
- 3.3 service资源
- 3.4 deployment资源
- 3.5 Daemon Set资源
- 3.6 Pet Set资源
- 3.7 Job资源
- 3.8 tomcat+mysql练习
- 4:k8s的附加组件
-
- 4.1 dns服务
- 4.2 namespace命令空间
- 4.3 健康检查
-
- 4.3.1 探针的种类
- 4.3.2 探针的检测方法
- 4.3.3 liveness探针的exec使用：
- 4.3.4 liveness探针的httpGet使用：
- 4.3.5 liveness探针的tcpSocket使用：
- 4.3.6 readiness探针的httpGet使用：
- 4.4 dashboard服务
- 4.5 通过apiservicer反向代理访问service
- 5: k8s弹性伸缩
-
- 5.1 安装heapster监控
- 5.2 弹性伸缩
- 6:持久化存储
-
- 数据持久化类型:
- 6.1 emptyDir：
- 6.2 HostPath：
- 6.3 nfs:
-
- Kubernetes使用NFS共享存储有两种方式：
- 6.4 pvc:
-
- 6.4.1:创建pv和pvc
- 6.4.2:创建mysql-rc,pod模板里使用volume：PVC
- 6.4.3: 验证持久化
- 6.5: 分布式存储glusterfs
- 6.6 k8s 对接glusterfs存储
-
-
- 扩展(不做演示)：
-
- 7:与jenkins集成实现cd
-
- 7. 1 :将本地代码上传到码云Gitee的仓库中
- 7.2 :安装jenkins,并自动构建docker镜像
-
- 1:安装jenkins
- 2:访问jenkins
- 3:编写Dockerfile，进行本地测试，然后推送到Git仓库中
- 4:配置jenkins拉取Git仓库代码凭据
- 5:Jenkins创建自由风格的工程“yiliao”；拉取代码测试
- 6:更改Jenkins配置中执行的shell命令；使其创建Docker镜像，并推送到私有仓库
- 7:Jenkins使用参数化构建,自动构建docker镜像并上传到私有仓库
- 7.3 手动部署实现“yiliao”的发布，升级，回滚；方面理解后面通过 Jenkins自动化部署应用
- 7.4 jenkins自动部署应用到k8s
- 8:部署Prometheus+Grafana
-
- Node节点安装cadvisor和node-exporter
- Master节点部署Prometheus+Grafana
- 回滚到指定版本
- Jenkins使用参数化构建回滚到指定版本；执行Shell语句为：（参数化构建回滚到指定版本）
- 构建工程“yiliao_undo”；不填写构建参数使其回滚到上一版本
- Web浏览器访问验证：标题为“ChironW-2医疗”
- 再次构建工程“yiliao_undo”；填写构建参数指定回滚到“REVISION=3”版本
- Web浏览器访问验证：标题为“ChironW医疗”
-
-
- Master节点部署Prometheus+Grafana
-
1:k8s集群的安装
1.1 k8s的架构
除了核心组件，还有一些推荐的Add-ons：
| 组件名称 | 说明 |
|---|---|
| kube-dns | 负责为整个集群提供DNS服务 |
| Ingress Controller | 为服务提供外网入口 |
| Heapster | 提供资源监控 |
| Dashboard | 提供GUI |
| Federation | 提供跨可用区的集群 |
| Fluentd-elasticsearch | 提供集群日志采集、存储与查询 |
1.2：修改所有节点IP地址、主机名和host解析
# 修改所有节点主机名
[root@chiron ~]# hostnamectl set-hostname k8s-master
[root@chiron ~]# hostnamectl set-hostname k8s-node-1
[root@chiron ~]# hostnamectl set-hostname k8s-node-2
# 所有节点需要做hosts解析
[root@chiron ~]# cat <>/etc/hosts
10.0.0.11 k8s-master
10.0.0.12 k8s-node-1
10.0.0.13 k8s-node-2
EOF
1.3：master节点安装etcd
# yum方式安装Etcd
[root@k8s-master ~]# yum install etcd -y
# 备份Etcd主配置文件
[root@k8s-master ~]# cp /etc/etcd/etcd.conf{,.bak}
# 修改Etcd主配置文件
[root@k8s-master ~]# vim /etc/etcd/etcd.conf
6行：ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
21行：ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.11:2379"
或者
[root@k8s-master ~]# grep -Ev '#|^$' /etc/etcd/etcd.conf
[root@k8s-master ~]# sed -i 's#ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"#ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"#g' /etc/etcd/etcd.conf
[root@k8s-master ~]# sed -i 's#ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"#ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.11:2379"#g' /etc/etcd/etcd.conf
[root@k8s-master ~]# grep -Ev '#|^$' /etc/etcd/etcd.conf
# 启动Etcd服务，并设置开机自启动
[root@k8s-master ~]# systemctl start etcd.service && systemctl enable etcd.service
# 创建一个Key并查看
[root@k8s-master ~]# etcdctl set testdir/testkey0 0 && etcdctl get testdir/testkey0
# 检查Etcd节点是否健康
[root@k8s-master ~]# etcdctl -C http://10.0.0.11:2379 cluster-health
etcd原生支持做集群，
作业1：安装部署etcd集群，要求三个节点
1.4：master节点安装kubernetes
# Master节点Yum方式安装K8S
[root@k8s-master ~]# yum install kubernetes-master.x86_64 -y
# 备份apiserver配置文件
[root@k8s-master ~]# cp /etc/kubernetes/apiserver{,.bak}
# 修改apiserver配置文件
[root@k8s-master ~]# vim /etc/kubernetes/apiserver
8行: KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
11行：KUBE_API_PORT="--port=8080"
14行: KUBELET_PORT="--kubelet-port=10250"
17行：KUBE_ETCD_SERVERS="--etcd-servers=http://10.0.0.11:2379"
23行：KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
或者
[root@k8s-master ~]# grep -Ev '#|^$' /etc/kubernetes/apiserver
[root@k8s-master ~]# sed -i 's#KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1"#KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"#g' /etc/kubernetes/apiserver
[root@k8s-master ~]# sed -i 's/^# KUBE_API_PORT="--port=8080"/KUBE_API_PORT="--port=8080/g' /etc/kubernetes/apiserver
[root@k8s-master ~]# sed -i 's/^# KUBELET_PORT="--kubelet-port=10250"/KUBELET_PORT="--kubelet-port=10250"/g' /etc/kubernetes/apiserver
[root@k8s-master ~]# sed -i 's#KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"#KUBE_ETCD_SERVERS="--etcd-servers=http://10.0.0.11:2379"#g' /etc/kubernetes/apiserver
[root@k8s-master ~]# sed -i 's#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"#g' /etc/kubernetes/apiserver
[root@k8s-master ~]# grep -Ev '#|^$' /etc/kubernetes/apiserver
# 备份config配置文件
[root@k8s-master ~]# cp /etc/kubernetes/config{,.bak}
# 修改config配置文件
[root@k8s-master ~]# vim /etc/kubernetes/config
22行：KUBE_MASTER="--master=http://10.0.0.11:8080"
或者
[root@k8s-master ~]# grep -Ev '#|^$' /etc/kubernetes/config
[root@k8s-master ~]# sed -i 's#KUBE_MASTER="--master=http://127.0.0.1:8080"#KUBE_MASTER="--master=http://10.0.0.11:8080"#g' /etc/kubernetes/config
[root@k8s-master ~]# grep -Ev '#|^$' /etc/kubernetes/config
# 启动 apiserver、controller-manager、scheduler 服务并加入系统服务
[root@k8s-master ~]# systemctl enable kube-apiserver.service kube-controller-manager.service kube-scheduler.service
[root@k8s-master ~]# systemctl restart kube-apiserver.service kube-controller-manager.service kube-scheduler.service
[root@k8s-master ~]# systemctl status kube-apiserver.service kube-controller-manager.service kube-scheduler.service
检查服务是否安装正常
# 检查集群组件间运行状态是否正常
[root@k8s-master ~]# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
etcd-0 Healthy {"health":"true"}
scheduler Healthy ok
controller-manager Healthy ok
1.5：所有node节点安装kubernetes
# 所有Node节点Yum方式安装K8S
[root@k8s-node ~]# yum install kubernetes-node.x86_64 -y
# 备份config配置文件
[root@k8s-node ~]# cp /etc/kubernetes/config{,.bak}
# 修改config配置文件
[root@k8s-node ~]# vim /etc/kubernetes/config
22行：KUBE_MASTER="--master=http://10.0.0.11:8080"
或者
[root@k8s-node ~]# grep -Ev '^$|#' /etc/kubernetes/config
[root@k8s-node ~]# sed -i 's#KUBE_MASTER="--master=http://127.0.0.1:8080"#KUBE_MASTER="--master=http://10.0.0.11:8080"#g' /etc/kubernetes/config
[root@k8s-node ~]# grep -Ev '^$|#' /etc/kubernetes/config
# 备份kubelet配置文件
[root@k8s-node ~]# cp /etc/kubernetes/kubelet{,.bak}
# 修改kubelet配置文件
[root@k8s-node ~]# vim /etc/kubernetes/kubelet
5行：KUBELET_ADDRESS="--address=0.0.0.0"
8行：KUBELET_PORT="--port=10250"
11行：KUBELET_HOSTNAME="--hostname-override=10.0.0.12"
# 注意：此处的"hostname-override=10.0.0.x",要根据当前node节点的IP或主机名填写
14行：KUBELET_API_SERVER="--api-servers=http://10.0.0.11:8080"
或者
[root@k8s-node ~]# grep -Ev '^$|#' /etc/kubernetes/kubelet
[root@k8s-node ~]# sed -i 's#KUBELET_ADDRESS="--address=127.0.0.1"#KUBELET_ADDRESS="--address=0.0.0.0"#g' /etc/kubernetes/kubelet
[root@k8s-node ~]# sed -i 's/^# KUBELET_PORT="--port=10250"/KUBELET_PORT="--port=10250"/g' /etc/kubernetes/kubelet
[root@k8s-node ~]# sed -i 's#KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080"#KUBELET_API_SERVER="--api-servers=http://10.0.0.11:8080"#g' /etc/kubernetes/kubelet
# 注意此处"hostname-override=10.0.0.x",要根据当前node节点的IP或主机名填写
[root@k8s-node-1 ~]# sed -i 's#KUBELET_HOSTNAME="--hostname-override=127.0.0.1"#KUBELET_HOSTNAME="--hostname-override=k8s-node-1#g' /etc/kubernetes/kubelet
[root@k8s-node-2 ~]# sed -i 's#KUBELET_HOSTNAME="--hostname-override=127.0.0.1"#KUBELET_HOSTNAME="--hostname-override=k8s-node-2"#g' /etc/kubernetes/kubelet
[root@k8s-node ~]# grep -Ev '^$|#' /etc/kubernetes/kubelet
# 启动 kubelet.service、kube-proxy.service 服务并加入系统服务
[root@k8s-node ~]# systemctl enable kubelet.service kube-proxy.service
[root@k8s-node ~]# systemctl restart kubelet.service kube-proxy.service
[root@k8s-node ~]# systemctl status kubelet.service kube-proxy.service
在master节点检查Nodes节点状态是否正常
[root@k8s-master ~]# kubectl get nodes
NAME STATUS AGE
k8s-node-1 Ready 13s
k8s-node-2 Ready 16s
6：所有节点配置flannel网络
yum install flannel -y
sed -i 's#http://127.0.0.1:2379#http://10.0.0.11:2379#g' /etc/sysconfig/flanneld
##master节点：
# 向Etcd中创建一个值，为flannel网络指定网段；否则 falannel 网络插件服务启动失败
[root@k8s-master ~]# etcdctl mk /atomic.io/network/config '{ "Network": "172.18.0.0/16" }'
# 为Master主机安装Docker,方便后面将Master主机作为私有镜像仓库“registry”
[root@k8s-master ~]# yum install docker -y
# 将flanneld.service docker服务设置开机自启动，并重启master主机所有K8S相关服务
[root@k8s-master ~]# systemctl enable flanneld.service docker && systemctl restart flanneld.service docker
[root@k8s-master ~]# systemctl restart kube-apiserver.service kube-controller-manager.service kube-scheduler.service
[root@k8s-master ~]# systemctl status kube-apiserver.service kube-controller-manager.service kube-scheduler.service flanneld.service docker
##所有Node节点：
# 将flanneld.service docker服务启动；并添加到系统开机自启
systemctl enable flanneld.service && systemctl restart flanneld.service
# 查看Node主机"docker0"网卡IP网段；默认为"172.17.0.1/16"网段
[root@k8s-node-1 ~]# ip addr |grep docker0
inet 172.17.0.1/16 scope global docker0
# 重启Docker等Node主机所有K8S相关服务;重启Docker目的在于查看"docker0"网卡网段的变化
systemctl restart docker kubelet.service kube-proxy.service
systemctl status flanneld.service ocker kubelet.service kube-proxy.service
# 再次查看Node主机"docker0"网卡IP网段；已经变为flannel指定网段
[root@k8s-node-1 ~]# ip addr |grep docker0
inet 172.18.37.1/24 scope global docker0
##验证两台Node间Pod是否能够ping通
# 在两台Node主机中下载"busybox"镜像，各自启动一个Pod
docker pull busybox
# 在Node1中启动一个Pod，查看它的IP地址
[root@k8s-node-1 ~]# docker run -it docker.io/busybox:latest
/ # ip addr |grep eth0
inet 172.18.37.2/24 scope global eth0
/ #
# 在Node2启的Pod中ping Node1的Pod地址；发现网络 ping 不通，而 ping 其的网管确是正常的？
[root@k8s-node-2 ~]# docker run -it docker.io/busybox:latest
/ # ping 172.18.37.2
PING 172.18.37.2 (172.18.37.2): 56 data bytes
--- 172.18.37.2 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
/ # ping 172.18.37.1
PING 172.18.37.1 (172.18.37.1): 56 data bytes
64 bytes from 172.18.37.1: seq=0 ttl=61 time=1.700 ms
# 这是因为当Docker服务启动时，会拉起 iptables 防火墙规则，其中"DROP"了"FORWARD"规则链；需要修改"docker.service"文件，使其更改为"ACCEPT"；需要在所有节点(包括Master）操作。
# 备份"docker.service"文件
cp /usr/lib/systemd/system/docker.service{,.bak}
# 修改"docker.service"文件
vim /usr/lib/systemd/system/docker.service
#在[Service]区域下增加一行
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
或者
sed -i 'N;12a\ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT' /usr/lib/systemd/system/docker.service
# 参数解释：
# 在某行（指具体行号）前或后加一行内容
# sed -i 'N;4a\ddpdf' a.txt
# sed -i 'N;4i\eepdf' a.txt
# N;4a 代表在第四行之后加入"ddpdf"；即"a"代表在之后添加一行数据
# N;4i 代表在第四行之后加入"eepdf"；即"i"代表在之前添加一行数据
# 修改之后，需要重新加载"docker.service"文件，并进行重启
systemctl daemon-reload && systemctl restart docker
# 再次在Node2中Pod，ping Node中的Pod，网络正常
[root@k8s-node-2 ~]# docker attach 576a192fcc4b
/ # ping 172.18.37.2
PING 172.18.37.2 (172.18.37.2): 56 data bytes
64 bytes from 172.18.37.2: seq=0 ttl=60 time=1.549 ms
7：配置master为镜像仓库
# 因为我们不是使用HTTPS方式的访问；所以在"/etc/docker/daemon.json"文件中添加"insecure-registries"的意义是信任指定的私有仓库；否则会报如下错误：
# The push refers to a repository [10.0.0.11:5000/centos:6.10]
# Get https://10.0.0.11:5000/v1/_ping: http: server gave HTTP response to HTTPS client
# 所有节点
vi /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com"],
"insecure-registries": ["10.0.0.11:5000"]
}
或者
[root@k8s-master ~]# cat >/etc/docker/daemon.json<
> {
> "registry-mirrors": ["https://registry.docker-cn.com"],
> "insecure-registries": ["10.0.0.11:5000"]
> }
> EOF
systemctl restart docker
# master节点部署registry私有镜像仓库(不带registry认证)
docker run -d -p 5000:5000 --restart=always --name registry -v /opt/myregistry:/var/lib/registry registry
# master节点部署registry私有镜像仓库(带registry认证)；此处不建议使用，否则后面"kubectl create"创建容器时，会pull不到镜像，除非事先pull到Nodes节点中；
# 比较推荐带认证的私有仓库为：Harbor仓库，且拥有Web管控台。
yum install httpd-tools -y
mkdir /opt/registry-var/auth/ -p
htpasswd -Bbn chiron 123456 >> /opt/registry-var/auth/htpasswd
docker run -d -p 5000:5000 --restart=always --name registry -v /opt/registry-var/auth/:/auth/ -v /opt/myregistry:/var/lib/registry -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd registry
# 提示：上传下载镜像都需要登陆哦
docker login 10.0.0.11:5000
chiron
123456
docker logout 10.0.0.11:5000登出
# 镜像推送与拉取验证：
# 向私有仓库"registry"上传本地镜像tar包；存放到"/opt/images"目录下
[root@k8s-node-1 ~]# mkdir /opt/images && cd /opt/images
[root@k8s-node-1 ~]# wget http://192.168.2.254:8888/docker_nginx1.13.tar.gz
[root@k8s-node-1 ~]# docker load -i docker_nginx1.13.tar.gz
[root@k8s-node-1 ~]# docker tag docker.io/nginx:1.13 10.0.0.11:5000/nginx:1.13
[root@k8s-node-1 ~]# docker images
[root@k8s-node-1 ~]# docker push 10.0.0.11:5000/nginx:1.13
# 前边我们pull过一个"busybox:latest"的镜像，这里一并打个标签上传到"registry"私有仓库，作为后续使用
[root@k8s-node-1 ~]# docker tag docker.io/busybox:latest 10.0.0.11:5000/busybox:latest
[root@k8s-node-1 ~]# docker push 10.0.0.11:5000/busybox:latest
# Registry私有镜像仓库查看镜像是否存在
[root@k8s-master nginx]# ls /opt/myregistry/docker/registry/v2/repositories/nginx/_manifests/tags/
# Web页面访问验证URL地址：
http://10.0.0.11:5000/v2/nginx/tags/list
# 拉取镜像
[root@k8s-node-2 ~]# docker images
[root@k8s-node-2 ~]# docker pull 10.0.0.11:5000/nginx:1.13
[root@k8s-node-2 ~]# docker images
2:什么是k8s,k8s有什么功能?
k8s是一个docker集群的管理工具
k8s是容器的编排工具
2.1 k8s的核心功能
自愈: 重新启动失败的容器，在节点不可用时，替换和重新调度节点上的容器，对用户定义的健康检查不响应的容器会被中止，并且在容器准备好服务之前不会把其向客户端广播。
弹性伸缩: 通过监控容器的cpu的负载值,如果这个平均高于80%,增加容器的数量,如果这个平均低于10%,减少容器的数量
服务的自动发现和负载均衡: 不需要修改您的应用程序来使用不熟悉的服务发现机制，Kubernetes 为容器提供了自己的 IP 地址和一组容器的单个 DNS 名称，并可以在它们之间进行负载均衡。
滚动升级和一键回滚: Kubernetes 逐渐部署对应用程序或其配置的更改，同时监视应用程序运行状况，以确保它不会同时终止所有实例。 如果出现问题，Kubernetes会为您恢复更改，利用日益增长的部署解决方案的生态系统。
私密配置文件管理. web容器里面,数据库的账户密码
2.2 k8s的历史
2014年 docker容器编排工具，立项
2015年7月 发布kubernetes 1.0, 加入cncf基金会 孵化
2016年，kubernetes干掉两个对手，docker swarm，mesos marathon 1.2版
2017年 1.5 -1.9
2018年 k8s 从cncf基金会 毕业项目
2019年： 1.13, 1.14 ，1.15,1.16
cncf :cloud native compute foundation
kubernetes （k8s）: 希腊语 舵手，领航 容器编排领域，
谷歌15年容器使用经验，borg容器管理平台，使用golang重构borg，kubernetes
2.3 k8s的安装方式
yum安装 1.5 最容易安装成功，最适合学习的
源码编译安装—难度最大 可以安装最新版
二进制安装—步骤繁琐 可以安装最新版 shell,ansible,saltstack
kubeadm 安装最容易, 网络 可以安装最新版
minikube 适合开发人员体验k8s, 网络
2.4 k8s的应用场景
k8s最适合跑微服务项目!
微服务的好处：
提供更高的并发，可用性更强，发布周期更短
微服务的缺点：
管理难度大，自动化代码上线，Ansible，saltstack；
docker，docker管理平台K8S，弹性伸缩
拓展：
MVC：软件开发架构，同一域名，不同功能，放在不同目录下
微服务：软件开发架构，不同域名，不同功能，放在独立的站点
3:k8s常用的资源
3.1 创建pod资源
pod是最小资源单位.
任何的一个k8s资源都可以由yml清单文件来定义
pod资源:至少由两个容器组成,pod基础容器和业务容器组成(最多1+4)
yaml文件声明，至少启动一个容器，创建一个Pod资源
Kubectl调用docker启动了两个容器
nginx:13 业务容器；提供访问
Pod 容器 实现K8S的高级功能
k8s中yaml的主要组成：
apiVersion: v1 api版本
kind: pod 资源类型
metadata: 属性
spec: 详细
k8s_pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: web
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
master节点创建后面存放K8S各种资源Yaml文件的存放路径
[root@k8s-master ~]# mkdir -p /root/k8s_yaml/pod && cd /root/k8s_yaml/pod/
[root@k8s-master pod]# vim k8s_pod.yaml
[root@k8s-master pod]# kubectl create -f k8s_pod.yaml
[root@k8s-master pod]# kubectl get pod
# 注意：此处查看到的容器状态非“Running”
[root@k8s-master pod]# kubectl describe pod nginx
# 注意：此处查看到的Pod详情下方会报错如下：
# Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request. details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory)"
# 解决方法见下方修改Node节点“/etc/kubernetes/kubelet”文件(此处是为了踩坑)
# 解决完上方踩坑问题，删除旧的pod“nginx”，重新创建一次，则运行正常
[root@k8s-master pod]# kubectl delete pods nginx
[root@k8s-master pod]# kubectl create -f k8s_pod.yaml
[root@k8s-master pod]# kubectl get pod
# 可以查看到Pod更多信息
[root@k8s-master pod]# kubectl get pod -o wide
[root@k8s-master pod]# kubectl describe pod nginx
# 查看到pod的IP后，CURL访问验证是否能访问到nginx首页
[root@k8s-master pod]# curl 172.18.95.2
# 查看为什么启动了两个容器，存在什么不同与关联
[root@k8s-node-2 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fd8681f40db0 10.0.0.11:5000/nginx:1.13 "nginx -g 'daemon ..." 5 minutes ago Up 5 minutes k8s_nginx.91390390_nginx_default_c69aed1e-1012-11ea-bb8d-000c29c6d194_48670ad4
b9aff18feced 10.0.0.11:5000/rhel7/pod-infrastructure:latest "/pod" 5 minutes ago Up 5 minutes k8s_POD.e5ea03c1_nginx_default_c69aed1e-1012-11ea-bb8d-000c29c6d194_20a42fd7
# 查看业务容器“nginx:1.13”的详细信息
[root@k8s-node-2 ~]# docker inspect fd8681f40db0 |tail -20
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {}
}
}
]
# 查看基础容器的信息
[root@k8s-node-2 ~]# docker inspect b9aff18feced |tail -20
"MacAddress": "02:42:ac:12:5f:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "d8db8b8692bd9f06bf1e041c869191cd304d9337bcfad041b7354650359203ba",
"EndpointID": "568288f7bb2bbba16ce0d4ca903b7f6174d74a620380bcf2b46146f1210b0fe9",
"Gateway": "172.18.95.1",
"IPAddress": "172.18.95.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:5f:02"
}
}
}
}
]
# 此处看到 nginxPod 的网卡使用的基础容器：b9aff18feced 的网络资源
[root@k8s-node-2 ~]# docker inspect fd8681f40db0 |grep -i network
"NetworkMode": "container:b9aff18feced757e44b5b4d5e58686523777d4779349c63da7d9cd654731137a",
"NetworkSettings": {
"Networks": {}
node节点:
# 向私有仓库"registry"上传本地镜像tar包；存放到"/opt/images"目录下
# 导入本地上传基础镜像
[root@k8s-node-1 images]# wget http://192.168.2.254:8888/pod-infrastructure-latest.tar.gz
[root@k8s-node-1 images]# docker load -i pod-infrastructure-latest.tar.gz
[root@k8s-node-1 images]# docker tag docker.io/tianyebj/pod-infrastructure:latest 10.0.0.11:5000/rhel7/pod-infrastructure:latest
[root@k8s-node-1 images]# docker push 10.0.0.11:5000/rhel7/pod-infrastructure:latest
# 所有Node节点修改配置文件；使其启动Pod时，从本地私有仓库"registry"拉取基础镜像
vim /etc/kubernetes/kubelet
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.0.0.11:5000/rhel7/pod-infrastructure:latest"
或者
grep -Ev '^$|#' /etc/kubernetes/kubelet
sed -i 's#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.0.0.11:5000/rhel7/pod-infrastructure:latest"#g' /etc/kubernetes/kubelet
grep -Ev '^$|#' /etc/kubernetes/kubelet
# 重启kubelet服务，使配置生效
systemctl restart kubelet.service
pod配置文件2：k8s_pod2.yaml；目的是演示让一个基础Pod，带起两个业务容器(共同使用基础Pod的网络地址)
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
app: web
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
- name: busybox
image: 10.0.0.11:5000/busybox:latest
command: ["sleep","10000"]
# Master创建一个基础Pod，两个业务Pod的示例
[root@k8s-master pod]# kubectl create -f k8s_pod2.yaml
[root@k8s-master pod]# kubectl get pod -o wide
[root@k8s-master pod]# kubectl describe pod test
# Node节点查看容器；进行验证
[root@k8s-node-1 images]# docker ps -a
pod是k8s最小的资源单位
3.2 ReplicationController资源
rc:保证指定数量的pod始终存活,rc通过标签选择器来关联pod
k8s资源的常见操作:
# 根据yaml创建资源
kubectl create -f xxx.yaml
# 查看资源状态信息：-o wide 显示更多 --show-labels 查看标签
kubectl get (pod / rc / nodes) -o wide --show-labels
# 查看资源从创建到异常的详细信息
kubectl describe pod nginx
# 删除指定资源
kubectl delete pod nginx 或者kubectl delete -f xxx.yaml
# 编辑指定资源的配置信息
kubectl edit pod nginx
创建一个rc：nginx-rc.yaml
# 创建属于RC相关的yaml文件的存放目录
[root@k8s-master pod]# mkdir /root/k8s_yaml/rc && cd /root/k8s_yaml/rc
[root@k8s-master rc]# vim nginx-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 5 #副本5
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
创建RC并查看RC副本数量状态
[root@k8s-master rc]# kubectl create -f nginx-rc.yaml
replicationcontroller "nginx" created
[root@k8s-master rc]# kubectl get rc
NAME DESIRED CURRENT READY AGE
nginx 5 5 5 43s
[root@k8s-master rc]# kubectl get pod
导入并推送“10.0.0.11:5000/nginx:1.15”镜像；方便下面进行升级与回滚演示
# 向私有仓库"registry"上传本地镜像tar包；存放到"/opt/images"目录下
# 导入本地上传基础镜像
rz docker_nginx1.15.tar.gz
[root@k8s-node-1 images]# docker load -i docker_nginx1.15.tar.gz
[root@k8s-node-1 images]# docker tag docker.io/nginx:latest 10.0.0.11:5000/nginx:1.15
[root@k8s-node-1 images]# docker push 10.0.0.11:5000/nginx:1.15
rc的滚动升级
新建一个nginx-rc1.15.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx2
spec:
replicas: 5 #副本5
selector:
app: myweb2
template:
metadata:
labels:
app: myweb2
spec:
containers:
- name: myweb2
image: 10.0.0.11:5000/nginx:1.15
ports:
- containerPort: 80
创建RC – nginx1.13版
[root@k8s-master rc]# kubectl create -f nginx-rc.yaml
[root@k8s-master rc]# kubectl get pod
[root@k8s-master rc]# kubectl get rc
升级
# 滚动升级
[root@k8s-master rc]# kubectl rolling-update nginx -f nginx-rc1.15.yaml --update-period=10s
# 升级过程中可以观察Pod的变化
[root@k8s-master rc]# kubectl get pod -w
回滚
滚动回滚
[root@k8s-master rc]# kubectl rolling-update nginx2 -f nginx-rc.yaml --update-period=1s
参数解释与验证版本方法
参数解释：
--update-period=10s 指定间隔时间，不指定则默认为一分钟
验证：
方法一：访问启动的Nginx容器 curl -I IP:Port 查看返回信息中版本号
方法二：在Node节点查看容器启动时使用的镜像 docker ps -a
方法三：查看标签与RC的Name名称 kubectl get rc -o wide 或者 kubectl get pod -o wide --show-labels
删除其中某个Pod，验证RC是否会始终保证指定数量Pod存活
[root@k8s-master rc]# kubectl get pod
[root@k8s-master rc]# kubectl delete pod nginx-1r7pg
pod "nginx-1r7pg" deleted
[root@k8s-master rc]# kubectl get pod
注意：
当某个Pod被删除时，RC会立即启动一个新的Pod；但是被删除的Pod已经找不到；可以从Pod的名称和运行时间中查看验证到。
当Node2节点服务挂掉，RC会将原本在Node2节点的Pod数量，重新在正常的节点运行相应数量的Pod；但是，当Node2节点恢复正常后，RC并不会重新将之前的Pod运行到Node2中。
删除其中某个Node节点，验证RC是否会始终保证指定数量Pod存活
[root@k8s-master rc]# kubectl get pod -o wide
[root@k8s-master rc]# kubectl get node
[root@k8s-master rc]# kubectl delete node k8s-node-2
[root@k8s-master rc]# kubectl get pod -o wide
[root@k8s-node-2 ~]# systemctl restart kubelet.service
[root@k8s-master rc]# kubectl get node
[root@k8s-master rc]# kubectl get pod -o wide
注意：
当Node2节点服务挂掉，RC会将原本在Node2节点的Pod数量，重新在正常的节点运行相应数量的Pod；但是，当Node2节点恢复正常后，RC并不会重新将之前的Pod运行到Node2中。
修改我们之前创建的Pod – nginx 的标签属性为"myweb"；验证RC是否会删除运行时间最小的Pod数量
[root@k8s-master rc]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE LABELS
nginx 1/1 Running 0 2m 172.18.95.2 k8s-node-2 app=web
nginx-ctvmk 1/1 Running 0 59m 172.18.37.6 k8s-node-1 app=myweb
nginx-gpg4g 1/1 Running 0 1h 172.18.37.4 k8s-node-1 app=myweb
nginx-hr791 1/1 Running 0 59m 172.18.37.7 k8s-node-1 app=myweb
nginx-m5pwh 1/1 Running 0 2s 172.18.95.4 k8s-node-2 app=myweb
nginx-q1rf7 1/1 Running 0 59m 172.18.37.3 k8s-node-1 app=myweb
[root@k8s-master rc]# kubectl edit pod nginx
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: 2019-11-26T09:49:10Z
labels:
app: myweb #此处原本是：web；更改为：myweb 然后保存
name: nginx
namespace: default
[root@k8s-master rc]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE LABELS
nginx 1/1 Running 0 2m 172.18.95.2 k8s-node-2 app=myweb
nginx-ctvmk 1/1 Running 0 1h 172.18.37.6 k8s-node-1 app=myweb
nginx-gpg4g 1/1 Running 0 1h 172.18.37.4 k8s-node-1 app=myweb
nginx-hr791 1/1 Running 0 1h 172.18.37.7 k8s-node-1 app=myweb
nginx-q1rf7 1/1 Running 0 1h 172.18.37.3 k8s-node-1 app=myweb
# 这里可以看到运行时间最小的Pod -- "nginx-m5pwh" 被删除了；
# 说明 rc通过标签选择器来关联pod；如果有别的Pod混进来了，RC会默认删除运行时间最小的Pod
3.3 service资源
service帮助pod暴露端口和做负载均衡
创建一个service：k8s_svc.yaml
# 查看现有Pod；此处只保留了RC创建的资源
[root@k8s-master rc]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE LABELS
nginx-2wrjn 1/1 Running 0 3m 172.18.95.3 k8s-node-2 app=myweb
nginx-9hfzl 1/1 Running 0 3m 172.18.37.2 k8s-node-1 app=myweb
nginx-bshlt 1/1 Running 0 3m 172.18.37.3 k8s-node-1 app=myweb
nginx-ltktb 1/1 Running 0 3m 172.18.95.2 k8s-node-2 app=myweb
nginx-zngfd 1/1 Running 0 3m 172.18.95.4 k8s-node-2 app=myweb
# 创建属于Service相关的yaml文件的存放目录
[root@k8s-master rc]# mkdir /root/k8s_yaml/svc && cd /root/k8s_yaml/svc
[root@k8s-master svc]# vim k8s_svc.yaml
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort #ClusterIP
ports:
- port: 80 #clusterIP
nodePort: 30000 #node port
targetPort: 80 #pod port
selector:
app: myweb
创建service,查看SVC与RC的标签是否一致，并在外部浏览器进行访问验证
[root@k8s-master svc]# kubectl create -f k8s_svc.yaml
service "myweb" created
[root@k8s-master svc]# kubectl get svc -o wide
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes 10.254.0.1 <none> 443/TCP 1d <none>
myweb 10.254.218.48 <nodes> 80:30000/TCP 1m app=myweb
[root@k8s-master svc]# kubectl get rc -o wide
NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
nginx 5 5 5 19m myweb 10.0.0.11:5000/nginx:1.13 app=myweb
# 浏览器访问：http://node_IP:30000；访问正常，出现Nginx的首页；
# 验证SVC实现负载的效果详情见下方。
#扩容或缩容 Deployment、ReplicaSet、Replication Controller或 Job 中Pod数量。
# 查看RC:nginx的Pod数量
[root@k8s-master svc]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE LABELS
nginx-2wrjn 1/1 Running 0 3m 172.18.95.3 k8s-node-2 app=myweb
nginx-9hfzl 1/1 Running 0 3m 172.18.37.2 k8s-node-1 app=myweb
nginx-bshlt 1/1 Running 0 3m 172.18.37.3 k8s-node-1 app=myweb
nginx-ltktb 1/1 Running 0 3m 172.18.95.2 k8s-node-2 app=myweb
nginx-zngfd 1/1 Running 0 3m 172.18.95.4 k8s-node-2 app=myweb
# 修改rc:nginx的pod数量
[root@k8s-master svc]# kubectl scale rc nginx --replicas=2
replicationcontroller "nginx" scaled
或者
[root@k8s-master svc]# kubectl edit rc nginx
apiVersion: v1
kind: ReplicationController
metadata:
creationTimestamp: 2019-11-23T06:51:57Z
generation: 7
labels:
app: myweb
name: nginx
namespace: default
resourceVersion: "84460"
selfLink: /api/v1/namespaces/default/replicationcontrollers/nginx
uid: be236c49-0dbd-11ea-9d96-000c29c6d194
spec:
replicas: 2 # 此处原本数值为“5”；现修改为“2”；然后观察RC创建的Pod数量
selector:
app: myweb
# 再次观察RC:nginx的Pod数量
[root@k8s-master svc]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE LABELS
nginx-9hfzl 1/1 Running 0 28m 172.18.37.2 k8s-node-1 app=myweb
nginx-bshlt 1/1 Running 0 28m 172.18.37.3 k8s-node-1 app=myweb
# 编辑现有svc:myweb的标签等配置信息
[root@k8s-master svc]# kubectl edit svc myweb
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2019-11-23T07:09:09Z
name: myweb
namespace: default
resourceVersion: "84421"
selfLink: /api/v1/namespaces/default/services/myweb
uid: 24d4ec0a-0dc0-11ea-9d96-000c29c6d194
spec:
clusterIP: 10.254.198.29
ports:
- nodePort: 30000
port: 80
protocol: TCP
targetPort: 80
selector:
app: myweb
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
"/tmp/kubectl-edit-fplln.yaml" 26L, 656C written
service "myweb" edited
# 修改剩余两个Pod的首页index.html;方便下面验证负载均衡
# Service默认只支持RR(轮询模式）
# Round-Robin(RR)，轮询调度，通信中信道调度的一种策略，该调度策略使用户轮流使用共享资源，不会考虑瞬时信道条件。
[root@k8s-master svc]# kubectl exec -it nginx-9hfzl /bin/bash
root@nginx-9hfzl:/# echo Chiron-web01 > /usr/share/nginx/html/index.html
root@nginx-9hfzl:/# exit
exit
[root@k8s-master svc]# kubectl exec -it nginx-bshlt /bin/bash
root@nginx-bshlt:/# echo Chiron-web02 > /usr/share/nginx/html/index.html
root@nginx-bshlt:/# exit
exit
# 验证
[root@k8s-master svc]# curl http://10.0.0.12:30000/
Chiron-web02
[root@k8s-master svc]# curl http://10.0.0.12:30000/
Chiron-web01
# 查看指定SVC(service)的详细信息；如VIP、Pod、Port信息等
[root@k8s-master svc]# kubectl describe svc myweb
Name: myweb
Namespace: default
Labels: <none>
Selector: app=myweb
Type: NodePort
IP: 10.254.218.48
Port: <unset> 80/TCP
NodePort: <unset> 30000/TCP
Endpoints: 172.18.37.2:80,172.18.37.3:80
Session Affinity: None
No events.
扩展nodePort范围；默认支持的范围为"30000-32767"，在大型网络架构中，port端口不够分配可以修改此值
[root@k8s-master svc]# vim /etc/kubernetes/apiserver
KUBE_API_ARGS="--service-node-port-range=3000-50000"
或者
[root@k8s-master svc]# grep -Ev '^$|#' /etc/kubernetes/apiserver
[root@k8s-master svc]# sed -i 's/KUBE_API_ARGS=""/KUBE_API_ARGS="--service-node-port-range=3000-50000"/g' /etc/kubernetes/apiserver
[root@k8s-master svc]# grep -Ev '^$|#' /etc/kubernetes/apiserver
[root@k8s-master svc]# systemctl restart kube-apiserver.service
命令行创建service资源暴露随机端口方法：
kubectl expose rc nginx --type=NodePort --port=80
kubectl get service
service默认使用iptables来实现负载均衡, k8s 1.8以后的新版本中推荐使用lvs(四层负载均衡 传输层tcp,udp)
3.4 deployment资源
有rc在滚动升级之后,会造成服务访问中断,于是k8s引入了deployment资源；
造成RC在滚动升级之后访问中断的原因是：
RC滚动升级时，要保证RC定义的名称和标签等值不能一致；
这样就会造成滚动升级之后，原有的SVC与新的RC之间标签不一致；
这样SVC和Pod之间就没有办法进行关联；用户访问自然就中断了。
Deployment资源相比RC资源的好处：
升级服务不中断；
不依赖于Yaml文件；
通过改配置文件直接进行升级；
RC标签选择器，不支持模糊匹配；
Deployment标签选择器，支持模糊匹配
创建一个Deployment：k8s_svc.yaml
# 创建属于Deployment相关的yaml文件的存放目录
[root@k8s-master svc]# mkdir /root/k8s_yaml/deploy && cd /root/k8s_yaml/deploy
[root@k8s-master deploy]# vim k8s_deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
requests:
cpu: 100m
# 创建Deploy资源
[root@k8s-master deploy]# kubectl create -f k8s_deploy.yaml
deployment "nginx-deployment" created
Deployment资源关联Service资源实现负载均衡
# 查看所有资源
[root@k8s-master deploy]# kubectl get all
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx-deployment 3 3 3 3 6m
NAME DESIRED CURRENT READY AGE
rc/nginx 5 5 5 41m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 1d
svc/myweb 10.254.218.48 <nodes> 80:30000/TCP 3h
NAME DESIRED CURRENT READY AGE
rs/nginx-deployment-2807576163 3 3 3 6m
NAME READY STATUS RESTARTS AGE
po/nginx-3xb2r 1/1 Running 0 41m
po/nginx-c8g2g 1/1 Running 0 41m
po/nginx-deployment-2807576163-2np1q 1/1 Running 0 6m
po/nginx-deployment-2807576163-5r55r 1/1 Running 0 6m
po/nginx-deployment-2807576163-nsntt 1/1 Running 0 6m
po/nginx-tq9j1 1/1 Running 0 41m
po/nginx-w3d06 1/1 Running 0 41m
po/nginx-xln8n 1/1 Running 0 41m
# 为Deployment资源：nginx-deployment；创建service资源并暴露随机端口
[root@k8s-master deploy]# kubectl expose deployment nginx-deployment --port=80 --target-port=80 --type=NodePort
service "nginx-deployment" exposed
# 再次查看所有资源
[root@k8s-master deploy]# kubectl get all
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx-deployment 3 3 3 3 15m
NAME DESIRED CURRENT READY AGE
rc/nginx 5 5 5 50m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 1d
svc/myweb 10.254.218.48 <nodes> 80:30000/TCP 4h
svc/nginx-deployment 10.254.133.177 <nodes> 80:31469/TCP 3s
NAME DESIRED CURRENT READY AGE
rs/nginx-deployment-2807576163 3 3 3 15m
NAME READY STATUS RESTARTS AGE
po/nginx-3xb2r 1/1 Running 0 50m
po/nginx-c8g2g 1/1 Running 0 50m
po/nginx-deployment-2807576163-2np1q 1/1 Running 0 15m
po/nginx-deployment-2807576163-5r55r 1/1 Running 0 15m
po/nginx-deployment-2807576163-nsntt 1/1 Running 0 15m
po/nginx-tq9j1 1/1 Running 0 50m
po/nginx-w3d06 1/1 Running 0 50m
po/nginx-xln8n 1/1 Running 0 50m
# 验证访问正常；如果想验证是否负载均衡了，可以依据SVC中方法进行验证
[root@k8s-master deploy]# curl -I 10.0.0.12:31469
HTTP/1.1 200 OK
Server: nginx/1.13.12
[root@k8s-master deploy]# curl -I 10.0.0.13:31469
HTTP/1.1 200 OK
Server: nginx/1.13.12
注意：
Deployment资源在升级回滚时，不再像RC一样需要依据Yaml配置文件；
可通过直接更改配置文件，既能达到立即升级的效果。
[root@k8s-master deploy]# kubectl edit deployment nginx-deployment
spec:
containers:
- image: 10.0.0.11:5000/nginx:1.15 #将原有"1.13"的镜像更改为"1.15"
imagePullPolicy: IfNotPresent
name: nginx
# 查看资源的变化
[root@k8s-master deploy]# kubectl get all
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx-deployment 3 3 3 3 30m
NAME DESIRED CURRENT READY AGE
rc/nginx 5 5 5 1h
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 1d
svc/myweb 10.254.218.48 <nodes> 80:30000/TCP 4h
svc/nginx-deployment 10.254.133.177 <nodes> 80:31469/TCP 15m
NAME DESIRED CURRENT READY AGE
rs/nginx-deployment-2807576163 0 0 0 30m
rs/nginx-deployment-3014407781 3 3 3 5m
NAME READY STATUS RESTARTS AGE
po/nginx-3xb2r 1/1 Running 0 1h
po/nginx-c8g2g 1/1 Running 0 1h
po/nginx-deployment-3014407781-k5crz 1/1 Running 0 5m
po/nginx-deployment-3014407781-mh9hq 1/1 Running 0 5m
po/nginx-deployment-3014407781-wqdc7 1/1 Running 0 5m
po/nginx-tq9j1 1/1 Running 0 1h
po/nginx-w3d06 1/1 Running 0 1h
po/nginx-xln8n 1/1 Running 0 1h
# 验证访问到的最新版本号；并且服务未中断
[root@k8s-master deploy]# curl -I 10.0.0.12:31469
HTTP/1.1 200 OK
Server: nginx/1.15.5
[root@k8s-master deploy]# curl -I 10.0.0.13:31469
HTTP/1.1 200 OK
Server: nginx/1.15.5
# 服务不中断的原因，在于Deployment资源启动的Pod，拥有两个标签，升级回滚过程中变化的只会是另一个标签，而标签“app=nginx”是始终不会变化的，此处也证明了Deployment资源是支持模糊匹配的。
[root@k8s-master deploy]# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-3xb2r 1/1 Running 0 1h app=myweb
nginx-c8g2g 1/1 Running 0 1h app=myweb
nginx-deployment-3014407781-k5crz 1/1 Running 0 18m app=nginx,pod-template-hash=3014407781
nginx-deployment-3014407781-mh9hq 1/1 Running 0 18m app=nginx,pod-template-hash=3014407781
nginx-deployment-3014407781-wqdc7 1/1 Running 0 18m app=nginx,pod-template-hash=3014407781
nginx-tq9j1 1/1 Running 0 1h app=myweb
nginx-w3d06 1/1 Running 0 1h app=myweb
nginx-xln8n 1/1 Running 0 1h app=myweb
deployment升级和回滚命令总结：
命令行创建deployment：
kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
命令行升级版本：
kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
查看deployment所有历史版本：
kubectl rollout history deployment nginx
deployment回滚到上一个版本：
kubectl rollout undo deployment nginx
deployment回滚到指定版本：
kubectl rollout undo deployment nginx --to-revision=2
命令行创建deployment：
# 查看我们之前将Deployment资源从"1.13"升级到"1.15"的历史版本记录；是没有做记录的改变；后面我们将通过下述方法进行有历史版本记录的演示
[root@k8s-master deploy]# kubectl rollout history deployment nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
1 <none>
2 <none>
# 新建一个Deployment资源：nginx；并记录命令与版本
[root@k8s-master deploy]# kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
deployment "nginx" created
# 参数解释：
--image：指定镜像版本
--replicas：指定副本数
--record：记录命令，方便查看每次版本的变化；在资源注释中记录当前的kubectl命令。
如果设置为false，则不记录命令。
如果设置为true，则记录命令。
如果未设置，则默认为仅在现有注释值存在时更新该值。
# 再次查看Deployment资源历史版本操作，与上面对比，发现有完整的历史命令操作；同事也方便后期回滚到指定的版本
[root@k8s-master deploy]# kubectl rollout history deployment nginx
deployments "nginx"
REVISION CHANGE-CAUSE
1 kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
# 弊端；为Deployment资源进行版本升级时，不能再通过修改配置文件“kubectl edit deployment nginx”进行升级了；否则记录的历史版本操作命令会如下，不方便我们查看做了哪些操作
[root@k8s-master deploy]# kubectl edit deployment nginx
[root@k8s-master deploy]# kubectl edit deployment nginx
# 修改镜像版本为“1.15”与“1.17”后，查看内历史版本只会记录命令，而不清楚执行了哪些操作
[root@k8s-master deploy]# kubectl rollout history deployment nginx
deployments "nginx"
REVISION CHANGE-CAUSE
1 kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
2 kubectl edit deployment nginx
3 kubectl edit deployment nginx
# 注意：
kubectl run 命令：在K8S中默认启动一个Deployment资源
docker run 命令：在Docker中默认启动一个Pod容器
命令行升级版本；并且能通过查看历史版本做了哪些操作
# Node节点上传"nginx:1.17"的镜像到仓库中
[root@k8s-node-1 images]# wget http://192.168.2.254:8888/docker_nginx.tar.gz
[root@k8s-node-1 images]# docker load -i docker_nginx.tar.gz
[root@k8s-node-1 images]# docker tag nginx:latest 10.0.0.11:5000/nginx:1.17
[root@k8s-node-1 images]# docker push 10.0.0.11:5000/nginx:1.17
# 首先清除Deployment资源“nginx”然后新建
[root@k8s-master deploy]# kubectl delete deployment nginx
deployment "nginx" deleted
[root@k8s-master deploy]# kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
deployment "nginx" created
# 升级版本为“1.15”
[root@k8s-master deploy]# kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
deployment "nginx" image updated
# 升级版本为“1.17”
[root@k8s-master deploy]# kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.17
deployment "nginx" image updated
# 查看Pod,找到其中一个Pod的IP地址；验证版本“1.17”
[root@k8s-master deploy]# kubectl get pod -o wide
[root@k8s-master deploy]# curl -I 172.18.37.4
HTTP/1.1 200 OK
Server: nginx/1.17.5
# 通过过滤RS也能查看到每次版本的不同；因为Deployment资源升级时，会新生成RS资源，而旧的不会被删除
[root@k8s-master deploy]# kubectl get rs -o wide
NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
nginx-1147444844 3 3 3 2m nginx 10.0.0.11:5000/nginx:1.17 pod-template-hash=1147444844,run=nginx
nginx-847814248 0 0 0 2m nginx 10.0.0.11:5000/nginx:1.13 pod-template-hash=847814248,run=nginx
nginx-997629546 0 0 0 2m nginx 10.0.0.11:5000/nginx:1.15 pod-template-hash=997629546,run=nginx
nginx-deployment-2807576163 0 0 0 1h nginx 10.0.0.11:5000/nginx:1.13 app=nginx,pod-template-hash=2807576163
nginx-deployment-3014407781 3 3 3 1h nginx 10.0.0.11:5000/nginx:1.15 app=nginx,pod-template-hash=3014407781
# 查看带有升级详情的历史记录
[root@k8s-master deploy]# kubectl rollout history deployment nginx
deployments "nginx"
REVISION CHANGE-CAUSE
1 kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
2 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
3 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.17
查看deployment所有历史版本：
kubectl rollout history deployment nginx
deployment回滚到上一个版本；查看历史记录有什么变化？
[root@k8s-master deploy]# kubectl rollout undo deployment nginx
deployment "nginx" rolled back
[root@k8s-master deploy]# kubectl rollout history deployment nginx
deployments "nginx"
REVISION CHANGE-CAUSE
1 kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
3 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.17
4 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
# 结论：
不指定版本号回滚，默认回滚到上一个版本“1.15”，
并且“1.15”原本版本号“2”变成了“4”；即版本“4”相当于是拿版本“2”的命令重新执行了一次；
且更新了版本号，由最大版本值向下进行迭代。
deployment回滚到指定版本；查看历史记录有什么变化？
[root@k8s-master deploy]# kubectl rollout undo deployment nginx --to-revision=1
deployment "nginx" rolled back
[root@k8s-master deploy]# kubectl rollout history deployment nginx
deployments "nginx"
REVISION CHANGE-CAUSE
3 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.17
4 kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
5 kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
# 参数解释：
--to-revision：指定回滚的版本号
# 结论如上所属
3.5 Daemon Set资源
DaemonSet确保集群中每个（部分）node运行一份pod副本，当node加入集群时创建pod，当node离开集群时回收pod；如果删除DaemonSet，其创建的所有pod也被删除；
DaemonSet中的pod覆盖整个集群；
当需要在集群内每个node运行同一个类型pod，使用DaemonSet是有价值的，以下是典型使用场景：
运行集群存储守护进程，如glusterd、ceph。
运行集群日志收集守护进程，如fluentd、logstash、filebeat。
运行节点监控守护进程，如Prometheus Node Exporter, collectd, Datadog agent, New Relic agent, or Ganglia gmond, cadvisor, node-exporter。
创建一个DaemonSet：k8s_daemon_set.yaml
# 创建属于DaemonSet相关的yaml文件的存放目录
[root@k8s-master deploy]# mkdir /root/k8s_yaml/daemon && cd /root/k8s_yaml/daemon
[root@k8s-master daemon]# vim k8s_daemon_set.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: daemon
spec:
template:
metadata:
labels:
app: daemon
spec:
containers:
- name: daemon
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
# 创建DaemonSet资源
[root@k8s-master daemon]# kubectl create -f k8s_daemon_set.yaml
daemonset "daemon" created
# 查看DaemonSet创建的Pod;并且是一个节点分配一个
[root@k8s-master daemon]# kubectl get pod -o wide|grep daemon
daemon-0bpjx 1/1 Running 0 2s 172.18.95.4 k8s-node-2
daemon-8wrb8 1/1 Running 0 2s 172.18.37.8 k8s-node-1
# 查看DaemonSet状态
[root@k8s-master daemon]# kubectl get daemonset
NAME DESIRED CURRENT READY NODE-SELECTOR AGE
daemon 2 2 2 <none> 1m
3.6 Pet Set资源
在Kubernetes中，大多数的Pod管理都是基于无状态、一次性的理念。例如Replication Controller、Deployment，它只是简单的保证可提供服务的Pod数量。
无数据的容器，无状态的容器
PetSet，是由一组有状态的Pod组成，每个Pod有自己特殊且不可改变的ID，且每个Pod中都有自己独一无二、不能删除的数据。
有数据的容器，有状态的容器
当应用有以下任意要求时，Pet Set的价值就体现出来了。
● 稳定的、唯一的网络标识。
● 稳定的、持久化的存储。
● 有序的、优雅的部署和扩展。
● 有序的、优雅的删除和停止。
一个有序的index（比如PetSet的名字叫mysql，那么第一个启起来的Pet就叫mysql-0，第二个叫mysql-1，如此下去。当一个Pet down掉后，新创建的Pet会被赋予跟原来Pet一样的名字，通过这个名字就能匹配到原来的存储，实现状态保存。）
当应用有以下任意要求时，StatefulSet的价值就体现出来了。
● 稳定的、唯一的网络标识。
● 稳定的、持久化的存储。
● 有序的、优雅的部署和扩展。
● 有序的、优雅的删除和停止。
3.7 Job资源
对于ReplicaSet、ReplicationController等类型的控制器而言，它希望pod保持预期数目、持久运行下去，除非用户明确删除，否则这些对象一直存在，它们针对的是耐久性任务，如web服务等。
对于非耐久性任务，比如压缩文件，任务完成后，pod需要结束运行，不需要pod继续保持在系统中，这个时候就要用到Job。因此说Job是对ReplicaSet、ReplicationController等持久性控制器的补充。
通常执行一次性任务，执行结束就删除。还可以设置其定期运行。
3.8 tomcat+mysql练习
在k8s中容器之间相互访问,通过VIP地址!
Yaml文件：
::::::::::::::
./mysql-rc.yaml
::::::::::::::
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
::::::::::::::
./mysql-svc.yaml
::::::::::::::
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: mysql
::::::::::::::
./tomcat-rc.yaml
::::::::::::::
apiVersion: v1
kind: ReplicationController
metadata:
name: myweb
spec:
replicas: 1
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: 10.0.0.11:5000/tomcat-app:v2
ports:
- containerPort: 8080
env:
- name: MYSQL_SERVICE_HOST
value: 'mysql'
- name: MYSQL_SERVICE_PORT
value: '3306'
::::::::::::::
./tomcat-svc.yaml
::::::::::::::
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 8080
nodePort: 30008
selector:
app: myweb
::::::::::::::
mysql_pvc.yaml
::::::::::::::
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: tomcat-mysql
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
::::::::::::::
mysql_pv.yaml
::::::::::::::
apiVersion: v1
kind: PersistentVolume
metadata:
name: tomcat-mysql
labels:
type: nfs001
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
nfs:
path: "/data/pv"
server: 10.0.0.11
readOnly: false
::::::::::::::
mysql-rc-pvc.yaml
::::::::::::::
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
volumeMounts:
- name: nfs
mountPath: /var/lib/mysql
volumes:
- name: nfs
persistentVolumeClaim:
claimName: tomcat-mysql
前期准备：
# 清空前面创建的所有资源
[root@k8s-master deploy]# kubectl get all
[root@k8s-master deploy]# kubectl delete svc --all
[root@k8s-master deploy]# kubectl delete rc --all
[root@k8s-master deploy]# kubectl delete deployment --all
[root@k8s-master deploy]# kubectl get all
# 上传事先写好的Yaml文件的压缩包到Master节点(与上方Yaml文件一致)
[root@k8s-master k8s_yaml]# wget http://192.168.2.254:8888/tomcat_demo.zip
# 解压并进入该目录
[root@k8s-master k8s_yaml]# unzip tomcat_demo.zip && cd /root/k8s_yaml/tomcat_demo/
# 将文件名包含"pv"的yaml文件存放到“bak_pv”目录下，因为我们还没涉及到持久化存储的知识
[root@k8s-master tomcat_demo]# mkdir bak_pv && mv *pv*.yaml bak_pv && ls
bak_pv mysql-rc.yaml mysql-svc.yaml tomcat-rc.yaml tomcat-svc.yaml
# 上传本地镜像包并导入到镜像仓库中
[root@k8s-node-1 images]# wget http://192.168.2.254:8888/tomcat-app-v2.tar.gz
[root@k8s-node-1 images]# wget http://192.168.2.254:8888/docker-mysql-5.7.tar.gz
[root@k8s-node-1 images]# docker load -i tomcat-app-v2.tar.gz
[root@k8s-node-1 images]# docker load -i docker-mysql-5.7.tar.gz
[root@k8s-node-1 images]# docker tag docker.io/kubeguide/tomcat-app:v2 10.0.0.11:5000/tomcat-app:v2
[root@k8s-node-1 images]# docker tag docker.io/mysql:5.7 10.0.0.11:5000/mysql:5.7
[root@k8s-node-1 images]# docker push 10.0.0.11:5000/mysql:5.7
[root@k8s-node-1 images]# docker push 10.0.0.11:5000/tomcat-app:v2
操作步骤：
# 创建Mysql5.7的RC和SVC；SVC的目的在于，如果容器重启了，还能关联到SVC,Tomcat容器需要连接Mysql数据库，也就是SVC的VIP
# 创建RC -- Mysql5.7
[root@k8s-master tomcat_demo]# kubectl create -f mysql-rc.yaml
replicationcontroller "mysql" created
# 创建SVC -- mysql5.7
[root@k8s-master tomcat_demo]# kubectl create -f mysql-svc.yaml
service "mysql" created
# 查看创建的资源，记录下“svc/mysql”的VIP地址，用于Tomcat容器连接
[root@k8s-master tomcat_demo]# kubectl get all
NAME DESIRED CURRENT READY AGE
rc/mysql 1 1 1 46s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 2d
svc/mysql 10.254.34.217 <none> 3306/TCP 39s
NAME READY STATUS RESTARTS AGE
po/mysql-nb2g8 1/1 Running 0 46s
# 修改“tomcat-rc.yaml”中Mysql的地址；因为没有创建DNS,所以需要将主机名替换为刚刚获取的“svc/mysql”的VIP地址
[root@k8s-master tomcat_demo]# vim tomcat-rc.yaml
- name: MYSQL_SERVICE_HOST
value: '10.254.34.217'
- name: MYSQL_SERVICE_PORT
value: '3306'
# 创建RC -- tomcat(myweb)
[root@k8s-master tomcat_demo]# kubectl create -f tomcat-rc.yaml
replicationcontroller "myweb" created
# 创建SVC -- tomcat(myweb)
[root@k8s-master tomcat_demo]# kubectl create -f tomcat-svc.yaml
service "myweb" created
# 查看所有资源
[root@k8s-master tomcat_demo]# kubectl get all
NAME DESIRED CURRENT READY AGE
rc/mysql 1 1 1 11m
rc/myweb 1 1 1 58s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 2d
svc/mysql 10.254.34.217 <none> 3306/TCP 11m
svc/myweb 10.254.100.94 <nodes> 8080:30008/TCP 53s
NAME READY STATUS RESTARTS AGE
po/mysql-nb2g8 1/1 Running 0 11m
po/myweb-9ghl7 1/1 Running 0 58s
# 验证：浏览器访问
http://10.0.0.12:30008/demo/
http://10.0.0.13:30008/demo/
# 随后点击“ADD”插入一条数据，进行数据插入验证
# 登录mysql -- Pod；查看数据库中是否有相同数据
[root@k8s-master tomcat_demo]# kubectl exec -it mysql-nb2g8 /bin/bash
# 默认创建的库为“HPE_APP”
root@mysql-nb2g8:/# mysql -uroot -p123456
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| HPE_APP |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
mysql> use HPE_APP;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+-------------------+
| Tables_in_HPE_APP |
+-------------------+
| T_USERS |
+-------------------+
1 row in set (0.00 sec)
mysql> select * from T_USERS;
+----+-----------+-------+
| ID | USER_NAME | LEVEL |
+----+-----------+-------+
| 1 | me | 100 |
| 2 | our team | 100 |
| 3 | HPE | 100 |
| 4 | teacher | 100 |
| 5 | docker | 100 |
| 6 | google | 100 |
| 7 | Chiron | 10000 |
+----+-----------+-------+
7 rows in set (0.00 sec)
mysql> exit
Bye
root@mysql-nb2g8:/# exit
exit
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WyPvFkEO-1576134541636)(F:\Docker与K8S训练营\Mysql+Tomcat.jpg)]
4:k8s的附加组件
k8s集群中dns服务的作用,就是将SVC的名字解析成VIP地址
4.1 dns服务
安装dns服务(指定Node2节点进行部署)
1:上传本地dns_docker镜像包并导入dns_docker镜像包(node2节点)
[root@k8s-node-2 ~]# mkdir /opt/images && cd /opt/images
[root@k8s-node-2 images]# wget http://192.168.2.254:8888/docker_k8s_dns.tar.gz
[root@k8s-node-2 images]# docker load -i docker_k8s_dns.tar.gz
[root@k8s-node-2 images]# docker images|grep gcr
gcr.io/google_containers/kubedns-amd64 1.9 26cf1ed9b144 3 years ago 47 MB
gcr.io/google_containers/dnsmasq-metrics-amd64 1.0 5271aabced07 3 years ago 14 MB
gcr.io/google_containers/kube-dnsmasq-amd64 1.4 3ec65756a89b 3 years ago 5.13 MB
gcr.io/google_containers/exechealthz-amd64 1.2 93a43bfb39bf 3 years ago 8.37 MB
2:将DNS服务的相关Yaml文件上传到Master节点指定位置
# Master节点创建存放DNS服务相关的Yaml文件存放位置
[root@k8s-master tomcat_demo]# mkdir /root/k8s_yaml/dns && cd /root/k8s_yaml/dns
rz skydns-rc.yaml skydns-svc.yaml
3:修改skydns-rc.yaml, 指定在node2 创建dns服务
spec:
nodeName: k8s-node-2 #固定在Node2节点创建资源
containers:
- name: kubedns
image: gcr.io/google_containers/kubedns-amd64:1.9
4:修改skydns-svc.yaml，指定创建SVC资源的clusterIP – VIP
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.254.230.254 #固定SVC的VIP地址
5:创建dns服务资源RC/Deployment
[root@k8s-master dns]# kubectl create -f skydns-rc.yaml
deployment "kube-dns" created
[root@k8s-master dns]# kubectl create -f skydns-svc.yaml
service "kube-dns" created
6:检查
[root@k8s-master dns]# kubectl get all --namespace=kube-system
或者
[root@k8s-master dns]# kubectl get all -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kube-dns 1 1 1 1 40s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kube-dns 10.254.230.254 <none> 53/UDP,53/TCP 39s
NAME DESIRED CURRENT READY AGE
rs/kube-dns-1656925611 1 1 1 40s
NAME READY STATUS RESTARTS AGE
po/kube-dns-1656925611-96k7m 4/4 Running 0 40s
7:修改所有node节点kubelet的配置文件：指定docker启动容器使用K8S-DNS服务器
vim /etc/kubernetes/kubelet
KUBELET_ARGS="--cluster_dns=10.254.230.254 --cluster_domain=cluster.local"
或者
grep -Ev '^$|#' /etc/kubernetes/kubelet
sed -i 's#KUBELET_ARGS=""#KUBELET_ARGS="--cluster_dns=10.254.230.254 --cluster_domain=cluster.local"#g' /etc/kubernetes/kubelet
grep -Ev '^$|#' /etc/kubernetes/kubelet
# 重启服务
systemctl restart kubelet
8:示例Mysql+Tomcat，将之前我们修改的“tomcat-rc.yaml”中Mysql的地址；由VIP替换为原来的主机名“mysql”
# 清除之前创建的Mysql+Tomcat资源
[root@k8s-master dns]# cd /root/k8s_yaml/tomcat_demo/
[root@k8s-master tomcat_demo]# kubectl delete -f .
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 修改“tomcat-rc.yaml”中Mysql的地址
[root@k8s-master tomcat_demo]# vim tomcat-rc.yaml
env:
- name: MYSQL_SERVICE_HOST
value: 'mysql'
- name: MYSQL_SERVICE_PORT
value: '3306'
# 创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f .
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 访问验证服务是否正常；浏览器亦可
[root@k8s-master tomcat_demo]# curl -I http://10.0.0.13:30008/demo/
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
# 服务器进入Tomcat容器中验证；看到resolv.conf中先走k8S的DNS解析
[root@k8s-master tomcat_demo]# kubectl exec -it myweb-qjc5x bash
root@myweb-qjc5x:/usr/local/tomcat# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.254.230.254
nameserver 180.76.76.76
nameserver 223.5.5.5
options ndots:5
# 当ping -- Mysql时，将返回mysql的VIP地址
# 注意：在容器中，ping -- IP 是永远不通的
root@myweb-qjc5x:/usr/local/tomcat# ping mysql
PING mysql.default.svc.cluster.local (10.254.100.255): 56 data bytes
4.2 namespace命令空间
namespace做资源隔离
默认K8S的Namespace为：default
生产中一般建议，一个业务对应一个单独的Namepace
# 查看K8S集群中所有namespace
[root@k8s-master tomcat_demo]# kubectl get namespace
NAME STATUS AGE
default Active 2d
kube-system Active 2d
# 查看Pod，默认选择的Namespace为：default
[root@k8s-master tomcat_demo]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-dr2fz 1/1 Running 0 57m
myweb-qjc5x 1/1 Running 0 57m
[root@k8s-master tomcat_demo]# kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
mysql-dr2fz 1/1 Running 0 57m
myweb-qjc5x 1/1 Running 0 57m
# 查看指定Namespace：kube-system的资源
[root@k8s-master tomcat_demo]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kube-dns-1656925611-c2270 4/4 Running 0 58m
# 将Namespace：default中Mysql+Tomcat资源删除
[root@k8s-master tomcat_demo]# kubectl delete -f .
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 创建Namespace：tomcat
[root@k8s-master tomcat_demo]# kubectl create namespace tomcat
namespace "tomcat" created
# 查看Namespace列表
[root@k8s-master tomcat_demo]# kubectl get namespace
NAME STATUS AGE
default Active 2d
kube-system Active 2d
tomcat Active 48s
# 更改“yaml”文件；指定Namespace；不指定就默认选择Namespace为：default
# 测试在第三行之后追加一行指定Namespace;并查看前五行效果
[root@k8s-master tomcat_demo]# sed '3a \ \ namespace: tomcat' mysql-svc.yaml |head -5
apiVersion: v1
kind: Service
metadata:
namespace: tomcat #新增的行
name: mysql
# 将当前目录下所有“Yaml”文件指定Namespace
[root@k8s-master tomcat_demo]# sed -i '3a \ \ namespace: tomcat' *.yaml
# 创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f .
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看验证Mysql+Tomcat资源；分配到了Namespace：tomcat中
[root@k8s-master tomcat_demo]# kubectl get all -n default
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.254.0.1 <none> 443/TCP 2d
[root@k8s-master tomcat_demo]# kubectl get all -n tomcat
NAME DESIRED CURRENT READY AGE
rc/mysql 1 1 1 40s
rc/myweb 1 1 1 40s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/mysql 10.254.170.16 <none> 3306/TCP 40s
svc/myweb 10.254.130.189 <nodes> 8080:30008/TCP 39s
NAME READY STATUS RESTARTS AGE
po/mysql-pwtsp 1/1 Running 0 40s
po/myweb-jbvd6 1/1 Running 0 40s
# 访问Tomcat依旧是正常的
[root@k8s-master tomcat_demo]# curl -I http://10.0.0.13:30008/demo/
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
4.3 健康检查
4.3.1 探针的种类
livenessProbe：健康状态检查，周期性检查服务是否存活，检查结果失败，将重启容器
readinessProbe：可用性检查，周期性检查服务是否可用，不可用将从service的endpoints中移除
4.3.2 探针的检测方法
- exec：执行一段命令 返回值为0, 非0
- httpGet：检测某个 http 请求的返回状态码 2xx,3xx正常, 4xx,5xx错误
- tcpSocket：测试某个端口是否能够连接
# Master节点创建存放Health健康检查相关的Yaml文件存放位置
[root@k8s-master tomcat_demo]# mkdir /root/k8s_yaml/health && cd /root/k8s_yaml/health
4.3.3 liveness探针的exec使用：
nginx_pod_exec.yaml – 通过检测文件是否存在(即执行一段命令返回的状态码是否正常)
apiVersion: v1
kind: Pod
metadata:
name: exec
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 1
扩展：
K8S中容器的初始命令：
args:
command:
Dockerfile中容器的初始命令：
CMD:
ENTRYPOINT:
# 创建nginx_pod_exec资源
[root@k8s-master health]# kubectl create -f nginx_pod_exec.yaml
pod "exec" created
# 查看nginx_pod_exec资源的Pod状态；发现每过一段时间，Pod将执行一次重启
[root@k8s-master health]# kubectl get pod -w
NAME READY STATUS RESTARTS AGE
exec 1/1 Running 0 2s
NAME READY STATUS RESTARTS AGE
exec 1/1 Running 1 1m
exec 1/1 Running 2 2m
# 查看nginx_pod_exec资源的Pod详情，也是可以看到Pod的状态变化
[root@k8s-master ~]# kubectl describe pod exec
container "nginx" is unhealthy, it will be killed and re-created.
4.3.4 liveness探针的httpGet使用：
nginx_pod_httpGet.yaml – 通过检测服务访问的URL返回的状态码是否正常(即观察访问的值是否为“302”等)
apiVersion: v1
kind: Pod
metadata:
name: httpget
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /index.html
port: 80
initialDelaySeconds: 3
periodSeconds: 3
# 创建nginx_pod_httpGet资源
[root@k8s-master health]# kubectl create -f nginx_pod_httpGet.yaml
pod "httpget" created
# 进入容器删除首页
[root@k8s-master health]# kubectl exec -it httpget bash
root@httpget:/# rm -rf /usr/share/nginx/html/index.html
root@httpget:/# exit
exit
# 查看容器状态，发现立马进行了一次重启；这时容器有了首页就正常了
[root@k8s-master ~]# kubectl get pod httpget -w
NAME READY STATUS RESTARTS AGE
httpget 1/1 Running 0 1m
httpget 1/1 Running 1 2m
# 查看容器详情，发现检测到Nginx状态404，就有了下面的重启操作
[root@k8s-master ~]# kubectl describe pod httpget
Liveness probe failed: HTTP probe failed with statuscode: 404
4.3.5 liveness探针的tcpSocket使用：
nginx_pod_tcpSocket.yaml – 通过检测“80”端口是否启用，来判断容器的正常与否(想要正常，需进入容器将“80”端口的Nginx服务启动)
apiVersion: v1
kind: Pod
metadata:
name: tcpsocket
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
args:
- /bin/sh
- -c
- tail -f /etc/hosts
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 10
periodSeconds: 3
# 创建nginx_pod_tcpSocket资源
[root@k8s-master health]# kubectl create -f nginx_pod_tcpSocket.yaml
pod "tcpsocket" created
# 因为我们未进入容器将Nginx服务启动；80端口检测不到，经过一个周期，容器被重启了
[root@k8s-master ~]# kubectl get pod tcpsocket -w
NAME READY STATUS RESTARTS AGE
tcpsocket 1/1 Running 0 15s
tcpsocket 1/1 Running 1 48s
# 此时我们进入容器将Nginx服务启动
[root@k8s-master health]# kubectl exec -it tcpsocket bash
root@tcpsocket:/# /usr/sbin/nginx
root@tcpsocket:/# exit
exit
# 这时再看到Pod状态和详情，发现没有任何变化
[root@k8s-master ~]# kubectl get pod tcpsocket -w
[root@k8s-master ~]# kubectl describe pod tcpsocket
4.3.6 readiness探针的httpGet使用：
nginx-rc-httpGet.yaml – 通过检测服务访问的URL返回的状态码是否正常(即容器启动并不会显示正常关联到SVC；只有一切正常时，才会关联到SVC；这就诠释了“可用性”)
apiVersion: v1
kind: ReplicationController
metadata:
name: readiness
spec:
replicas: 2
selector:
app: readiness
template:
metadata:
labels:
app: readiness
spec:
containers:
- name: readiness
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /Chiron.html
port: 80
initialDelaySeconds: 3
periodSeconds: 3
# 创建nginx-rc-httpGet资源
[root@k8s-master health]# kubectl create -f nginx-rc-httpGet.yaml
replicationcontroller "readiness" created
# 查看Pod状态，发现是运行的，但是容器并没有准备好
[root@k8s-master health]# kubectl get pod |grep readiness
readiness-db8cw 0/1 Running 0 1m
readiness-rmpjc 0/1 Running 0 1m
# 创建SVC资源，管理POD：readiness
[root@k8s-master health]# kubectl expose rc readiness --port=80 --target-port=80 --type=NodePort
service "readiness" exposed
# 查看SVC：readiness资源详情；并没有关联到Pod
[root@k8s-master health]# kubectl describe svc readiness
Name: readiness
Namespace: default
Labels: app=readiness
Selector: app=readiness
Type: NodePort
IP: 10.254.227.196
Port: <unset> 80/TCP
NodePort: <unset> 32089/TCP
Endpoints:
Session Affinity: None
No events.
# 进入其中一个Pod,添加Chiron.html页面
[root@k8s-master health]# kubectl exec -it readiness-db8cw bash
root@readiness-db8cw:/# echo 'web01' > /usr/share/nginx/html/Chiron.html
root@readiness-db8cw:/# exit
exit
# 再次查看SVC：readiness资源详情；关联到了一个；验证了“可用性”
[root@k8s-master health]# kubectl describe svc readiness
Name: readiness
Namespace: default
Labels: app=readiness
Selector: app=readiness
Type: NodePort
IP: 10.254.227.196
Port: <unset> 80/TCP
NodePort: <unset> 32089/TCP
Endpoints: 172.18.95.4:80
Session Affinity: None
No events.
# 查看Pod：readiness状态；一个正常(准备好的)，一个异常(未准备好)
[root@k8s-master health]# kubectl get pod -o wide |grep readiness
readiness-db8cw 1/1 Running 0 10m 172.18.95.4 k8s-node-2
readiness-rmpjc 0/1 Running 0 10m 172.18.37.6 k8s-node-1
4.4 dashboard服务
1:Node2节点上传并导入镜像,打标签
rz kubernetes-dashboard-amd64_v1.4.1.tar.gz
[root@k8s-node-2 images]# docker load -i kubernetes-dashboard-amd64_v1.4.1.tar.gz
[root@k8s-node-2 images]# docker tag index.tenxcloud.com/google_containers/kubernetes-dashboard-amd64:v1.4.1 10.0.0.10:5000/kubernetes-dashboard-amd64:v1.4.1
2:Master节点创建dashborad的deployment和service
# 创建dashborad相关的Yaml文件存放位置
[root@k8s-master health]# mkdir /root/k8s_yaml/dashboard && cd /root/k8s_yaml/dashboard
# 上传事先准备好的Yaml文件
rz dashboard.yaml && rz dashboard-svc.yaml
# 修改dashboard.yaml文件，指定Pod创建时运行在Node2节点上
[root@k8s-master dashboard]# vim dashboard.yaml
spec:
nodeName: k8s-node-2
containers:
- name: kubernetes-dashboard
# 创建dashborad相关资源
[root@k8s-master dashboard]# kubectl create -f dashboard.yaml
deployment "kubernetes-dashboard-latest" created
[root@k8s-master dashboard]# kubectl create -f dashboard-svc.yaml
service "kubernetes-dashboard" created
# 查看Namespace：kube-system 中的资源
[root@k8s-master dashboard]# kubectl get all -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kube-dns 1 1 1 1 4h
deploy/kubernetes-dashboard-latest 1 1 1 1 2s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kube-dns 10.254.230.254 <none> 53/UDP,53/TCP 4h
svc/kubernetes-dashboard 10.254.1.236 <none> 80/TCP 1s
NAME DESIRED CURRENT READY AGE
rs/kube-dns-1656925611 1 1 1 4h
rs/kubernetes-dashboard-latest-1975875540 1 1 1 2s
NAME READY STATUS RESTARTS AGE
po/kube-dns-1656925611-c2270 4/4 Running 0 4h
po/kubernetes-dashboard-latest-1975875540-lnd2d 1/1 Running 0 2s
3:访问Dashboard：http://10.0.0.11:8080/ui/
4.5 通过apiservicer反向代理访问service
第一种：NodePort类型
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30008
第二种：ClusterIP类型
type: ClusterIP
ports:
- port: 80
targetPort: 80
http://10.0.0.11:8080/api/v1/proxy/namespaces/命令空间/services/service的名字/
http://10.0.0.13:30008/
http://10.0.0.11:8080/api/v1/proxy/namespaces/tomcat/services/myweb/ #同样可以访问Tomcat
5: k8s弹性伸缩
k8s弹性伸缩,需要附加插件heapster监控
扩展：
默认在Node节点中Kubelet集成了"cAdvisor"
http://10.0.0.12:4194/containers/
http://10.0.0.13:4194/containers/
5.1 安装heapster监控
1:在Node2中上传并导入镜像,打标签
# 创建放置Heapster监控用到的镜像包目录
[root@k8s-node-2 images]# mkdir /opt/images/heapster && cd /opt/images/heapster
# 上传本地的镜像包
[root@k8s-node-2 images]# wget http://192.168.2.254:8888/docker_heapster_grafana.tar.gz
[root@k8s-node-2 images]# wget http://192.168.2.254:8888/docker_heapster.tar.gz
[root@k8s-node-2 images]# wget http://192.168.2.254:8888/docker_heapster_influxdb.tar.gz
# 批量导入images
[root@k8s-node-2 heapster]# for n in `ls *.tar.gz`;do docker load -i $n ;done
# 为导入的images打标签
[root@k8s-node-2 heapster]# docker tag docker.io/kubernetes/heapster_grafana:v2.6.0 10.0.0.11:5000/heapster_grafana:v2.6.0
[root@k8s-node-2 heapster]# docker tag docker.io/kubernetes/heapster_influxdb:v0.5 10.0.0.11:5000/heapster_influxdb:v0.5
[root@k8s-node-2 heapster]# docker tag docker.io/kubernetes/heapster:canary 10.0.0.11:5000/heapster:canary
2:Maste节点创建存放“heapster”相关“Yaml”文件的命令；上传事先准备好的Yaml配置文件
[root@k8s-master dashboard]# mkdir /root/k8s_yaml/heapster && cd /root/k8s_yaml/heapster
[root@k8s-master heapster]# wget http://192.168.2.254:8888/grafana-service.yaml
[root@k8s-master heapster]# wget http://192.168.2.254:8888/heapster-service.yaml
[root@k8s-master heapster]# wget http://192.168.2.254:8888/influxdb-service.yaml
[root@k8s-master heapster]# wget http://192.168.2.254:8888/heapster-controller.yaml
[root@k8s-master heapster]# wget http://192.168.2.254:8888/influxdb-grafana-controller.yaml
3:修改配置文件指定在“Node2节点”创建资源
# 修改配置文件：heapster-controller.yaml
spec:
nodeName: k8s-node-2 #指定节点
containers:
- name: heapster
image: 10.0.0.11:5000/heapster:canary
imagePullPolicy: IfNotPresent #更改镜像下载策略；只要本地有镜像，就不进行更新
# 修改配置文件：influxdb-grafana-controller.yaml
spec:
nodeName: k8s-node-2
containers:
- name: influxdb
image: 10.0.0.11:5000/heapster_influxdb:v0.5
# 查看所有与Namespace中的Pod资源
[root@k8s-master heapster]# kubectl get pod --all-namespaces -o wide
# 创建资源
[root@k8s-master heapster]# kubectl create -f /root/k8s_yaml/heapster/.
service "monitoring-grafana" created
replicationcontroller "heapster" created
service "heapster" created
replicationcontroller "influxdb-grafana" created
service "monitoring-influxdb" created
4:打开dashboard验证
5.2 弹性伸缩
1:在deploy目录中创建弹性伸缩RC的配置文件
[root@k8s-master heapster]# cd /root/k8s_yaml/deploy
[root@k8s-master deploy]# vim k8s_deploy_nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
requests:
cpu: 100m
# 创建deploy_nginx资源
[root@k8s-master deploy]# kubectl create -f k8s_deploy_nginx.yaml
deployment "nginx" created
# 获取Pod的IP地址
[root@k8s-master deploy]# kubectl get pod -o wide |grep nginx
nginx-2807576163-pkgp9 1/1 Running 0 1m 172.18.37.9 k8s-node-1
# 查看资源(这里只截取了deploy_nginx资源)
[root@k8s-master deploy]# kubectl get all -o wide
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx 1 1 1 1 8m
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
hpa/nginx Deployment/nginx 10% 0% 1 8 4m
NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
rc/readiness 3 3 0 4h readiness 10.0.0.11:5000/nginx:1.15 app=readiness
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
svc/kubernetes 10.254.0.1 <none> 443/TCP 3d <none>
svc/readiness 10.254.227.196 <nodes> 80:32089/TCP 4h app=readiness
NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
rs/nginx-2807576163 1 1 1 8m nginx 10.0.0.11:5000/nginx:1.13 app=nginx,pod-template-hash=2807576163
NAME READY STATUS RESTARTS AGE IP NODE
po/nginx-2807576163-pkgp9 1/1 Running 0 8m 172.18.37.9 k8s-node-1
[root@k8s-master deploy]# kubectl get hpa nginx
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
nginx Deployment/nginx 10% 0% 1 8 1m
2:创建弹性伸缩规则
# 创建SVC,与当前deploy_nginx资源进行关联
[root@k8s-master deploy]# kubectl expose deployment nginx --type=NodePort --port=80
service "nginx" exposed
# 查看SVC资源；记录SVC：nginx 的随机端口；作为后面压测时使用
# 启用SVC是为了，当压力过大，Pod水平扩展时，SVC会做负载，压测时流量会相对平均分配
[root@k8s-master deploy]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 3d
nginx 10.254.249.86 <nodes> 80:30138/TCP 11s
readiness 10.254.227.196 <nodes> 80:32089/TCP 4h
[root@k8s-master deploy]# kubectl autoscale deploy nginx --max=8 --min=1 --cpu-percent=10
deployment "nginx" autoscaled
3:压力测试
# 安装httpd-tools工具，进行压力测试
[root@k8s-master deploy]# yum install -y httpd-tools
# 查看“ab”命令是否存在
[root@k8s-master deploy]# which ab
/usr/bin/ab
# 压力测试
[root@k8s-master deploy]# ab -n 1000000 -c 40 http://10.0.0.12:30138/index.html
# 注意：
当压力减小时，水平扩展的Pod默认是五分钟后才会根据实际情况进行水平缩容；运行时间越少，将会优先删除
扩容截图
缩容:
6:持久化存储
前面我们部署的Mysql+Tomcat，是没有做持久化存储的，当Mysql_Pod重启后，之前添加的数据将丢失不存在。
数据持久化类型:
6.1 emptyDir：
emptryDir，顾名思义是一个空目录，它的生命周期和所属的 Pod 是完全一致的。emptyDir 类型的 Volume 在 Pod 分配到 Node 上时会被创建，Kubernetes 会在 Node 上自动分配一个目录，因此无需指定 Node 宿主机上对应的目录文件。这个目录的初始内容为空，当 Pod 从 Node 上移除（Pod 被删除或者 Pod 发生迁移）时，emptyDir 中的数据会被永久删除。
emptyDir Volume 主要用于某些应用程序无需永久保存的临时目录，在多个容器之间共享数据等。缺省情况下，emptryDir 是使用主机磁盘进行存储的。你也可以使用其它介质作为存储，比如：网络存储、内存等。设置 emptyDir.medium 字段的值为 Memory 就可以使用内存进行存储，使用内存做为存储可以提高整体速度，但是要注意一旦机器重启，内容就会被清空，并且也会受到容器内存的限制。
emptyDir类型的volume在pod分配到node上时被创建，kubernetes会在node上自动分配 一个目录，因此无需指定宿主机node上对应的目录文件。这个目录的初始内容为空，当Pod从node上移除时，emptyDir中的数据会被永久删除。
emptyDir Volume主要用于某些应用程序无需永久保存的临时目录，多个容器的共享目录等。
[root@k8s-master deploy]# cd /root/k8s_yaml/tomcat_demo/
# 创建mysql-rc-V2.yaml文件；添加“emptyDir”持久化机制；后面创建的Mysql-Pod资源使用这个文件
[root@k8s-master tomcat_demo]# vim mysql-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
namespace: tomcat
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: mysql
emptyDir: {}
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
# 删除旧的Namespace：Tomcat中所有资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 将旧版本mysql-rc.yaml文件移动到下一级目录中；不影响后面的批量创建资源
[root@k8s-master tomcat_demo]# mv mysql-rc.yaml bak_pv/
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看创建的资源Tomcat：Pod运行的节点
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-c2v1j 1/1 Running 0 26s 172.18.37.7 k8s-node-1
myweb-0lzls 1/1 Running 0 26s 172.18.37.2 k8s-node-1
# 访问Web_URL，插入一条数据：Name=Chiron Level=10000
http://10.0.0.13:30008/demo/index.jsp
# 在“Node1节点”删除刚刚创建的Pod资源
[root@k8s-node-1 images]# docker rm -f `docker ps |grep my`
9213f64fe0f2
ffd769cd457b
7b05a7711987
861e6f27af6e
# 等待RC新起Pod,然后访问Web_URL，插入的数据依然存在
http://10.0.0.13:30008/demo/index.jsp
# 缺点：不能删除Mysql-Pod,如果删除Pod，数据将丢失
[root@k8s-master tomcat_demo]# kubectl delete pod -n tomcat mysql-c2v1j
pod "mysql-c2v1j" deleted
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-vn36w 1/1 Running 0 27s 172.18.37.10 k8s-node-1
myweb-0lzls 1/1 Running 0 14m 172.18.37.2 k8s-node-1
# 等待Mysql新起Pod,然后访问Web_URL，插入的数据已经丢失
http://10.0.0.13:30008/demo/index.jsp
# 在“Node1节点”查看Mysql-Pod的volumes-mysql数据目录
[root@k8s-node-1 images]# find / -name "mysql" |grep volumes
/var/lib/kubelet/pods/57970a59-1250-11ea-b36a-000c29c6d194/volumes/kubernetes.io~empty-dir/mysql
/var/lib/kubelet/pods/57970a59-1250-11ea-b36a-000c29c6d194/volumes/kubernetes.io~empty-dir/mysql/mysql
# 进入mysql数据目录,并查看目录下有哪些东西
[root@k8s-node-1 images]# cd /var/lib/kubelet/pods/57970a59-1250-11ea-b36a-000c29c6d194/volumes/kubernetes.io~empty-dir/mysql && ls
auto.cnf HPE_APP ib_buffer_pool ibdata1 ib_logfile0 ib_logfile1 ibtmp1 mysql performance_schema sys
# 我们访问tomcat时，数据库是“HPE_APP”，查看它的下面有哪些表
[root@k8s-node-1 mysql]# ls HPE_APP/
db.opt T_USERS.frm T_USERS.ibd
6.2 HostPath：
hostPath Volume为pod挂载宿主机上的目录或文件，使得容器可以使用宿主机的高速文件系统进行存储。
缺点：
在k8s中，pod都是动态在各node节点上调度。当一个pod在当前node节点上启动并通过hostPath存储了文件到本地以后，下次调度到另一个节点上启动时，就无法使用在之前节点上存储的文件。
所以使用“HostPath”方式做持久化时，不必须固定Pod资源运行在哪个节点。
# 删除旧的“Mysql+Tomcat”资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 移动“mysql-rc-V2.yaml”文件到“bak_pv”目录下
[root@k8s-master tomcat_demo]# mv mysql-rc-V2.yaml /root/k8s_yaml/tomcat_demo/bak_pv/
# 创建mysql-rc-V3.yaml文件
[root@k8s-master tomcat_demo]# vim mysql-rc-V3.yaml
apiVersion: v1
kind: ReplicationController
metadata:
namespace: tomcat
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: mysql
hostPath:
path: /data/tomcat_mysql
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看Pod被分配到了哪个节点
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-hc0h2 1/1 Running 0 3m 172.18.37.7 k8s-node-1
myweb-q7d2m 1/1 Running 0 3m 172.18.37.2 k8s-node-1
# 访问Web_URL，触发Tomcat页面；插入一条数据：Name=Chiron Level=10000
http://10.0.0.13:30008/demo/index.jsp
# 在Node1节点查看我们挂载的目录是否存有数据
[root@k8s-node-1 mysql]# cd /data/tomcat_mysql/ && ls
auto.cnf ib_buffer_pool ib_logfile0 ibtmp1 performance_schema
HPE_APP ibdata1 ib_logfile1 mysql sys
# 删除“Mysql-Pod”；待重新运行一个新的后，查看之前的数据是否依然存在
[root@k8s-master tomcat_demo]# kubectl delete pod -n tomcat mysql-hc0h2
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-sp6sg 1/1 Running 0 4s 172.18.37.10 k8s-node-1
myweb-q7d2m 1/1 Running 0 10m 172.18.37.2 k8s-node-1
# 等待Mysql新起Pod,然后访问Web_URL，插入的数据依然存在
http://10.0.0.13:30008/demo/index.jsp
# 注意：
只要再次创建的Pod没有被分配到其他节点，数据将始终存在；
再次创建的Pod如果被分配到了其他节点，数据将丢失。
6.3 nfs:
NFS是网络文件系统Network File System的缩写，NFS服务器可以让PC将网络中的NFS服务器共享的目录挂载到本地的文件系统中，而在本地的系统中来看，那个远程主机的目录就好像是自己的一个磁盘分区一样。
Kubernetes使用NFS共享存储有两种方式：
1.手动方式静态创建所需要的PV和PVC；
2.通过创建PVC动态地创建对应PV，无需手动创建PV。
# 所有节点安装NFS工具
yum install nfs-utils -y
# Master节点建立存放持久化数据的目录；后面会在Mysql_RC的Yaml文件中用到
[root@k8s-master tomcat_demo]# mkdir /data
# Master节点作为服务端；更改NFS配置文件，共享哪个目录，允许谁访问，目录权限，同步方式；不做任何用户映射
[root@k8s-master tomcat_demo]# cat <>/etc/exports
> /data 10.0.0.0/24(rw,async,no_all_squash,no_root_squash)
> EOF
[root@k8s-master tomcat_demo]# more /etc/exports
/data 10.0.0.0/24(rw,async,no_all_squash,no_root_squash)
# 启动并查看 NFS相关服务
[root@k8s-master tomcat_demo]# systemctl restart rpcbind nfs
[root@k8s-master tomcat_demo]# systemctl enable rpcbind nfs
[root@k8s-master tomcat_demo]# systemctl status rpcbind nfs
# 在Node节点验证是否能查看到共享的目录
[root@k8s-node-1 tomcat_mysql]# showmount -e 10.0.0.11
Export list for 10.0.0.11:
/data 10.0.0.0/24
[root@k8s-node-2 images]# showmount -e 10.0.0.11
Export list for 10.0.0.11:
/data 10.0.0.0/24
# 删除旧的“Mysql+Tomcat”资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 移动“mysql-rc-V3.yaml”文件到“bak_pv”目录下
[root@k8s-master tomcat_demo]# mv mysql-rc-V3.yaml /root/k8s_yaml/tomcat_demo/bak_pv/
# 创建mysql-rc-V4.yaml文件
[root@k8s-master tomcat_demo]# vim mysql-rc-V4.yaml
apiVersion: v1
kind: ReplicationController
metadata:
namespace: tomcat
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: mysql
nfs:
path: /data/tomcat_mysql
server: 10.0.0.11
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看Pod被分配到了哪个节点
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-3rxjv 1/1 Running 0 6s 172.18.37.7 k8s-node-1
myweb-vcd21 1/1 Running 0 6s 172.18.37.2 k8s-node-1
# 访问Web_URL，触发Tomcat页面；插入一条数据：Name=Chiron Level=10000
http://10.0.0.13:30008/demo/index.jsp
# 在“Node1节点”查看是否有Mount挂载记录
[root@k8s-node-1 tomcat_mysql]# df -Th |grep nfs
10.0.0.11:/data/tomcat_mysql nfs4 48G 3.5G 45G 8% /var/lib/kubelet/pods/169c35cb-128e-11ea-b36a-000c29c6d194/volumes/kubernetes.io~nfs/mysql
# 在Master节点查看本地共享目录下，是否存在数据
[root@k8s-master tomcat_demo]# ls /data/tomcat_mysql/
auto.cnf HPE_APP ib_buffer_pool ibdata1 ib_logfile0 ib_logfile1 ibtmp1 mysql performance_schema sys
# 修改“mysql-rc-V4.yaml”文件，指定在“Node2节点”运行Mysql-Pod资源，方便后面验证数据依然存在
[root@k8s-master tomcat_demo]# vim mysql-rc-V4.yaml
spec:
nodeName: k8s-node-2 #指定运行节点
volumes:
- name: mysql
nfs:
path: /data/tomcat_mysql
server: 10.0.0.11
# 批量删除“Mysql+Tomcat”资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看Tomcat创建的资源中Pod：Mysql是否运行在“Node2”节点中
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-f4npf 1/1 Running 0 5s 172.18.95.10 k8s-node-2
myweb-qlr3p 1/1 Running 0 5s 172.18.37.2 k8s-node-1
# 访问Web_URL，验证插入的数据依然存在
http://10.0.0.13:30008/demo/index.jsp
# 在“Node2节点”查看是否有Mount挂载记录；挂载的为同一份数据
[root@k8s-node-2 images]# df -Th |grep nfs
10.0.0.11:/data/tomcat_mysql nfs4 48G 3.5G 45G 8% /var/lib/kubelet/pods/b3083a2b-128f-11ea-b36a-000c29c6d194/volumes/kubernetes.io~nfs/mysql
# 注意：
存在单点故障；
当磁盘空间不足，无法扩容
6.4 pvc:
pv: persistent volume 全局资源,k8s集群
pvc: persistent volume claim, 局部资源,属于某一个namespace
Pod --> PVC --> PV --> NFS
6.4.1:创建pv和pvc
上传yaml配置文件,创建pv和pvc
# 创建存放PV(C)相关Yaml文件的目录
[root@k8s-master tomcat_demo]# mkdir /root/k8s_yaml/volume && cd /root/k8s_yaml/volume
# 将之前随Mysql+Tomcat一起上传的PV(C)相关Yaml文件复制到本地
[root@k8s-master volume]# cp /root/k8s_yaml/tomcat_demo/bak_pv/mysql_pv* /root/k8s_yaml/volume/ && ls
mysql_pvc.yaml mysql_pv.yaml
# 创建单独存放PV数据的相关目录
[root@k8s-master volume]# mkdir /data/pv
# 创建"mysql_PV"资源
[root@k8s-master volume]# kubectl create -f mysql_pv.yaml
persistentvolume "tomcat-mysql" created
# 验证"PV"属于全局资源，无论查看PV资源是否指定Namespace都可以查看到
[root@k8s-master volume]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
tomcat-mysql 10Gi RWX Recycle Available 1m
[root@k8s-master volume]# kubectl get pv -n tomcat
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
tomcat-mysql 10Gi RWX Recycle Available 1m
# 并且在Dashboard中也能查看到创建的PV资源
http://10.0.0.11:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/persistentvolume?namespace=_all
# 再创建一个“PV2”;后面会根据这个设置它的挑选机制
[root@k8s-master volume]# cp mysql_pv.yaml mysql_pv2.yaml
[root@k8s-master volume]# vim mysql_pv2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: tomcat-mysql2 #名称指定为“PV2”
labels:
type: nfs001
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
nfs:
path: "/data/pv2" #数据存放在“PV2”中
server: 10.0.0.11
readOnly: false
# 创建“PV2”的数据存储目录
[root@k8s-master volume]# mkdir /data/pv2
# 创建"mysql_PV2"资源
[root@k8s-master volume]# kubectl create -f mysql_pv2.yaml
persistentvolume "tomcat-mysql2" created
# 查看所有“PV”资源
[root@k8s-master volume]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
tomcat-mysql 10Gi RWX Recycle Available 25m
tomcat-mysql2 50Gi RWX Recycle Available 26s
# 修改PVC文件；指定挑选机制
[root@k8s-master volume]# vim mysql_pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: tomcat-mysql
namespace: tomcat
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30Gi
# 创建“PVC”资源
[root@k8s-master volume]# kubectl create -f mysql_pvc.yaml
persistentvolumeclaim "tomcat-mysql" created
# 再次查看“PV”资源，发现符合机制的PV：“tomcat-mysql2”被分配给了“tomcat/tomcat-mysql”
[root@k8s-master volume]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
tomcat-mysql 10Gi RWX Recycle Available 33m
tomcat-mysql2 50Gi RWX Recycle Bound tomcat/tomcat-mysql 8m
# 验证“PVC”为局部资源，必须指定查看的“Namespace”
[root@k8s-master volume]# kubectl get pvc -n tomcat
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
tomcat-mysql Bound tomcat-mysql2 50Gi RWX 2m
[root@k8s-master volume]# kubectl get pvc
No resources found.
6.4.2:创建mysql-rc,pod模板里使用volume：PVC
[root@k8s-master volume]# cd /root/k8s_yaml/tomcat_demo/
# 移动“mysql-rc-V4.yaml”文件到“bak_pv”目录下
[root@k8s-master tomcat_demo]# mv mysql-rc-V4.yaml /root/k8s_yaml/tomcat_demo/bak_pv/
# 创建mysql-rc-V5.yaml文件
[root@k8s-master tomcat_demo]# vim mysql-rc-V5.yaml
apiVersion: v1
kind: ReplicationController
metadata:
namespace: tomcat
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
nodeName: k8s-node-2
volumes:
- name: mysql
persistentVolumeClaim:
claimName: tomcat-mysql
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
# 批量删除“Mysql+Tomcat”资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看Pod被分配到了哪个节点
[root@k8s-master tomcat_demo]# kubectl get pod -o wide -n tomcat
NAME READY STATUS RESTARTS AGE IP NODE
mysql-lfrnc 1/1 Running 0 6s 172.18.95.9 k8s-node-2
myweb-b6dvc 1/1 Running 0 6s 172.18.37.3 k8s-node-1
# 访问Web_URL，触发Tomcat页面；插入一条数据：Name=Chiron Level=10000
http://10.0.0.13:30008/demo/index.jsp
6.4.3: 验证持久化
验证方法1:删除mysql的pod,数据库不丢
[root@k8s-master tomcat_demo]# kubectl delete -n tomcat pod mysql-lfrnc
pod "mysql-lfrnc" deleted
[root@k8s-master tomcat_demo]# kubectl get pod -n tomcat -o wide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-0kcf4 1/1 Running 0 14s 172.18.95.10 k8s-node-2
myweb-b6dvc 1/1 Running 0 7m 172.18.37.3 k8s-node-1
# 待Pod：Mysql重新运行后；查看到插入的数据依然存在
http://10.0.0.13:30008/demo/index.jsp
验证方法2:查看nfs服务端,是否有mysql的数据文件
# 查看验证指定的“PVC”下的“/data/pv2/”目录中是否有数据
[root@k8s-master tomcat_demo]# ls /data/pv2/
auto.cnf HPE_APP ib_buffer_pool ibdata1 ib_logfile0 ib_logfile1 ibtmp1 mysql performance_schema sys
6.5: 分布式存储glusterfs
a: 什么是glusterfs
Glusterfs是一个开源分布式文件系统，具有强大的横向扩展能力，可支持数PB级存储容量和数千客户端，通过网络互联成一个并行的网络文件系统。具有可扩展性、高性能、高可用性等特点。
b: 安装glusterfs
# 所有节点安装glusterfs：
yum install centos-release-gluster -y
yum install install glusterfs-server -y
systemctl start glusterd.service
systemctl enable glusterd.service
# 为gluster集群每个节点增加两个存储单元brick
mkdir -p /gfs/test1 /gfs/test2
c: 添加存储资源池
# master节点：
# 查看资源池节点列表
[root@k8s-master tomcat_demo]# gluster pool list
UUID Hostname State
cd5fe643-be60-45ed-8e10-64f11592b829 localhost Connected
# 加入类外两个节点到资源池
[root@k8s-master tomcat_demo]# gluster peer probe k8s-node-1
peer probe: success.
[root@k8s-master tomcat_demo]# gluster peer probe k8s-node-2
peer probe: success.
# 再次查看资源池节点列表
[root@k8s-master tomcat_demo]# gluster pool list
UUID Hostname State
da21e75e-aacd-42bd-8f8d-9e0307280606 k8s-node-1 Connected
b90e3f52-956e-4769-80d6-cbd7ebe3f70e k8s-node-2 Connected
cd5fe643-be60-45ed-8e10-64f11592b829 localhost Connected
d: glusterfs卷管理
#创建分布式复制卷:副本数2;不指定默认创建分布式卷
[root@k8s-master tomcat_demo]# gluster volume create Chiron replica 2 k8s-master:/gfs/test1 k8s-node-1:/gfs/test1 k8s-master:/gfs/test2 k8s-node-1:/gfs/test2 force
volume create: Chiron: success: please start the volume to access data
# 参数解释：
Chiron：卷名称
replica 2：指定副本数
force：因为官方建议副本数为"3"，加上这个参数，忽略强制创建
# 启动卷
[root@k8s-master tomcat_demo]# gluster volume start Chiron
# 查看卷
[root@k8s-master tomcat_demo]# gluster volume info Chiron
# 查看卷列表
[root@k8s-master tomcat_demo]# gluster volume list
# 所有Node节点挂载卷；其实可以挂载任何一个节点的卷
[root@k8s-node]# mount -t glusterfs 10.0.0.11:/Chiron /mnt
# 所有Node节点查看挂载
[root@k8s-node]# df -Th |grep Chiron
10.0.0.11:/Chiron fuse.glusterfs 48G 5.6G 43G 12% /mnt
# 在"Node3节点"向“/mnt”目录导入一些数据并查看里面文件与目录数量
[root@k8s-node-2 images]# cp -a /etc/rc* /mnt/
[root@k8s-node-2 images]# tree /mnt/ |tail -n 1
16 directories, 20 files
# 在"Master节点"分别查看"/gfs/test1/"与"/gfs/test2/"目录下文件与目录数量
# 想查看完整数据，需要将卷挂载到一个挂载点钟进行查看
[root@k8s-master tomcat_demo]# tree /gfs/test1 |tail -n 1
12 directories, 8 files
[root@k8s-master tomcat_demo]# tree /gfs/test2 |tail -n 1
13 directories, 12 files
e: 分布式复制卷讲解
f: 分布式复制卷扩容
# 扩容前查看容量：
[root@k8s-node-2 images]# df -h |grep mnt
10.0.0.11:/Chiron 48G 5.6G 43G 12% /mnt
# 扩容命令：
[root@k8s-node-2 images]# gluster volume add-brick Chiron k8s-node-2:/gfs/test1 k8s-node-2:/gfs/test2 force
volume add-brick: success
# 扩容后查看容量:
[root@k8s-node-2 images]# df -h |grep mnt
10.0.0.11:/Chiron 72G 9.3G 63G 13% /mnt
# 查看“/gfs/test”目录下文件与目录数量；只是放进来了目录，并没有文件
[root@k8s-node-2 images]# tree /gfs/test1/ |tail -n 1
9 directories, 0 files
[root@k8s-node-2 images]# tree /gfs/test2/ |tail -n 1
9 directories, 0 files
# 新增存储单元后，将数据均衡到新的存储单元
# 均衡数据会造成磁盘I/O过高，建议在访问量比较小时操作
[root@k8s-node-2 images]# gluster volume rebalance Chiron start
volume rebalance: Chiron: success: Rebalance on Chiron has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 42052f66-c0ed-41fd-ab13-805823d3ebd0
# 再次“tree”查看时，数据已经均衡到了新增存储单元
[root@k8s-node-2 images]# tree /gfs/test2 |tail -n 1
11 directories, 6 files
[root@k8s-node-2 images]# tree /gfs/test1 |tail -n 1
11 directories, 6 files
6.6 k8s 对接glusterfs存储
扩展：
endpoint资源可以理解为与SVC资源同名
[root@k8s-master tomcat_demo]# kubectl get endpoints -n tomcat
NAME ENDPOINTS AGE
mysql 172.18.95.10:3306 5h
myweb 172.18.37.3:8080 5h
[root@k8s-master tomcat_demo]# kubectl get svc -n tomcat
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql 10.254.84.147 <none> 3306/TCP 5h
myweb 10.254.229.158 <nodes> 8080:30008/TCP 5h
[root@k8s-master tomcat_demo]# kubectl get pod -n tomcat -o wide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-0kcf4 1/1 Running 0 5h 172.18.95.10 k8s-node-2
myweb-b6dvc 1/1 Running 0 5h 172.18.37.3 k8s-node-1
# 创建属于glusterfs相关的yaml文件的存放目录
[root@k8s-master tomcat_demo]# mkdir /root/k8s_yaml/glusterfs && cd /root/k8s_yaml/glusterfs
[root@k8s-master glusterfs]# vim glusterfs-ep.yaml
a：创建endpoint
apiVersion: v1
kind: Endpoints
metadata:
name: glusterfs
namespace: tomcat
subsets:
- addresses:
- ip: 10.0.0.11
- ip: 10.0.0.12
- ip: 10.0.0.13
ports:
- port: 49152
protocol: TCP
[root@k8s-master glusterfs]# kubectl create -f glusterfs-ep.yaml
endpoints "glusterfs" created
[root@k8s-master glusterfs]# kubectl get endpoints -n tomcat
NAME ENDPOINTS AGE
glusterfs 10.0.0.11:49152,10.0.0.12:49152,10.0.0.13:49152 12s
mysql 172.18.95.10:3306 5h
myweb 172.18.37.3:8080 5h
# 还缺少“glusterfs”的SVC做负载均衡
[root@k8s-master glusterfs]# kubectl get svc -n tomcat
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql 10.254.84.147 <none> 3306/TCP 5h
myweb 10.254.229.158 <nodes> 8080:30008/TCP 5h
b: 创建service：glusterfs-svc.yaml
[root@k8s-master glusterfs]# kubectl describe svc mysql -n tomcat
Name: mysql
Namespace: tomcat #必须和“glusterfs-svc.yaml”中的“namespace”一致，否则无法进行关联
Labels: <none>
Selector: app=mysql
Type: ClusterIP
IP: 10.254.84.147
Port: <unset> 3306/TCP
Endpoints: 172.18.95.10:3306
Session Affinity: None
No events.
[root@k8s-master glusterfs]# vim glusterfs-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: glusterfs
namespace: tomcat #必须与SVC:mysql中的“namespace”一致，否则无法进行关联
spec:
ports:
- port: 49152
protocol: TCP
targetPort: 49152
sessionAffinity: None
type: ClusterIP
[root@k8s-master glusterfs]# kubectl create -f glusterfs-svc.yaml
service "glusterfs" created
[root@k8s-master glusterfs]# kubectl get svc -n tomcat
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs 10.254.93.25 <none> 49152/TCP 6s
mysql 10.254.84.147 <none> 3306/TCP 6h
myweb 10.254.229.158 <nodes> 8080:30008/TCP 6h
[root@k8s-master glusterfs]# kubectl describe svc glusterfs -n tomcat
Name: glusterfs
Namespace: tomcat
Labels: <none>
Selector: <none>
Type: ClusterIP
IP: 10.254.93.25
Port: <unset> 49152/TCP
Endpoints: 10.0.0.11:49152,10.0.0.12:49152,10.0.0.13:49152
Session Affinity: None
No events.
[root@k8s-master glusterfs]# cd /root/k8s_yaml/tomcat_demo/
# 删除旧的“Mysql+Tomcat”资源
[root@k8s-master tomcat_demo]# kubectl delete -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" deleted
service "mysql" deleted
replicationcontroller "myweb" deleted
service "myweb" deleted
# 移动“mysql-rc-V5.yaml”文件到“bak_pv”目录下
[root@k8s-master tomcat_demo]# mv mysql-rc-V5.yaml /root/k8s_yaml/tomcat_demo/bak_pv/
# 创建mysql-rc-V6.yaml文件
[root@k8s-master tomcat_demo]# vim mysql-rc-V6.yaml
apiVersion: v1
kind: ReplicationController
metadata:
namespace: tomcat
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
nodeName: k8s-node-2
volumes:
- name: mysql
glusterfs:
endpoints: glusterfs
path: /Chiron
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
# 清空前边测试用的数据；否则后面启动Pod:mysql会失败：初始化时，目录中含有数据
[root@k8s-node-2 images]# ls /mnt/
rc0.d rc1.d rc2.d rc3.d rc4.d rc5.d rc6.d rc.d rc.local
[root@k8s-node-2 images]# rm -rf /mnt/*
# 批量创建Mysql+Tomcat资源
[root@k8s-master tomcat_demo]# kubectl create -f /root/k8s_yaml/tomcat_demo/.
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "myweb" created
service "myweb" created
# 查看创建的Namespace：tomcat中所有资源
[root@k8s-master tomcat_demo]# kubectl get all -n tomcat
NAME DESIRED CURRENT READY AGE
rc/mysql 1 1 1 12s
rc/myweb 1 1 1 12s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/glusterfs 10.254.93.25 <none> 49152/TCP 18m
svc/mysql 10.254.204.230 <none> 3306/TCP 12s
svc/myweb 10.254.106.148 <nodes> 8080:30008/TCP 12s
NAME READY STATUS RESTARTS AGE
po/mysql-xxg78 1/1 Running 0 12s
po/myweb-582tl 1/1 Running 0 12s
# 访问Web_URL，触发Tomcat页面；插入一条数据：Name=Chiron Level=10000
http://10.0.0.13:30008/demo/index.jsp
# 在"glusterfs:Chiron"挂载出的"mnt目录"查看我们挂载的目录已经生成数据
[root@k8s-node-2 images]# ls /mnt/
auto.cnf HPE_APP ib_buffer_pool ibdata1 ib_logfile0 ib_logfile1 ibtmp1 mysql performance_schema sys
扩展(不做演示)：
c: 创建gluster类型pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster
labels:
type: glusterfs
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs"
path: "Chiron"
readOnly: false
d: 创建pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: gluster
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 200Gi
e：在pod中使用gluster
vi nginx_pod.yaml
……
volumeMounts:
- name: nfs-vol2
mountPath: /usr/share/nginx/html
volumes:
- name: nfs-vol2
persistentVolumeClaim:
claimName: gluster
7:与jenkins集成实现cd
| ip地址 | 服务 | 内存 |
|---|---|---|
| 10.0.0.11 | kube-apiserver 8080 | 2G |
| 10.0.0.12 | jenkins(tomcat + jdk) 8080 | 2G |
| 借助码云Gitee | gitlab 8080,80 |
7. 1 :将本地代码上传到码云Gitee的仓库中
创建“码云Gitee”存储仓库"K8S_yiliao"；创建时，将初始化仓库勾选取消，其他地方保持默认即可
# 本地仓库在"Node2节点"中
# Git 全局设置:
[root@k8s-node-2 images]# git config --global user.name "ChironW"
[root@k8s-node-2 images]# git config --global user.email "[email protected]"
# 创建 git 仓库:
[root@k8s-node-2 images]# mkdir /opt/gitee && cd /opt/gitee
[root@k8s-node-2 gitee]# wget http://192.168.2.254:8888/yiliaoqixie.zip
[root@k8s-node-2 gitee]# unzip yiliaoqixie.zip
[root@k8s-node-2 gitee]# cd yiliaoqixie/
# 初始化仓库
[root@k8s-node-2 yiliaoqixie]# git init
Initialized empty Git repository in /opt/gitee/yiliaoqixie/.git/
# 提交到本地仓库
[root@k8s-node-2 yiliaoqixie]# git add /opt/gitee/yiliaoqixie/.
[root@k8s-node-2 yiliaoqixie]# git commit -m "第一版"
# 由本地仓库推送到远端仓库
[root@k8s-node-2 yiliaoqixie]# git remote add origin [email protected]:chironW/K8S_yiliao.git
[root@k8s-node-2 yiliaoqixie]# git push -u origin master
The authenticity of host 'gitee.com (212.64.62.174)' can't be established.
ECDSA key fingerprint is SHA256:FQGC9Kn/eye1W8icdBgrQp+KkGYoFgbVr17bmjey0Wc.
ECDSA key fingerprint is MD5:27:e5:d3:f7:2a:9e:eb:6c:93:cd:1f:c1:47:a3:54:b1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitee.com,212.64.62.174' (ECDSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
# 报错原因：权限被拒绝；远端没有配置“SSH-publickey”
# 本地主机生成公钥，将获取的公钥添加到“Gitee”的设置“SSH公钥”中
[root@k8s-node-2 yiliaoqixie]# ssh-keygen -t rsa
[root@k8s-node-2 yiliaoqixie]# cat /root/.ssh/id_rsa.pub
再次推送本地代码
[root@k8s-node-2 yiliaoqixie]# git push -u origin master
7.2 :安装jenkins,并自动构建docker镜像
1:安装jenkins
# “k8s-node-1节点”安装部署Jenkins：
[root@k8s-node-1 images]# cd /usr/local/src/
[root@k8s-node-1 src]# wget http://192.168.2.254:8888/apache-tomcat-8.0.27.tar.gz
[root@k8s-node-1 src]# wget http://192.168.2.254:8888/jenkin-data.tar.gz
[root@k8s-node-1 src]# wget http://192.168.2.254:8888/jdk-8u102-linux-x64.rpm
[root@k8s-node-1 src]# wget http://192.168.2.254:8888/jenkins.war
# 安装JDK
[root@k8s-node-1 src]# rpm -ivh jdk-8u102-linux-x64.rpm
# 解压部署Tomcat
[root@k8s-node-1 src]# mkdir /usr/local/src/tomcat
[root@k8s-node-1 src]# tar xf apache-tomcat-8.0.27.tar.gz -C tomcat
[root@k8s-node-1 src]# mv tomcat/apache-tomcat-8.0.27/* /usr/local/src/tomcat/
[root@k8s-node-1 src]# rm -rf tomcat/apache-tomcat-8.0.27/ /usr/local/src/tomcat/webapps/*
# 部署Jenkins
[root@k8s-node-1 src]# cd /usr/local/src/tomcat/webapps/
[root@k8s-node-1 webapps]# mv /usr/local/src/jenkins.war /usr/local/src/tomcat/webapps/ROOT.war
# 这里将之前部署好的Jenkins归档包，进行解压即可，插件等已经配置完成
[root@k8s-node-1 webapps]# tar xf /usr/local/src/jenkin-data.tar.gz -C /root/
[root@k8s-node-1 webapps]# ls /root/.jenkins/
# 启动Tomcat
[root@k8s-node-1 webapps]# /usr/local/src/tomcat/bin/startup.sh
[root@k8s-node-1 webapps]# netstat -lntup |grep java
2:访问jenkins
访问：http://10.0.0.12:8080/
默认账号密码：admin/123456
3:编写Dockerfile，进行本地测试，然后推送到Git仓库中
# 编写Dockerfile文件
[root@k8s-node-2 yiliaoqixie]# vim dockerfile
FROM 10.0.0.11:5000/nginx:1.17
ADD . /usr/share/nginx/html
# Build测试镜像并运行查看
[root@k8s-node-2 yiliaoqixie]# docker build -t yiliao:v1 .
[root@k8s-node-2 yiliaoqixie]# docker run -d -P yiliao:v1
[root@k8s-node-2 yiliaoqixie]# docker ps -a -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afb48da6470b yiliao:v1 "nginx -g 'daemon ..." 20 seconds ago Up 19 seconds 0.0.0.0:32768->80/tcp festive_hypatia
# Web浏览器访问验证：
http://10.0.0.13:32768/
# 推送“dockerfile”文件到gitee仓库
[root@k8s-node-2 yiliaoqixie]# git add .
[root@k8s-node-2 yiliaoqixie]# git commit -m 'add dockerfile'
[root@k8s-node-2 yiliaoqixie]# git push -u origin master
4:配置jenkins拉取Git仓库代码凭据
a:在jenkins上生成秘钥对；将获取的公钥添加到“K8S_yiliao”仓库的管理-公钥中
[root@k8s-node-1 webapps]# ssh-keygen -t rsa
[root@k8s-node-1 webapps]# cat /root/.ssh/id_rsa.pub
b:将一起生成的私钥添加到Jenkins的全局凭据中；类型：SSH Username with private key
[root@k8s-node-1 webapps]# cat /root/.ssh/id_rsa
5:Jenkins创建自由风格的工程“yiliao”；拉取代码测试
Jenkins构建后；可以查看到拉取的代码
[root@k8s-node-1 webapps]# ls /root/.jenkins/workspace/yiliao/
6:更改Jenkins配置中执行的shell命令；使其创建Docker镜像，并推送到私有仓库
docker build -t 10.0.0.11:5000/yiliao:v1 .
docker push 10.0.0.11:5000/yiliao:v1
再次构建后，在私有仓库中查看是否有“yiliao:v1”的镜像
[root@k8s-master ~]# ls /opt/myregistry/docker/registry/v2/repositories/yiliao/_manifests/tags/
v1
7:Jenkins使用参数化构建,自动构建docker镜像并上传到私有仓库
Gitee仓库中直接修改index.html文件中的标题为“ChironW医疗器械有限公司”；添加提交信息为“修改了首页标题v2”
使用参数化构建（文本参数:version），指定构建时Build镜像的版本号：执行shell语句变为：
if [ -z $version ];then
echo "请输入版本！"
exit 4
fi
docker build -t 10.0.0.11:5000/yiliao:$version .
docker push 10.0.0.11:5000/yiliao:$version
再次构建；指定构建版本号为“v2”；在私有仓库中查看是否有“yiliao:v2”的镜像
[root@k8s-master ~]# ls /opt/myregistry/docker/registry/v2/repositories/yiliao/_manifests/tags/
v1 v2
7.3 手动部署实现“yiliao”的发布，升级，回滚；方面理解后面通过 Jenkins自动化部署应用
# Node节点执行“kubectl”命令方法：
[root@k8s-node-1 webapps]# kubectl -s 10.0.0.11:8080 get pod -n tomcat
# 发布
# Master节点部署“yiliao”相关资源
# 创建Namespace资源
[root@k8s-master ~]# kubectl create namespace yiliao
namespace "yiliao" created
# 创建Deployment资源
[root@k8s-master ~]# kubectl run -n yiliao yiliao --image=10.0.0.11:5000/yiliao:v1 --replicas=2 --record
deployment "yiliao" created
# 查看历史版本操作
[root@k8s-master ~]# kubectl rollout history -n yiliao deployment yiliao
deployments "yiliao"
REVISION CHANGE-CAUSE
1 kubectl run -n yiliao yiliao --image=10.0.0.11:5000/yiliao:v1 --replicas=2 --record
# 创建Service资源
[root@k8s-master ~]# kubectl expose -n yiliao deployment yiliao --port=80 --target-port=80 --type=NodePort
service "yiliao" exposed
# 查看"Namespace:yiliao"中的所有资源
[root@k8s-master ~]# kubectl get all -n yiliao
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/yiliao 2 2 2 2 1m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/yiliao 10.254.32.77 <nodes> 80:32175/TCP 35s
NAME DESIRED CURRENT READY AGE
rs/yiliao-4098334581 2 2 2 1m
NAME READY STATUS RESTARTS AGE
po/yiliao-4098334581-9148v 1/1 Running 0 1m
po/yiliao-4098334581-vd5j0 1/1 Running 0 1m
# Web浏览器访问验证：标题为“某某医疗”
http://10.0.0.12:32175/
http://10.0.0.13:32175/
# 升级
# 升级版本：
[root@k8s-master ~]# kubectl set image -n yiliao deployment yiliao yiliao=10.0.0.11:5000/yiliao:v2
deployment "yiliao" image updated
# 查看历史版本操作
[root@k8s-master ~]# kubectl rollout history -n yiliao deployment yiliao
deployments "yiliao"
REVISION CHANGE-CAUSE
1 kubectl run -n yiliao yiliao --image=10.0.0.11:5000/yiliao:v1 --replicas=2 --record
2 kubectl set image -n yiliao deployment yiliao yiliao=10.0.0.11:5000/yiliao:v2
# Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.12:32175/
http://10.0.0.13:32175/
# 回滚
# 回滚到上一个版本：
[root@k8s-master ~]# kubectl rollout undo -n yiliao deployment yiliao
deployment "yiliao" rolled back
# 查看历史版本操作
[root@k8s-master ~]# kubectl rollout history -n yiliao deployment yiliao
deployments "yiliao"
REVISION CHANGE-CAUSE
2 kubectl set image -n yiliao deployment yiliao yiliao=10.0.0.11:5000/yiliao:v2
3 kubectl run -n yiliao yiliao --image=10.0.0.11:5000/yiliao:v1 --replicas=2 --record
# Web浏览器访问验证：标题为“某某医疗”
http://10.0.0.12:32175/
http://10.0.0.13:32175/
# 回滚到指定版本“REVISION：2”
[root@k8s-master ~]# kubectl rollout undo -n yiliao deployment yiliao --to-revision=2
deployment "yiliao" rolled back
# 查看历史版本操作
[root@k8s-master ~]# kubectl rollout history -n yiliao deployment yiliao
deployments "yiliao"
REVISION CHANGE-CAUSE
3 kubectl run -n yiliao yiliao --image=10.0.0.11:5000/yiliao:v1 --replicas=2 --record
4 kubectl set image -n yiliao deployment yiliao yiliao=10.0.0.11:5000/yiliao:v2
# Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.12:32175/
http://10.0.0.13:32175/
# 删除相关"Namespace:yiliao"资源;方便后面演示Jenkins自动部署应用到K8S
[root@k8s-master ~]# kubectl delete namespace yiliao
namespace "yiliao" deleted
[root@k8s-master ~]# kubectl get all -n yiliao
No resources found.
7.4 jenkins自动部署应用到k8s
使用参数化构建（文本参数:version）发布与更新：
更改Jenkins工程中执行Shell为：
if [ -z $version ];then
echo "请输入版本！"
exit 4
fi
if [ -f /tmp/yiliao.lock ];then
docker build -t 10.0.0.11:5000/yiliao:$version .
docker push 10.0.0.11:5000/yiliao:$version
kubectl -s 10.0.0.11:8080 set image -n yiliao deploy yiliao yiliao=10.0.0.11:5000/yiliao:$version
port=`kubectl -s 10.0.0.11:8080 get svc -n yiliao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
echo "你的项目地址访问是http://10.0.0.12:$port"
echo "更新成功"
else
docker build -t 10.0.0.11:5000/yiliao:$version .
docker push 10.0.0.11:5000/yiliao:$version
kubectl -s 10.0.0.11:8080 create namespace yiliao
kubectl -s 10.0.0.11:8080 run yiliao -n yiliao --image=10.0.0.11:5000/yiliao:$version --replicas=3 --record
kubectl -s 10.0.0.11:8080 expose -n yiliao deployment yiliao --port=80 --type=NodePort
port=`kubectl -s 10.0.0.11:8080 get svc -n yiliao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
echo "你的项目地址访问是http://10.0.0.12:$port"
echo "发布成功"
touch /tmp/yiliao.lock
chattr +i /tmp/yiliao.lock
fi
# 发布
# 再次构建；指定构建版本号为“v1”；
# Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
# 升级
# Gitee中直接修改index.html文件中的标题为“Chiron-2医疗器械有限公司”；添加提交信息为“修改了首页标题v3”
# 再次构建；指定构建版本号为“v2”；
# Web浏览器访问验证：标题为“ChironW-2医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
回滚版本：
# Jenkins自动化回滚
# 回滚到上一版本
# Jenkins创建自由风格的工程“yiliao_undo”；执行Shell语句为：（未做参数化构建默认回滚上一版本）
kubectl -s 10.0.0.11:8080 rollout undo -n yiliao deployment yiliao
port=`kubectl -s 10.0.0.11:8080 get svc -n yiliao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
echo "你的项目地址访问是http://10.0.0.12:$port"
echo "回滚成功"
# 构建工程“yiliao_undo”
# Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
# 回滚到指定版本
# Jenkins使用参数化构建回滚到指定版本；执行Shell语句为：（参数化构建回滚到指定版本）
if [ -z $version ];then
echo "未输入回滚版本号，默认回滚到上一版本！"
kubectl -s 10.0.0.11:8080 rollout undo -n yiliao deployment yiliao
port=`kubectl -s 10.0.0.11:8080 get svc -n yiliao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
echo "你的项目地址访问是http://10.0.0.12:$port"
echo "回滚上一版本成功"
else
echo "回滚到指定版本:$REVISION！"
kubectl -s 10.0.0.11:8080 rollout undo -n yiliao deployment yiliao --to-revision=$REVISION
port=`kubectl -s 10.0.0.11:8080 get svc -n yiliao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
echo "你的项目地址访问是http://10.0.0.12:$port"
echo "回滚到指定版本:$REVISION成功"
fi
echo "你的所有历史版本如下"
kubectl -s 10.0.0.11:8080 rollout history -n yiliao deployment yiliao
# 构建工程“yiliao_undo”；不填写构建参数使其回滚到上一版本
# Web浏览器访问验证：标题为“ChironW-2医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
# 再次构建工程“yiliao_undo”；填写构建参数指定回滚到“REVISION=3”版本
# Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
8:部署Prometheus+Grafana
Node节点安装cadvisor和node-exporter
cadvisor在安装“kubelet”时，已经包含进去了，此处可省略
# 所有“Node节点”上传cadvisor和node-exporter镜像文件并导入
[root@k8s-node images]# wget http://192.168.2.254:8888/docker_monitor_node.tar.gz
[root@k8s-node images]# docker load -i docker_monitor_node.tar.gz
# 运行“node-exporter”容器
[root@k8s-node images]# docker run -d -p 9100:9100 -v "/:/host:ro,rslave" --name=node_exporter quay.io/prometheus/node-exporter --path.rootfs /host
# cadvisor在安装“kubelet”时，已经包含进去了，此处可省略
[root@k8s-node images]# docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
Master节点部署Prometheus+Grafana
# Master节点部署Prometheus+Grafana
# 部署Prometheus
[root@k8s-master /]# cd /opt/
# 上传本地prometheus归档包并解压
[root@k8s-master opt]# wget http://192.168.2.254:8888/prometheus-2.12.0.linux-amd64.tar.gz
[root@k8s-master opt]# tar xf prometheus-2.12.0.linux-amd64.tar.gz
[root@k8s-master opt]# cd prometheus-2.12.0.linux-amd64/
[root@k8s-master prometheus-2.12.0.linux-amd64]# cp prometheus.yml{,.bak}
# 修改文件“prometheus.yml”为以下内容
[root@k8s-master prometheus-2.12.0.linux-amd64]# grep -Ev '^$|#' prometheus.yml
global:
alerting:
alertmanagers:
- static_configs:
- targets:
rule_files:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'cadvisor'
static_configs:
- targets: ['10.0.0.12:4194','10.0.0.13:4194']
- job_name: 'node'
static_configs:
- targets: ['10.0.0.12:9100','10.0.0.13:9100']
# 启动prometheus服务，默认运行在前台
[root@k8s-master prometheus-2.12.0.linux-amd64]# ./prometheus --config.file="prometheus.yml"
# Web浏览器访问
http://10.0.0.11:9090/
# 部署Grafana
# 上传本地RPM包：Grafana进行安装
[root@k8s-master ~]# wget http://192.168.2.254:8888/grafana-6.3.3-1.x86_64.rpm
[root@k8s-master ~]# yum localinstall grafana-6.3.3-1.x86_64.rpm -y
# 启动Grafana服务并加入开机自启
[root@k8s-master ~]# systemctl start grafana-server.service
[root@k8s-master ~]# systemctl enable grafana-server.service
# 查看Grafana服务占用端口号
[root@k8s-master ~]# netstat -tnpl |grep grafana
tcp6 0 0 :::3000 :::* LISTEN 14962/grafana-serve
# Web浏览器访问
http://10.0.0.11:3000/login
默认用户名密码：admin/admin
# 配置prometheus数据源不做详解
#导入事先准备好的dashboard出图模板“grafana_docker_dashboard.json”文件
# 模板文件可以从“grafana”官方进行搜索下载：
https://grafana.com/grafana/dashboards
标题为“ChironW医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
回滚到指定版本
Jenkins使用参数化构建回滚到指定版本；执行Shell语句为：（参数化构建回滚到指定版本）
if [ -z KaTeX parse error: Undefined control sequence: \d at position 182: …p -oP '(?<=80:)\̲d̲+'` echo "你的项目…port"
echo “你的项目地址访问是http://10.0.0.12: p o r t " e c h o " 回 滚 上 一 版 本 成 功 " e l s e e c h o " 回 滚 到 指 定 版 本 : port" echo "回滚上一版本成功" else echo "回滚到指定版本: port"echo"回滚上一版本成功"elseecho"回滚到指定版本:REVISION！”
kubectl -s 10.0.0.11:8080 rollout undo -n yiliao deployment yiliao --to-revision=KaTeX parse error: Undefined control sequence: \d at position 80: …p -oP '(?<=80:)\̲d̲+'` echo "你的项目…port"
echo “你的项目地址访问是http://10.0.0.12: p o r t " e c h o " 回 滚 到 指 定 版 本 : port" echo "回滚到指定版本: port"echo"回滚到指定版本:REVISION成功”
fi
echo "你的所有历史版本如下"
kubectl -s 10.0.0.11:8080 rollout history -n yiliao deployment yiliao
构建工程“yiliao_undo”；不填写构建参数使其回滚到上一版本
Web浏览器访问验证：标题为“ChironW-2医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
再次构建工程“yiliao_undo”；填写构建参数指定回滚到“REVISION=3”版本
Web浏览器访问验证：标题为“ChironW医疗”
http://10.0.0.13:31900
http://10.0.0.12:31900
## 8:部署Prometheus+Grafana
### Node节点安装cadvisor和node-exporter
cadvisor在安装“kubelet”时，已经包含进去了，此处可省略
```shell
# 所有“Node节点”上传cadvisor和node-exporter镜像文件并导入
[root@k8s-node images]# wget http://192.168.2.254:8888/docker_monitor_node.tar.gz
[root@k8s-node images]# docker load -i docker_monitor_node.tar.gz
# 运行“node-exporter”容器
[root@k8s-node images]# docker run -d -p 9100:9100 -v "/:/host:ro,rslave" --name=node_exporter quay.io/prometheus/node-exporter --path.rootfs /host
# cadvisor在安装“kubelet”时，已经包含进去了，此处可省略
[root@k8s-node images]# docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
Master节点部署Prometheus+Grafana
# Master节点部署Prometheus+Grafana
# 部署Prometheus
[root@k8s-master /]# cd /opt/
# 上传本地prometheus归档包并解压
[root@k8s-master opt]# wget http://192.168.2.254:8888/prometheus-2.12.0.linux-amd64.tar.gz
[root@k8s-master opt]# tar xf prometheus-2.12.0.linux-amd64.tar.gz
[root@k8s-master opt]# cd prometheus-2.12.0.linux-amd64/
[root@k8s-master prometheus-2.12.0.linux-amd64]# cp prometheus.yml{,.bak}
# 修改文件“prometheus.yml”为以下内容
[root@k8s-master prometheus-2.12.0.linux-amd64]# grep -Ev '^$|#' prometheus.yml
global:
alerting:
alertmanagers:
- static_configs:
- targets:
rule_files:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'cadvisor'
static_configs:
- targets: ['10.0.0.12:4194','10.0.0.13:4194']
- job_name: 'node'
static_configs:
- targets: ['10.0.0.12:9100','10.0.0.13:9100']
# 启动prometheus服务，默认运行在前台
[root@k8s-master prometheus-2.12.0.linux-amd64]# ./prometheus --config.file="prometheus.yml"
# Web浏览器访问
http://10.0.0.11:9090/
# 部署Grafana
# 上传本地RPM包：Grafana进行安装
[root@k8s-master ~]# wget http://192.168.2.254:8888/grafana-6.3.3-1.x86_64.rpm
[root@k8s-master ~]# yum localinstall grafana-6.3.3-1.x86_64.rpm -y
# 启动Grafana服务并加入开机自启
[root@k8s-master ~]# systemctl start grafana-server.service
[root@k8s-master ~]# systemctl enable grafana-server.service
# 查看Grafana服务占用端口号
[root@k8s-master ~]# netstat -tnpl |grep grafana
tcp6 0 0 :::3000 :::* LISTEN 14962/grafana-serve
# Web浏览器访问
http://10.0.0.11:3000/login
默认用户名密码：admin/admin
# 配置prometheus数据源不做详解
#导入事先准备好的dashboard出图模板“grafana_docker_dashboard.json”文件
# 模板文件可以从“grafana”官方进行搜索下载：
https://grafana.com/grafana/dashboards