devops-4-kubernetes(k8s)
官方文档
尚硅谷文档笔记
尚硅谷视频
概念
满足 应用程序的
快速精准的部署
即时伸缩容
无缝展现新特性
限制资源
优势
私有、公有、混合、多态云
扩展 模块化 插件化
修复： 自动部署、自动重启、自动复制、自动扩缩容
协调辅助进程、挂在存储系统、分布式机密信息、检查程序状态、复制应用实例、负载均衡、滚动更新、资源监控、访问读取日志、程序调试、验证授权
kubectl ：命令行工具
apiserver：rest api接口，系统的控制入口
controller-manager：执行系统的后台任务，节点状态，pod个数 和 service服务的关联
etcd：键值对数据库，负责节点间的服务发现和配置共享（高可用、强一致服务发现存储仓库）
kubelet：管理node节点pod、存储、网络
kube-proxy：pod网络代理
namespace 命名空间 -资源隔离-不隔离网络
pods 部署应用 有所在的命名空间
命令
kubectl explain pod 对应pod有哪些模版
kubectl explain pod.apiVersion
kubectl explain pod.spec.containers
运行pod
kubectl apply -f server.yaml
kubectl create -f server.yaml #资源清单创建pod
kubectl create -f hao.yaml
删除
kubectl get deploy
kubectl delete apply server名称
kubectl delete podname --all
kubectl delete svc svcname
查看
kubectl get pod
kubectl get pod -o wide 查看详细信息 ip
kubectl get deploy
kubectl get svc 查看集群
kubectl get service
kubectl describe pod hao 查看pod中容器状态信息
kubectl log hao -c hao1 查看pod日志，多个容器-c指定名称
进入
kubectl exec pod podname --it /bin/sh
pod中多个容器需要-c
init 初始化master 各组件
join 其它node 加入集群 令牌24小时过期
重新生成令牌
kubeadm token create --print-join-command
kubectl get nodes
name 名字
status ready 完成
roles 角色 control-plane，master
age 时长
version 版本
kubectl get pod -A
namespace 命名空间
name 名字
ready 完成状态
status running containerCreating
restarts 重启次数，故障自愈
age 启动时长
kubectl apply -f xxx.yml
kubectl get pod -A -w # 监控状态变化
查看命名空间
kubectl get pod -A
查看命名空间下得pod
kubectl get pod -n kubernetes-dashboard
服务ip
集群访问
kubectl expose deployment my-dep --port=8000 --target-port=80 --type=ClusterIP
外网访问
kubectl expose deployment my-dep --port=8000 --target-port=80 --type=NodePort
Ingress service 统一网管关入口
安装
# 添加 k8s 安装源
cat < kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
mv kubernetes.repo /etc/yum.repos.d/
# 添加 Docker 安装源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y kubelet-1.22.4 kubectl-1.22.4 kubeadm-1.22.4 docker-ce
systemctl enable kubelet
systemctl start kubelet
systemctl enable docker
systemctl start docker
初始化异常
#kubeadm init --image-repository=registry.aliyuncs.com/google_containers
I0826 10:05:21.490091 4475 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.13
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
解决：
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
初始化
kubeadm init --image-repository=registry.aliyuncs.com/google_containers
I0826 10:20:56.374420 18984 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.13
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [diagnosis-tool011164062250.na62 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 11.164.62.250]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [diagnosis-tool011164062250.na62 localhost] and IPs [11.164.62.250 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [diagnosis-tool011164062250.na62 localhost] and IPs [11.164.62.250 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.002982 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node diagnosis-tool011164062250.na62 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node diagnosis-tool011164062250.na62 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: r54t2j.eqc99160c8slv45x
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 11.164.62.250:6443 --token r54t2j.eqc99160c8slv45x \
--discovery-token-ca-cert-hash sha256:62cc9771e7b45290fed79f2e099492aa3a9a8b3fd80652391ccf0498ec54b7ae
# 记得把 kubeadm join xxx 保存起来
# 忘记了重新获取：kubeadm token create --print-join-command
# 复制授权文件，以便 kubectl 可以有权限访问集群
# 如果你其他节点需要访问集群，需要从主节点复制这个文件过去其他节点
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
# 在其他机器上创建 ~/.kube/config 文件也能通过 kubectl 访问到集群
节点加入
- kubeadm token create --print-join-command 通过token 和 ca信息加入
- 报错：[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error: Get “http://localhost:10248/healthz”: dial tcp 127.0.0.1:10248: connect: connection refused.
- 参考 kubernetes cgroup驱动为systemd 而 yum 安装的docker驱动为system次该驱动
vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
- 加入集群
kubeadm join xxx.xxx.xx.xx:6443 --token 268fma.xxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
安装ipvs
#kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
flannel 启动报错
- Error registering network: failed to acquire lease: node “k8s-master01” pod cidr not assigned
- kubectl get ds -n kube-flannel
- kubect get pod -n kube-flannel
- kubectl logs kube-flannel-ds-xxxx -n kube-flannel
- 解决方案
- 编辑 /etc/kubernetes/manifests/kube-controller-manager.yaml
加入
–allocate-node-cidrs=true
–cluster-cidr=10.244.0.0/16
查看nodes
#kubectl get nodes
NAME STATUS ROLES AGE VERSION
diagnosis-tool011164062250.na62 Ready control-plane,master 54m v1.22.4
diagnosis-tool011164063137.na62 Ready 2m25s v1.22.4
命名空间下的pod
kubectl get pods -n kube-system
解决文件系统只读问题
touch /etc/a
touch: cannot touch ‘/etc/a’: Read-only file system
mount
查看文件系统为只读
/dev/vda1 on / type ext4 (ro,relatime,data=ordered)
解决：重新挂载
mount -o remount,rw /
/dev/vda1 on / type ext4 (rw,relatime,data=ordered)
pod
配置文件参数说明
- 生成 yaml
kubectl create deployment web --image=nginx -o yaml --dry-run > nginx.yaml
- 部署好的pod生成yaml
kubectl get pod testapp -o yaml > test.yaml
kubectl run testapp --image=ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v1
运行testapp pod 镜像是
#kubectl get pod
NAME READY STATUS RESTARTS AGE
testapp 0/1 Pending 0 149m
配置文件部署
vi pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
# 定义容器，可以多个
containers:
- name: test-k8s # 容器名字
image: ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v1 # 镜像
执行
kubectl apply -f pod.yaml
kubectl get pod
查看pod pading原因
#kubectl get pod
NAME READY STATUS RESTARTS AGE
test-pod 0/1 Pending 0 84m
testapp 0/1 Pending 0 3h58m
kubectl describe node diagnosis-tool011164063137.na62 > detail.node
58 Allocated resources:
59 (Total limits may be over 100 percent, i.e., overcommitted.)
60 Resource Requests Limits
61 -------- -------- ------
62 cpu 100m (2%) 100m (2%)
63 memory 50Mi (0%) 50Mi (0%)
64 ephemeral-storage 0 (0%) 0 (0%)
65 hugepages-1Gi 0 (0%) 0 (0%)
66 hugepages-2Mi 0 (0%) 0 (0%)
67 Events:
68 Type Reason Age From Message
69 ---- ------ ---- ---- -------
70 Warning ImageGCFailed 2m2s (x25 over 122m) kubelet (combined from similar events): wanted to free 10735946137 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository refer ence "redis:latest" (must force) - container f0e742b9be40 is using its referenced image 3e42dd4e79c7
处理：
删除镜像
#docker image rm 3e4
Error response from daemon: conflict: unable to delete 3e42dd4e79c7 (must be forced) - image is being used by stopped container f0e742b9be40
停止容器
docker ps -a
docker stop f0e742b9be40
docker rm -f f0e742b9be40
查看网络配置
kubectl describe pod test-pod
ror: code = Unknown desc = failed to set up sandbox container "c45eb240a3b47cd8f800b36b6bc56abb0ed98afdda47985c6eb494daed6bc5b6" network for pod "test-pod": networkPlugin cni failed to set up pod "test-pod_default" network: open /run/flannel/subnet.env: no such file or directory
查看 flannel的地址
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
vi kube-flannel.yml
82 net-conf.json: |
83 {
84 "Network": "10.244.0.0/16",
85 "Backend": {
86 "Type": "vxlan"
87 }
88 }
解决master 各节点
vi /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
监控pod状态变化
#kubectl get pod -A -w
NAMESPACE NAME READY STATUS RESTARTS AGE
default test-pod 1/1 Running 4 (41h ago) 12d
default testapp 1/1 Running 3 (41h ago) 12d
kube-flannel kube-flannel-ds-mp4pv 0/1 CrashLoopBackOff 3641 (80s ago) 12d
kube-flannel kube-flannel-ds-vclq8 0/1 CrashLoopBackOff 3643 (3m28s ago) 12d
k8s web界面 dashboard
master上
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml
修改类型 设置访问端口
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
修改 type: ClusterIP 改为 type: NodePort
kubectl get svc -A |grep kubernetes-dashboard
## 找到端口，在安全组放行
detail
#kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19; use the "seccompProfile" field instead
deployment.apps/dashboard-metrics-scraper created
#kubectl get svc -A |grep kubernetes-dashboard
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.107.5.58 <none> 8000/TCP 6m27s
kubernetes-dashboard kubernetes-dashboard NodePort 10.104.27.20 <none> 443:30993/TCP 6m27s
kubectl create -f hao.yaml
hao.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: hao
version: v1
name: hao-pod
spec:
containers:
- name: hao
image: nginx
pod声明周期
pod 生命周期
InitC 初始化容器：初始化完成后就会死亡，不会一直存在,多个initcs是线性过程
容器运行前后会有一个 start stop操作
readiness 就绪检测 pod已经就绪但是服务进程未就绪，命令、tcp检测服务是否可用
liveness 生存检测 pod中运行的主容器 main C，出现了僵尸进程
创建过程：
1、kubectl向api接口发送创建请求，调度会将请求分发给kubelet ,etcd在中间存储状态
2、cri完成初始化，在初始化的过程中会创建一个pause的基础容器负责网络和存储卷的共享，pod中所有容器都是共享的。接着会
进行initC的初始化为0就会正常退出
3、initC正常退出以后就会进入到MainC的主容器运行，在启动的时候运行一个start的命令，结束是也会运行stop命令，在这个过
程中会有liveness和readiness的参与
4、只有readiness检测成功pod才会变为running或者ready
5、liveness会伴随整个pod的生命周期，当检测MainC中主程序无法正常工作，可以更具策略执行重启或其他动作
探针检测
kubernetes二次开发
代码库
git clone https://github.com/kubernetes/kubernetes.git
git clone https://github.com/kubernetes-sigs/controller-runtime
kubernetes 问题
- 启动pod systemctl 服务报错:
- Failed to get D-Bus connection: Operation not permitted
- 解决 yaml添加 特权参数
16 containers:
17 - name: app
18 image: app:v1
19 securityContext:
20 privileged: true