开发环境k8s维护和管理手册
环境说明
| 功能名称 | IP | 配置 |
|---|---|---|
| k8s-master | 192.168.10.231 | 4c8g |
| k8s-node1 | 192.168.10.232 | 8c16g |
| K8s-node2 | 192.168.10.233 | 8c16g |
| k8s-node3 | 192.168.10.234 | 8c16g |
| k8s-node4 | 192.168.10.235 | 8c16g |
环境初始化
- 更新环境
yum update -y yum install -y wget vim net-tools epel-release -
关闭filewalld
systemctl disable firewalld systemctl stop firewalld - 关闭selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux if [ `getenforce` == "Enforcing" ];then setenforce 0 else echo "current selinux status..." `getenforce` fi - 关闭swap
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab- 增加主机名解析
cat << EOF /etc/host
192.168.10.231 dev-k8s-01.example.com
192.168.10.232 dev-k8s-02.example.com
192.168.10.233 dev-k8s-03.example.com
192.168.10.234 dev-k8s-04.example.com
192.168.10.235 dev-k8s-05.example.com
EOF- 优化内核参数
```bash
cat << EOF >> /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
7. 更新Yum源配置
```bash
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.`date +%F`.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache fast
cat << EOF /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF
yum clean all
yum makecache fast
yum -y update- 安装docker
yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-18.09.9-3.el7
mkdir /etc/docker -pv
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://********.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
systemctl enable --now docker.service- 安装初始化工具
yum install -y kubeadm kubelet- 获取基础镜像
KUBE_VERSION=v1.16.0
KUBE_PAUSE_VERSION=3.1
ETCD_VERSION=3.3.15-0
CORE_DNS_VERSION=1.6.2
GCR_URL=k8s.gcr.io
ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers
images=(kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${CORE_DNS_VERSION})
for imageName in ${images[@]} ; do
docker pull $ALIYUN_URL/$imageName
docker tag $ALIYUN_URL/$imageName $GCR_URL/$imageName
docker rmi $ALIYUN_URL/$imageName
done以上10部建议在所有的节点上安装,在node节点上可以不用安装kubeadm
部署集群
kubeadm初始化集群
[root@dev-k8s-01 ~]# sudo kubeadm init \
> --apiserver-advertise-address 192.168.10.231 \
> --kubernetes-version=v1.16.0 \
> --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [dev-k8s-01.example.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.231]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [dev-k8s-01.example.com localhost] and IPs [192.168.10.231 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [dev-k8s-01.example.com localhost] and IPs [192.168.10.231 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 39.003840 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node dev-k8s-01.example.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node dev-k8s-01.example.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[kubelet-check] Initial timeout of 40s passed.
[bootstrap-token] Using token: 9nwjok.ykyphybsveka8gev
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.10.231:6443 --token 9nwjok.ykyphybsveka8gev \
--discovery-token-ca-cert-hash sha256:b92d7553a1da683a315ad2f4f5fcc855e2d630da0c7553467cdf2db3bd25a3ff初始化kubectl配置文件
[root@dev-k8s-01 ~]# mkdir -p $HOME/.kube
[root@dev-k8s-01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@dev-k8s-01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config添加节点
- 添加192.168.10.232
[root@dev-k8s-05 ~]# kubeadm join 192.168.10.231:6443 --token 9pr3rj.0u8m510fai0op75h \
--discovery-token-ca-cert-hash sha256:b86bdaaa0bed56e846adb0abc625cf29902dec9e3130d0ff7dae42ffb2e13349
[preflight] Running pre-flight checks
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@dev-k8s-05 ~]# systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@dev-k8s-05 ~]# 如上所示依旧添加192.168.10.233节点
验证集群状态
[root@dev-k8s-01 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.10.231:6443
KubeDNS is running at https://192.168.10.231:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
➜ ~ (☸ kubernetes-admin@kubernetes:default) kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
dev-k8s-01.example.com Ready master 14h v1.16.3 192.168.10.231 CentOS Linux 7 (Core) 3.10.0-957.21.3.el7.x86_64 docker://18.9.9
dev-k8s-02.example.com Ready 14h v1.16.3 192.168.10.232 CentOS Linux 7 (Core) 3.10.0-957.21.3.el7.x86_64 docker://18.9.9
dev-k8s-03.example.com Ready 14h v1.16.3 192.168.10.233 CentOS Linux 7 (Core) 3.10.0-957.21.3.el7.x86_64 docker://18.9.9
dev-k8s-04.example.com Ready 14h v1.16.3 192.168.10.234 CentOS Linux 7 (Core) 3.10.0-1062.4.1.el7.x86_64 docker://18.9.9
dev-k8s-05.example.com Ready 13h v1.16.3 192.168.10.235 CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.9
➜ ~ (☸ kubernetes-admin@kubernetes:default) kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-5644d7b6d9-96xm6 1/1 Running 0 14h 10.244.3.2 dev-k8s-04.example.com
kube-system coredns-5644d7b6d9-nkb9f 1/1 Running 0 14h 10.244.1.2 dev-k8s-02.example.com
kube-system etcd-dev-k8s-01.example.com 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com
kube-system kube-apiserver-dev-k8s-01.example.com 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com
kube-system kube-controller-manager-dev-k8s-01.example.com 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com
kube-system kube-proxy-bhtjc 1/1 Running 0 14h 192.168.10.232 dev-k8s-02.example.com
kube-system kube-proxy-h2ltx 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com
kube-system kube-proxy-kh9k9 1/1 Running 0 14h 192.168.10.234 dev-k8s-04.example.com
kube-system kube-proxy-lfh46 1/1 Running 0 14h 192.168.10.233 dev-k8s-03.example.com
kube-system kube-proxy-pcm5d 1/1 Running 0 13h 192.168.10.235 dev-k8s-05.example.com
kube-system kube-scheduler-dev-k8s-01.example.com 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com 安装插件
安装flannel网络插件
wget -O /opt/k8sworkspces/kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
FLANNEL_VERSION=v0.11.0
QUAY_URL=quay.io/coreos
QINIU_URL=quay-mirror.qiniu.com/coreos
images=(flannel:${FLANNEL_VERSION}-amd64
flannel:${FLANNEL_VERSION}-arm64
flannel:${FLANNEL_VERSION}-arm
flannel:${FLANNEL_VERSION}-ppc64le
flannel:${FLANNEL_VERSION}-s390x)
for imageName in ${images[@]} ; do
docker pull $QINIU_URL/$imageName
docker tag $QINIU_URL/$imageName $QUAY_URL/$imageName
docker rmi $QINIU_URL/$imageName
done # 也可以只拉去你机器适配的架构版本`rpm -q centos-release`
kubectl apply -f /opt/k8sworkspces/kube-flannel.yml #安装flannel
➜ ~ (☸ kubernetes-admin@kubernetes:default) kubectl get pods --all-namespaces -o wide | grep flannel
kube-system kube-flannel-ds-amd64-9tnc7 1/1 Running 0 14h 192.168.10.234 dev-k8s-04.example.com
kube-system kube-flannel-ds-amd64-cjh4s 1/1 Running 0 14h 192.168.10.231 dev-k8s-01.example.com
kube-system kube-flannel-ds-amd64-fhlk4 1/1 Running 0 13h 192.168.10.235 dev-k8s-05.example.com
kube-system kube-flannel-ds-amd64-fnfpj 1/1 Running 0 14h 192.168.10.233 dev-k8s-03.example.com
kube-system kube-flannel-ds-amd64-v5qtj 1/1 Running 0 14h 192.168.10.232 dev-k8s-02.example.com krew
krew 能够很方便的管理kubectl的插件包,包括安装卸载，查询升级
安装
(
set -x; cd /opt/k8sworkspces/krew &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/download/v0.3.2/krew.{tar.gz,yaml}" &&
tar zxvf krew.tar.gz &&
./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" install \
--manifest=krew.yaml --archive=krew.tar.gz
)
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
[root@dev-k8s-01 krew]# kubectl krew install ca-cert # 安装一个ca-caert的插件
[root@dev-k8s-01 krew]# kubectl ca-cert
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTE5MTExNTA0MjEzOVoXDTI5MTExMjA0MjEzOVowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANEi
tPdWINQZfZqM4c/uaOzsBBByn0CLLLmMdiKF4Gpk9proDoR9eOMhQiiVLZ4tFFsb
POTwq+MvHe4kEsunl/hBwNbXvGfbvnr+vX9ZsDfU5FT5O55Zryq5jgANDKFChKx9
R91QsbCeQKIWlc9AFdot8ig9LhYTfHJRfMeUBYl5Xzoof8YRMsJ0jOKLWca+oCfd
doLKda9VpahU2AEmEFHuD6ctwBGFObadSktoOvr0Gfzo4cXRkjGXp4G1U8O1LLsU
HiypNN4m7Romy4tIjPAxDAoDDyjA8OrbPlvJt8Oo0CHcAxFZDJCsKAG1s0nS7PJj
vR2ULtIrHAm5QZa8BmMCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAKAi1Fg/2MlFxPbq9yaNkBhAV2ou
/VbbuEJF1c92Tk24cuJV3vuYoTmWIGp1LYTLTW/xcfwFoanLRPBlBONoJRzXLIZD
/mmuYMrTaKMwbCz2t4awqQyDb8A3RcgTrSfCWMs0uyvjPVgiJDfMlg0WDJ4kPb3Y
SQv7UaaNa57gkEHB1PJy10n1E3gAcb6NVxvly7cHVaJlenZY6mkT40K8zVOXuM/G
ausCNXEfEUXED2C8Ippj/sr1TgRlD8Gfi+Xp7XzHTeu5A+ac4YPmnoW8jurzo5z5
Q5TDBFRaOTyRgUxYt+PKv01S9tTiHgkxHoBzPQF7Z2TuRNKXoVQeXiUzW/s=
-----END CERTIFICATE-----
[root@dev-k8s-01 krew]# kubectl krew --help #查看krew的支持选项