2019独角兽企业重金招聘Python工程师标准>>> 
1 环境
| Host Name | Role | IP |
|---|---|---|
| master1 | k8s-1001 | 172.31.135.239 |
| node1 | k8s-1002 | 172.31.135.238 |
| node2 | k8s-1003 | 172.31.135.237 |
2 内核调优
cat > /etc/sysctl.d/k8s.conf <3 修改文件描述符
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536" >> /etc/security/limits.conf
echo "* hard nproc 65536" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf
echo "* hard memlock unlimited" >> /etc/security/limits.conf
hard limits自AIX 4.1版本开始引入。hard limits 应由AIX系统管理员设置，只有security组的成员可以将此值增大，用户本身可以减小此限定值,但是其更改将随着该用户从系统退出而失效
soft limits 是AIX核心使用的限制进程对系统资源的使用的上限值。此值可由任何人更改，但不能超出hard limits值。这里要注意的是只有security组的成员可使更改永久生效普通用户的更改在其退出系统后将失效
1)soft nofile和hard nofile示，单个用用户的软限制为1000，硬限制为1200，即表示单用户能打开的最大文件数量为1000，不管它开启多少个shell。
2）soft nproc和hard nproc 单个用户可用的最大进程数量,软限制和硬限制
3）memlock 一个任务锁住的物理内存的最大值（这里设置成无限制）4 配置k8s yum 源
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF 5 配置docker yum源
cd /etc/yum.repos.d
wget https://download.docker.com/linux/centos/docker-ce.repo6 时间同步
一个集群内的时间同步必不可少
systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
ntpdate -u ntp.api.bz7 关闭SELinux、防火墙
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config8 关闭系统的Swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab9 配置hosts 解析
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.135.239 k8s-1001 k8s-1001
172.31.135.237 k8s-1003 k8s-1003
172.31.135.238 k8s-1002 k8s-100210 配置节点免密登录
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub 用户名字@192.168.x.xxx11 安装依赖等
yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim ntpdate libseccomp libtool-ltdl lrzsz wget 12 配置ipvs模块
cat > /etc/sysconfig/modules/ipvs.modules <13 安装docker
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-18.06.1.ce-3.el7
最号先不要安装最新版本
systemctl daemon-reload
systemctl enable docker
systemctl start docker14 master 和node 节点安装 kubelet kubeadm kubectl
yum install -y kubelet kubeadm kubectl master
systemctl enable kubelet
暂不启动 kubelet15 如果在国内请按照下面步骤进行
生成默认配置
kubeadm config print init-defaults > /root/kubeadm.conf修改 /root/kubeadm.conf，使用国内阿里的imageRepository: registry.aliyuncs.com/google_containers
下载镜像
kubeadm config images pull --config /root/kubeadm.conf16 如过网络允许可以直接初始化集群
kubeadm init --kubernetes-version=v1.14.1 --pod-network-cidr=10.244.0.0/16
这里没有设置--service-cidr 是因为我们下面需要部署calico网络，calico会帮助我们设置service网络，如果此地设置了service网络会导致calico部署不成功保存这段内容
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
--discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1
16 创建相关文件夹
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config18 如果以后还有机器加入集群如何获取token 和 hash值？
获得token和hash值
1）获取token
kubeadm token list
默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token
kubeadm token create
2）获取hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'19 验证
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-pdf9r 0/1 Pending 0 6m50s
kube-system coredns-fb8b8dccf-rngcz 0/1 Pending 0 6m50s
kube-system etcd-k8s-1001 1/1 Running 0 5m52s
kube-system kube-apiserver-k8s-1001 1/1 Running 0 6m4s
kube-system kube-controller-manager-k8s-1001 1/1 Running 0 5m51s
kube-system kube-proxy-b8dhg 1/1 Running 0 6m50s
kube-system kube-scheduler-k8s-1001 1/1 Running 0 5m47s
coredns 是pending状态 先不用管它，因为这个没有网络插件的导致的
20 部署calico网络插件
官方文档
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml我们需要修改calico文件
https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
vim calico.yaml
1）修改ipip模式关闭 和typha_service_name
- name: CALICO_IPV4POOL_IPIP
value: "off"
typha_service_name: "calico-typha"
calico网络，默认是ipip模式（在每台node主机创建一个tunl0网口，这个隧道链接所有的node容器网络，官网推荐不同的ip网段适合，比如aws的不同区域主机），
修改成BGP模式，它会以daemonset方式安装在所有node主机，每台主机启动一个bird(BGP client)，它会将calico网络内的所有node分配的ip段告知集群内的主机，并通过本机的网卡eth0或者ens33转发数据；
2）修改replicas
replicas: 1
revisionHistoryLimit: 2
3）修改pod的网段CALICO_IPV4POOL_CIDR
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
4）如果手动下载镜像请查看calico.yaml 文件里面标注的镜像版本 否则可以直接执行会自动下载
5）部署calico
kubectl apply -f calico.yaml
6）查看
kubectl get po --all-namespaces
此时你会发现是pending状态是因为node节点还没有相关组件
7） 验证是否为bgp模式
# ip route show
default via 172.31.143.253 dev eth0
blackhole 10.244.0.0/24 proto bird
10.244.0.2 dev caliac6de7553e8 scope link
10.244.0.3 dev cali1591fcccf0f scope link
10.244.1.0/24 via 172.31.135.237 dev eth0 proto bird
10.244.2.0/24 via 172.31.135.238 dev eth0 proto bird
169.254.0.0/16 dev eth0 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.31.128.0/20 dev eth0 proto kernel scope link src 172.31.135.239
21 将node加入到节点
kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
--discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1
master上查看集群
[root@k8s-1001 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-1001 Ready master 37m v1.14.1
k8s-1002 Ready 99s v1.14.1
k8s-1003 Ready 115s v1.14.1
[root@k8s-1001 ~]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-1001 Ready master 37m v1.14.1 172.31.135.239 CentOS Linux 7 (Core) 3.10.0-862.14.4.el7.x86_64 docker://18.6.1
k8s-1002 Ready 103s v1.14.1 172.31.135.238 CentOS Linux 7 (Core) 3.10.0-862.14.4.el7.x86_64 docker://18.6.1
k8s-1003 Ready 119s v1.14.1 172.31.135.237 CentOS Linux 7 (Core) 3.10.0-862.14.4.el7.x86_64 docker://18.6.1
[root@k8s-1001 ~]# kubectl get pod -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-node-8z92v 2/2 Running 0 2m1s 172.31.135.238 k8s-1002
calico-node-k542k 2/2 Running 0 7m32s 172.31.135.239 k8s-1001
calico-node-n4jgf 2/2 Running 0 2m17s 172.31.135.237 k8s-1003
calico-typha-55968bfd7b-c5r4z 1/1 Running 0 7m33s 172.31.135.237 k8s-1003
coredns-fb8b8dccf-pdf9r 1/1 Running 0 37m 10.244.0.3 k8s-1001
coredns-fb8b8dccf-rngcz 1/1 Running 0 37m 10.244.0.2 k8s-1001
etcd-k8s-1001 1/1 Running 0 36m 172.31.135.239 k8s-1001
kube-apiserver-k8s-1001 1/1 Running 0 36m 172.31.135.239 k8s-1001
kube-controller-manager-k8s-1001 1/1 Running 0 36m 172.31.135.239 k8s-1001
kube-proxy-b8dhg 1/1 Running 0 37m 172.31.135.239 k8s-1001
kube-proxy-nvlmz 1/1 Running 0 2m17s 172.31.135.237 k8s-1003
kube-proxy-rfb77 1/1 Running 0 2m1s 172.31.135.238 k8s-1002
kube-scheduler-k8s-1001 1/1 Running 0 36m 172.31.135.239 k8s-1001
22 开启kube-proxy lvs
修改ConfigMap的kube-system/kube-proxy中的config.conf，`mode: "ipvs"`：
kubectl edit cm kube-proxy -n kube-system
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.31.135.239:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 10.244.0.2:53 Masq 1 0 0
-> 10.244.0.3:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.2:9153 Masq 1 0 0
-> 10.244.0.3:9153 Masq 1 0 0
TCP 10.111.3.127:5473 rr
-> 172.31.135.237:5473 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.2:53 Masq 1 0 0
-> 10.244.0.3:53 Masq 1 0 0
23 master默认是污点，pod是不会调度到master
1）开放master可被调度
kubectl taint node k8s-1001 node-role.kubernetes.io/master-
2）
如果要恢复 Master Only 状态
kubectl taint node k8s-1001 node-role.kubernetes.io/master="":NoSchedule24 创建一个测试pod 验证集群
kubectl run net-test --image=alpine --replicas=2 sleep 360
# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
net-test-7d6d58cc8-78wd6 1/1 Running 0 6s 10.244.1.2 k8s-1003
net-test-7d6d58cc8-hjdhw 1/1 Running 0 6s 10.244.2.2 k8s-1002