kubernetes 1.7.0 + flannel 二进制部署 kubernetes 1.7.0 + flannel 基于 二进制 文件部署 本地化 kube-apiserver, kube-controller-manager , kube-scheduler (1).环境说明 k8s-master-1: 192.168.54.12 k8s-node1: 192.168.54.13 k8s-node2: 192.168.54.14 (2).初始化环境 hostnamectl --static set-hostname hostname 192.168.54.12 - k8s-master-1 192.168.54.13 - k8s-node1 192.168.54.14 - k8s-node2 #编辑 /etc/hosts 文件,配置hostname 通信 vi /etc/hosts 192.168.54.12 k8s-master-1 192.168.54.13 k8s-node1 192.168.54.14 k8s-node2 创建 验证 这里使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件。 (1).安装 cfssl mkdir -p /opt/local/cfssl cd /opt/local/cfssl wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 mv cfssl_linux-amd64 cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 mv cfssljson_linux-amd64 cfssljson wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 mv cfssl-certinfo_linux-amd64 cfssl-certinfo chmod +x * (2).创建 CA 证书配置 mkd.ir /opt/ssl cd /opt/ssl /opt/local/cfssl/cfssl print-defaults config > config.json /opt/local/cfssl/cfssl print-defaults csr > csr.json # config.json 文件 { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } # csr.json 文件 { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } (3).生成 CA 证书和私钥 cd /opt/ssl/ /opt/local/cfssl/cfssl gencert -initca csr.json | /opt/local/cfssl/cfssljson -bare ca [root@k8s-master-1 ssl]# ls -lt 总用量 20 -rw-r--r-- 1 root root 1005 7月 3 17:26 ca.csr -rw------- 1 root root 1675 7月 3 17:26 ca-key.pem -rw-r--r-- 1 root root 1363 7月 3 17:26 ca.pem -rw-r--r-- 1 root root 210 7月 3 17:24 csr.json -rw-r--r-- 1 root root 292 7月 3 17:23 config.json (4).分发证书 # 创建证书目录 mkdir -p /etc/kubernetes/ssl # 拷贝所有文件到目录下 cp * /etc/kubernetes/ssl # 这里要将文件拷贝到所有的k8s 机器上 scp * 192.168.54.13:/etc/kubernetes/ssl/ scp * 192.168.54.14:/etc/kubernetes/ssl/ etcd 集群 etcd 是k8s集群的基础组件,这里感觉没必要创建双向认证。 (1).安装 etcd yum -y install etcd3 (2).修改 etcd 配置 # etcd-1 # 修改配置文件,/etc/etcd/etcd.conf 需要修改如下参数: mv /etc/etcd/etcd.conf /etc/etcd/etcd.conf-bak vi /etc/etcd/etcd.conf ETCD_NAME=etcd1 ETCD_DATA_DIR="/var/lib/etcd/etcd1.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.54.12:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.54.12:2379,http://127.0.0.1:2379" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.54.12:2380" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.54.12:2380,etcd2=http://192.168.54.13:2380,etcd3=http://192.168.54.14:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.54.12:2379" # etcd-2 # 修改配置文件,/etc/etcd/etcd.conf 需要修改如下参数: mv /etc/etcd/etcd.conf /etc/etcd/etcd.conf-bak vi /etc/etcd/etcd.conf ETCD_NAME=etcd2 ETCD_DATA_DIR="/var/lib/etcd/etcd2.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.54.13:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.54.13:2379,http://127.0.0.1:2379" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.54.13:2380" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.54.12:2380,etcd2=http://192.168.54.13:2380,etcd3=http://192.168.54.14:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.54.13:2379" # etcd-3 # 修改配置文件,/etc/etcd/etcd.conf 需要修改如下参数: mv /etc/etcd/etcd.conf /etc/etcd/etcd.conf-bak vi /etc/etcd/etcd.conf ETCD_NAME=etcd3 ETCD_DATA_DIR="/var/lib/etcd/etcd3.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.54.14:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.54.14:2379,http://127.0.0.1:2379" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.54.14:2380" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.54.12:2380,etcd2=http://192.168.54.13:2380,etcd3=http://192.168.54.14:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.54.14:2379" 修改 etcd 启动文件 /usr/lib/systemd/system/etcd.service sed -i 's/\\\"${ETCD_LISTEN_CLIENT_URLS}\\\"/\\\"${ETCD_LISTEN_CLIENT_URLS}\\\" --listen-client-urls=\\\"${ETCD_LISTEN_CLIENT_URLS}\\\" --advertise-client-urls=\\\"${ETCD_ADVERTISE_CLIENT_URLS}\\\" --initial-cluster-token=\\\"${ETCD_INITIAL_CLUSTER_TOKEN}\\\" --initial-cluster=\\\"${ETCD_INITIAL_CLUSTER}\\\" --initial-cluster-state=\\\"${ETCD_INITIAL_CLUSTER_STATE}\\\"/g' /usr/lib/systemd/system/etcd.service (3).启动 etcd 分别启动 所有节点的 etcd 服务 systemctl enable etcd systemctl start etcd systemctl status etcd (4).验证 etcd 集群状态 查看 etcd 集群状态: etcdctl cluster-health # 出现 cluster is healthy 表示成功 查看 etcd 集群成员: etcdctl member list member 4b622f1d4543c5f7 is healthy: got healthy result from http://192.168.54.13:2379 member 647542be2d7fdef3 is healthy: got healthy result from http://192.168.54.12:2379 member 83464a62a714c625 is healthy: got healthy result from http://192.168.54.14:2379 Flannel 网络 (1).安装 flannel 这边其实由于内网,就没有使用SSL认证,直接使用了 yum -y install flannel 清除网络中遗留的docker 网络 (docker0, flannel0 等) ifconfig 如果存在 请删除之,以免发生不必要的未知错误 ip link delete docker0 .... (2).配置 flannel 设置 flannel 所用到的IP段 etcdctl --endpoint http://192.168.54.12:2379 set /flannel/network/config '{"Network":"10.233.0.0/16","SubnetLen":25,"Backend":{"Type":"vxlan","VNI":1}}' 接下来修改 flannel 配置文件 vim /etc/sysconfig/flanneld # 旧版本: FLANNEL_ETCD="http://192.168.54.12:2379,http://192.168.54.13:2379,http://192.168.54.14:2379" # 修改为 集群地址 FLANNEL_ETCD_KEY="/flannel/network/config" # 修改为 上面导入配置中的 /flannel/network FLANNEL_OPTIONS="--iface=em1" # 修改为 本机物理网卡的名称 # 新版本: FLANNEL_ETCD="http://192.168.54.12:2379,http://192.168.54.13:2379,http://192.168.54.14:2379" # 修改为 集群地址 FLANNEL_ETCD_PREFIX="/flannel/network" # 修改为 上面导入配置中的 /flannel/network FLANNEL_OPTIONS="--iface=em1" # 修改为 本机物理网卡的名称 (3).启动 flannel systemctl enable flanneld systemctl start flanneld systemctl status flanneld 安装 docker # 导入 yum 源 # 安装 yum-config-manager yum -y install yum-utils # 导入 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo # 更新 repo yum makecache # 安装 yum install docker-ce (1).更改docker 配置 # 修改配置 vi /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS $DOCKER_DNS_OPTIONS ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target # 修改其他配置 cat >> /usr/lib/systemd/system/docker.service.d/docker-options.conf << EOF [Service] Environment="DOCKER_OPTS=--insecure-registry=10.254.0.0/16 --graph=/opt/docker --registry-mirror=http://b438f72b.m.daocloud.io" EOF # 重新读取配置,启动 docker systemctl daemon-reload systemctl start docker (3).查看docker网络 ifconfig docker0: flags=4099mtu 1500 inet 10.233.19.1 netmask 255.255.255.128 broadcast 0.0.0.0 ether 02:42:c1:2c:c5:be txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 em1: flags=4163 mtu 1500 inet 192.168.54.12 netmask 255.255.255.0 broadcast 10.6.0.255 inet6 fe80::d6ae:52ff:fed1:f0c9 prefixlen 64 scopeid 0x20 ether d4:ae:52:d1:f0:c9 txqueuelen 1000 (Ethernet) RX packets 16286600 bytes 1741928233 (1.6 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 15841272 bytes 1566357399 (1.4 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 flannel.1: flags=4163 mtu 1450 inet 10.233.19.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::d9:e2ff:fe46:9cdd prefixlen 64 scopeid 0x20 ether 02:d9:e2:46:9c:dd txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0 安装 kubectl 工具 (1).Master 端 # 首先安装 kubectl wget https://dl.k8s.io/v1.7.0/kubernetes-client-linux-amd64.tar.gz tar -xzvf kubernetes-client-linux-amd64.tar.gz cp kubernetes/client/bin/* /usr/local/bin/ chmod a+x /usr/local/bin/kube* # 验证安装 kubectl version Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} (2)创建 admin 证书 kubectl 与 kube-apiserver 的安全端口通信,需要为安全通信提供 TLS 证书和秘钥。 cd /opt/ssl/ vi admin-csr.json { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] } # 生成 admin 证书和私钥 cd /opt/ssl/ /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/config.json \ -profile=kubernetes admin-csr.json | /opt/local/cfssl/cfssljson -bare admin # 查看生成 [root@k8s-master-1 ssl]# ls admin* admin.csr admin-csr.json admin-key.pem admin.pem cp admin*.pem /etc/kubernetes/ssl/ (3).配置 kubectl kubeconfig 文件 # 配置 kubernetes 集群 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.54.12:6443 # 配置 客户端认证 kubectl config set-credentials admin \ --client-certificate=/etc/kubernetes/ssl/admin.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ssl/admin-key.pem kubectl config set-context kubernetes \ --cluster=kubernetes \ --user=admin kubectl config use-context kubernetes (4).分发 kubectl config 文件 # 将上面配置的 kubeconfig 文件分发到其他机器 # 其他服务器创建目录 mkdir /root/.kube scp /root/.kube/config 192.168.54.13:/root/.kube/ scp /root/.kube/config 192.168.54.14:/root/.kube/ 部署 kubernetes Master 节点 Master 需要部署 kube-apiserver , kube-scheduler , kube-controller-manager 这三个组件。 (1).安装 组件 # 从github 上下载版本 cd /tmp wget https://dl.k8s.io/v1.7.0/kubernetes-server-linux-amd64.tar.gz tar -xzvf kubernetes-server-linux-amd64.tar.gz cd kubernetes cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/ (2).创建 kubernetes 证书 /opt/ssl vi kubernetes-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.54.12", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } ## 这里 hosts 字段中 三个 IP 分别为 127.0.0.1 本机, 192.168.54.12 为 Master 的IP, 10.254.0.1 为 kubernetes SVC 的 IP, 一般是 部署网络的第一个IP , 如: 10.254.0.1 , 在启动完成后,我们使用 kubectl get svc , 就可以查看到 (3).生成 kubernetes 证书和私钥 /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/config.json \ -profile=kubernetes kubernetes-csr.json | /opt/local/cfssl/cfssljson -bare kubernetes # 查看生成 [root@k8s-master-1 ssl]# ls -lt kubernetes* -rw-r--r-- 1 root root 1245 7月 4 11:25 kubernetes.csr -rw------- 1 root root 1679 7月 4 11:25 kubernetes-key.pem -rw-r--r-- 1 root root 1619 7月 4 11:25 kubernetes.pem -rw-r--r-- 1 root root 436 7月 4 11:23 kubernetes-csr.json # 拷贝到目录 cp -r kubernetes* /etc/kubernetes/ssl/ (4).配置 kube-apiserver kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求,kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token 一致,如果一致则自动为 kubelet生成证书和秘钥。 # 生成 token [root@k8s-master-1 ssl]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 11849e4f70904706ab3e631e70e6af0d # 创建 token.csv 文件 /opt/ssl vi token.csv 11849e4f70904706ab3e631e70e6af0d,kubelet-bootstrap,10001,"system:kubelet-bootstrap" # 拷贝 cp token.csv /etc/kubernetes/ (4).创建 kube-apiserver.service 文件 一、 开启了 RBAC # 自定义 系统 service 文件一般存于 /etc/systemd/system/ 下 vi /etc/systemd/system/kube-apiserver.service [Unit] Description=kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] User=root ExecStart=/usr/local/bin/kube-apiserver \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --advertise-address=192.168.54.12 \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/lib/audit.log \ --authorization-mode=RBAC \ --bind-address=192.168.54.12 \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --enable-swagger-ui=true \ --etcd-cafile=/etc/kubernetes/ssl/ca.pem \ --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \ --etcd-servers=http://192.168.54.12:2379,http://192.168.54.13:2379,http://192.168.54.14:2379 \ --event-ttl=1h \ --kubelet-https=true \ --insecure-bind-address=192.168.54.12 \ --runtime-config=rbac.authorization.k8s.io/v1alpha1 \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-cluster-ip-range=10.254.0.0/16 \ --service-node-port-range=30000-60000 \ --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --experimental-bootstrap-token-auth \ --token-auth-file=/etc/kubernetes/token.csv \ --v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target 二、 关闭了 RBAC # 自定义 系统 service 文件一般存于 /etc/systemd/system/ 下 vi /etc/systemd/system/kube-apiserver.service [Unit] Description=kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] User=root ExecStart=/usr/local/bin/kube-apiserver \ --storage-backend=etcd2 \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --advertise-address=192.168.54.12 \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/lib/audit.log \ --bind-address=192.168.54.12 \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --enable-swagger-ui=true \ --etcd-cafile=/etc/kubernetes/ssl/ca.pem \ --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \ --etcd-servers=http://192.168.54.12:2379,http://192.168.54.13:2379,http://192.168.54.14:2379 \ --event-ttl=1h \ --kubelet-https=true \ --insecure-bind-address=192.168.54.12 \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-cluster-ip-range=10.254.0.0/16 \ --service-node-port-range=30000-32000 \ --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --experimental-bootstrap-token-auth \ --token-auth-file=/etc/kubernetes/token.csv \ --v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target # 这里面要注意的是 --service-node-port-range=30000-32000 # 这个地方是 映射外部端口时 的端口范围,随机映射也在这个范围内映射,指定映射端口必须也在这个范围内。 (5).启动 kube-apiserver systemctl daemon-reload systemctl enable kube-apiserver systemctl start kube-apiserver systemctl status kube-apiserver --storage-backend=etcd2 (6).配置 kube-controller-manager # 创建 kube-controller-manager.service 文件 vi /etc/systemd/system/kube-controller-manager.service [Unit] Description=kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/usr/local/bin/kube-controller-manager \ --address=127.0.0.1 \ --master=http://192.168.54.12:8080 \ --allocate-node-cidrs=true \ --service-cluster-ip-range=10.254.0.0/16 \ --cluster-cidr=10.233.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --leader-elect=true \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target (7).启动 kube-controller-manager systemctl daemon-reload systemctl enable kube-controller-manager systemctl start kube-controller-manager systemctl status kube-controller-manager (8).配置 kube-scheduler # 创建 kube-cheduler.service 文件 vi /etc/systemd/system/kube-scheduler.service [Unit] Description=kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/usr/local/bin/kube-scheduler \ --address=127.0.0.1 \ --master=http://192.168.54.12:8080 \ --leader-elect=true \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target (9).启动 kube-scheduler systemctl daemon-reload systemctl enable kube-scheduler systemctl start kube-scheduler systemctl status kube-scheduler (10).验证 Master 节点 [root@k8s-master-1 opt]# kubectl get componentstatuses NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} 部署 kubernetes Node 节点 (首先部署 192.168.54.13) Node 节点 需要部署的组件有 docker flannel kubectl kubelet kube-proxy 这几个组件。 (1).配置 kubectl wget https://dl.k8s.io/v1.7.0/kubernetes-client-linux-amd64.tar.gz tar -xzvf kubernetes-client-linux-amd64.tar.gz cp kubernetes/client/bin/* /usr/local/bin/ chmod a+x /usr/local/bin/kube* # 验证安装 kubectl version Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} (2).配置 kubelet kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求,需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper 角色,然后 kubelet 才有权限创建认证请求(certificatesigningrequests)。 # 先创建认证请求 # user 为 master 中 token.csv 文件里配置的用户 # 只需在一个node中创建一次就可以 kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap (3).下载 二进制文件 cd /tmp wget https://dl.k8s.io/v1.7.0/kubernetes-server-linux-amd64.tar.gz tar zxvf kubernetes-server-linux-amd64.tar.gz cp -r kubernetes/server/bin/{kube-proxy,kubelet} /usr/local/bin/ (4).创建 kubelet kubeconfig 文件 # 配置集群 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.54.12:6443 \ --kubeconfig=bootstrap.kubeconfig # 配置客户端认证 kubectl config set-credentials kubelet-bootstrap \ --token=11849e4f70904706ab3e631e70e6af0d \ --kubeconfig=bootstrap.kubeconfig # 配置关联 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 配置默认关联 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig # 拷贝生成的 bootstrap.kubeconfig 文件 mv bootstrap.kubeconfig /etc/kubernetes/ (5).创建 kubelet.service 文件 # 创建 kubelet 目录 mkdir /var/lib/kubelet vi /etc/systemd/system/kubelet.service [Unit] Description=kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/usr/local/bin/kubelet \ --address=192.168.54.13 \ --hostname-override=192.168.54.13 \ --pod-infra-container-image=jicki/pause-amd64:3.0 \ --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \ --require-kubeconfig \ --cert-dir=/etc/kubernetes/ssl \ --cluster_dns=10.254.0.2 \ --cluster_domain=cluster.local. \ --hairpin-mode promiscuous-bridge \ --allow-privileged=true \ --serialize-image-pulls=false \ --logtostderr=true \ --v=2 ExecStopPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT ExecStopPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT ExecStopPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT ExecStopPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target # 如上配置: 192.168.54.13 为本机的IP 10.254.0.2 预分配的 dns 地址 cluster.local. 为 kubernetes 集群的 domain jicki/pause-amd64:3.0 这个是 pod 的基础镜像,既 gcr 的 gcr.io/google_containers/pause-amd64:3.0 镜像, 下载下来修改为自己的仓库中的比较快。 (6).启动 kubelet systemctl daemon-reload systemctl enable kubelet systemctl start kubelet systemctl status kubelet 1.如果服务启动日志出现如下;(报错1) 18000 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth" 18000 kubelet.go:517] Hairpin mode set to "hairpin-veth" 18000 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op k8s-new-cluster-node2 kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd" 新增配置参数; cat /etc/systemd/system/kubelet.service --cgroup-driver=systemd\ 1.如果服务启动日志出现如下;(报错2) ze:26214400 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:54:09 k8s-new-cluster-test-host01 kubelet: I1030 17:54:09.775906 22962 manager.go:222] Version: {KernelVersion:4.18.15-1.el7. elrepo.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:18.06.1-ce DockerAPIVersion:1.38 CadvisorVersion: CadvisorRevision:} Oct 30 17:54:09 k8s-new-cluster-test-host01 kubelet: I1030 17:54:09.777164 22962 server.go:422] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / Oct 30 17:54:09 k8s-new-cluster-test-host01 kubelet: error: failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename#011#011#011#011Type#011#011Size#011Used#011Priority /dev/sda4 partition#0118388604#0110#011-2] Oct 30 17:54:09 k8s-new-cluster-test-host01 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE Oct 30 17:54:09 k8s-new-cluster-test-host01 systemd: Unit kubelet.service entered failed state. Oct 30 17:54:09 k8s-new-cluster-test-host01 systemd: kubelet.service failed. [root@k8s-new-cluste]#swapoff -a [root@k8s-new-cluste]cat /etc/systemd/system/kubelet.service --fail-swap-on=false\ (7).配置 TLS 认证 # 查看 csr 的名称 [root@k8s-node1 ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-EUE41uO5bofZZ-7GKD_V31oHXsENKFXCkLPy6Dj35Sc 1m kubelet-bootstrap Pending # 增加 认证 kubectl certificate approve node-csr-EUE41uO5bofZZ-7GKD_V31oHXsENKFXCkLPy6Dj35Sc # 提示 certificatesigningrequest "node-csr-EUE41uO5bofZZ-7GKD_V31oHXsENKFXCkLPy6Dj35Sc" approved (8).验证 nodes [root@k8s-master-1 ~]# kubectl get nodes NAME STATUS AGE VERSION 192.168.54.13 Ready 33s v1.7.0 # 成功以后会自动生成配置文件与密钥 # 配置文件 ls /etc/kubernetes/kubelet.kubeconfig /etc/kubernetes/kubelet.kubeconfig # 密钥文件 ls /etc/kubernetes/ssl/kubelet* /etc/kubernetes/ssl/kubelet-client.crt /etc/kubernetes/ssl/kubelet.crt /etc/kubernetes/ssl/kubelet-client.key /etc/kubernetes/ssl/kubelet.key (9).配置 kube-proxy (10).创建 kube-proxy 证书 # 证书方面由于我们node端没有装 cfssl # 我们回到 master 端 机器 去配置证书,然后拷贝过来 [root@k8s-master-1 ~]# cd /opt/ssl vi kube-proxy-csr.json { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } (11).生成 kube-proxy 证书和私钥 /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/config.json \ -profile=kubernetes kube-proxy-csr.json | /opt/local/cfssl/cfssljson -bare kube-proxy # 查看生成 ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem # 拷贝到目录 cp kube-proxy*.pem /etc/kubernetes/ssl/ (12).拷贝到Node节点 scp kube-proxy*.pem 192.168.54.13:/etc/kubernetes/ssl/ scp kube-proxy*.pem 192.168.54.14:/etc/kubernetes/ssl/ (13).创建 kube-proxy kubeconfig 文件 # 配置集群 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.54.12:6443 \ --kubeconfig=kube-proxy.kubeconfig # 配置客户端认证 kubectl config set-credentials kube-proxy \ --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \ --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig # 配置关联 kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig # 配置默认关联 kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig # 拷贝到目录 mv kube-proxy.kubeconfig /etc/kubernetes/ (14).创建 kube-proxy.service 文件 # 创建 kube-proxy 目录 mkdir -p /var/lib/kube-proxy vi /etc/systemd/system/kube-proxy.service [Unit] Description=kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/usr/local/bin/kube-proxy \ --bind-address=192.168.54.13 \ --hostname-override=192.168.54.13 \ --cluster-cidr=10.254.0.0/16 \ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \ --logtostderr=true \ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target (150).启动 kube-proxy systemctl daemon-reload systemctl enable kube-proxy systemctl start kube-proxy systemctl status kube-proxy (16)部署其他Node 节点 (第二个节点部署 192.168.54.14) .........省略了....... 参照以上配置 (17).测试集群 # 创建一个 nginx deplyment apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-dm spec: replicas: 2 template: metadata: labels: name: nginx spec: containers: - name: nginx image: nginx:alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: name: nginx [root@k8s-master-1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-dm-2214564181-lxff5 1/1 Running 0 14m nginx-dm-2214564181-qm1bp 1/1 Running 0 14m [root@k8s-master-1 ~]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-dm 2 2 2 2 14m [root@k8s-master-1 ~]# kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.254.0.1 443/TCP 4h nginx-svc 10.254.129.54 80/TCP 15m # 在 node 里 curl [root@k8s-node2 ~]# curl 10.254.129.54 Welcome to nginx! Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.Thank you for using nginx.
配置 KubeDNS 官方 github yaml 相关 https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns (1).下载镜像 # 官方镜像(国内墙的原因导致下载失败) gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4 # 使用此地址pull;(选择其一) docker pull hongchhe/k8s-dns-sidecar-amd64:1.14.4 docker pull hongchhe/k8s-dns-kube-dns-amd64:1.14.4 docker pull hongchhe/k8s-dns-dnsmasq-nanny-amd64:1.14.4 docker pull jicki/k8s-dns-sidecar-amd64:1.14.4 docker pull jicki/k8s-dns-kube-dns-amd64:1.14.4 docker pull jicki/k8s-dns-dnsmasq-nanny-amd64:1.14.4 (2).下载 yaml 文件 curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-cm.yaml curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-sa.yaml curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-controller.yaml.base curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kubedns-svc.yaml.base # 修改后缀 mv kubedns-controller.yaml.base kubedns-controller.yaml mv kubedns-svc.yaml.base kubedns-svc.yaml (3).系统预定义的 RoleBinding 预定义的 RoleBinding system:kube-dns 将 kube-system 命名空间的 kube-dns ServiceAccount 与 system:kube-dns Role 绑定, 该 Role 具有访问 kube-apiserver DNS 相关 API 的权限; [root@k8s-master-1 kubedns]# kubectl get clusterrolebindings system:kube-dns -o yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: 2017-07-04T04:15:13Z labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-dns resourceVersion: "106" selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/system%3Akube-dns uid: 60c1e0e1-606f-11e7-b212-d4ae52d1f0c9 roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-dns subjects: - kind: ServiceAccount name: kube-dns namespace: kube-system (4).修改 kubedns-svc.yaml # kubedns-svc.yaml 中 clusterIP: __PILLAR__DNS__SERVER__ 修改为我们之前定义的 dns IP 10.254.0.2 cat kubedns-svc.yaml apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "KubeDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.254.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP (5).修改 kubedns-controller.yaml 1. # 修改 --domain=__PILLAR__DNS__DOMAIN__. 为 我们之前 预定的 domain 名称 --domain=cluster.local. 2. # 修改 --server=/__PILLAR__DNS__DOMAIN__/127.0.0.1#10053 中 domain 为我们之前预定的 --server=/cluster.local./127.0.0.1#10053 3. # 修改 --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.__PILLAR__DNS__DOMAIN__, 中的 domain 为我们之前预定的 --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local., 4. # 修改 --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.__PILLAR__DNS__DOMAIN__, 中的 domain 为我们之前预定的 --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local., (6).导入 yaml 文件 [root@k8s-master-1 kubedns]# kubectl create -f . configmap "kube-dns" created deployment "kube-dns" created serviceaccount "kube-dns" created service "kube-dns" created (7).查看 kubedns 服务 [root@k8s-master-1 kubedns]# kubectl get all --namespace=kube-system NAME READY STATUS RESTARTS AGE po/kube-dns-1511229508-llfgs 3/3 Running 0 1m NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kube-dns 10.254.0.253/UDP,53/TCP 1m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deploy/kube-dns 1 1 1 1 1m NAME DESIRED CURRENT READY AGE rs/kube-dns-1511229508 1 1 1 1m (8).验证 dns 服务 # 导入之前的 nginx-dm yaml文件 [root@k8s-master-1 ~]# kubectl get svc nginx-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-svc 10.254.79.137 80/TCP 29s # 创建一个 pods 来测试一下 nameserver apiVersion: v1 kind: Pod metadata: name: alpine spec: containers: - name: alpine image: alpine command: - sh - -c - while true; do sleep 1; done # 查看 pods [root@k8s-master-1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE alpine 1/1 Running 0 1m nginx-dm-2214564181-4zjbh 1/1 Running 0 5m nginx-dm-2214564181-tpz8t 1/1 Running 0 5m # 测试 [root@k8s-master-1 ~]# kubectl exec -it alpine ping nginx-svc PING nginx-svc (10.254.207.143): 56 data bytes [root@k8s-master-1 ~]# kubectl exec -it alpine nslookup nginx-svc nslookup: can't resolve '(null)': Name does not resolve Name: nginx-svc Address 1: 10.254.207.143 nginx-svc.default.svc.cluster.local 部署 Ingress 与 Dashboard (1).部署 dashboard 官方 dashboard 的github https://github.com/kubernetes/dashboard 这里注意,以下部署的应用为 api-service 关闭了 RBAC 的, 在开启了 RBAC 的情况下,无论是 dashboard 与 nginx ingress 都需要修改,默认是有问题的。 (2).下载 dashboard 镜像 # 官方镜像 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1 # 国内镜像 jicki/kubernetes-dashboard-amd64:v1.6.1 #内网镜像: reg.chehejia.com/k8s-web-ui:v1.6.1 (3).下载 yaml 文件 curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-controller.yaml curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-service.yaml (4).导入 yaml [root@k8s-master-1 dashboard]# kubectl apply -f . deployment "kubernetes-dashboard" created service "kubernetes-dashboard" created # 查看 svc 与 pod [root@k8s-master-1 dashboard]# kubectl get svc -n kube-system kubernetes-dashboard NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard 10.254.167.28 80/TCP 31s (5).部署 Nginx Ingress kubernetes 暴露服务的方式目前只有三种:LoadBlancer Service、NodePort Service、Ingress; 什么是 Ingress ? Ingress 就是利用 Nginx Haproxy 等负载均衡工具来暴露 kubernetes 服务。 官方 Nginx Ingress github https://github.com/kubernetes/ingress/tree/master/examples/deployment/nginx # 下载镜像 # 官方镜像 gcr.io/google_containers/defaultbackend:1.0 gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.10 # 国内镜像 jicki/defaultbackend:1.0 jicki/nginx-ingress-controller:0.9.0-beta.10 #内网镜像; reg.chehejia.com/ingress-defaultbackend:v1 reg.chehejia.com/nginx-ingress-controller:v0.9-beta.10 # 部署 Nginx backend , Nginx backend 用于统一转发 没有的域名 到指定页面。 curl -O https://raw.githubusercontent.com/kubernetes/ingress/master/examples/deployment/nginx/default-backend.yaml # 直接导入既可, 这里不需要修改 [root@k8s-master-1 ingress]# kubectl apply -f default-backend.yml deployment "default-http-backend" created service "default-http-backend" created # 查看服务 [root@k8s-master-1 ingress]# kubectl get deployment -n kube-system default-http-backend NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 36s # 部署 Ingress Controller 组件 # 下载 yaml 文件 curl -O https://raw.githubusercontent.com/kubernetes/ingress/master/examples/daemonset/nginx/nginx-ingress-daemonset.yaml # 导入 yaml 文件 [root@k8s-master-1 ingress]# kubectl apply -f nginx-ingress-daemonset.yaml daemonset "nginx-ingress-lb" created # 查看服务 [root@k8s-master-1 ingress]# kubectl get daemonset -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE nginx-ingress-lb 2 2 2 2 2 11s # 创建一个 ingress # 查看我们原有的 svc [root@k8s-master-1 Ingress]# kubectl get svc nginx-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-svc 10.254.207.143 80/TCP 1d # 创建 yaml 文件 vi nginx-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress spec: rules: - host: nginx.jicki.me http: paths: - backend: serviceName: nginx-svc servicePort: 80 # 导入 yaml [root@k8s-master-1 ingress]# kubectl apply -f nginx-ingress.yaml ingress "nginx-ingress" created # 查看 ingress [root@k8s-master-1 Ingress]# kubectl get ingress NAME HOSTS ADDRESS PORTS AGE nginx-ingress nginx.jicki.me 192.168.54.13,10... 80 24s # 测试访问 [root@k8s-master-1 ingress]# curl -I nginx.jicki.me HTTP/1.1 200 OK Server: nginx/1.13.2 Date: Thu, 06 Jul 2017 04:21:43 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Last-Modified: Wed, 28 Jun 2017 18:27:36 GMT ETag: "5953f518-264" Accept-Ranges: bytes # 配置一个 Dashboard Ingress # 查看 dashboard 的 svc [root@k8s-master-1 ingress]# kubectl get svc -n kube-system kubernetes-dashboard NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard 10.254.81.94 80/TCP 2h # 编辑一个 yaml 文件 vi dashboard-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: dashboard-ingress namespace: kube-system spec: rules: - host: dashboard.jicki.me http: paths: - backend: serviceName: kubernetes-dashboard servicePort: 80 # 查看 ingress [root@k8s-master-1 dashboard]# kubectl get ingress -n kube-system NAME HOSTS ADDRESS PORTS AGE dashboard-ingress dashboard.jicki.me 192.168.54.13,10... 80 1m # 测试访问 [root@k8s-master-1 dashboard]# curl -I dashboard.jicki.me HTTP/1.1 200 OK Server: nginx/1.13.2 Date: Thu, 06 Jul 2017 06:32:00 GMT Content-Type: text/html; charset=utf-8 Content-Length: 848 Connection: keep-alive Accept-Ranges: bytes Cache-Control: no-store Last-Modified: Tue, 16 May 2017 12:53:01 GMT