部署kubernetes集群

在本例中使用三台机器来部署kubernetes集群

172.16.36.50    master
172.16.36.51    cti-1
172.16.36.54    cti-4

关闭三台机器的防火墙

systemctl stop firewalld
systemctl disable firewalld

分别修改三台机器的selinux配置文件

[root@cti-m kubernetes]# vi /etc/sysconfig/selinux

==# This file controls the state of SELinux on the system.==

==# SELINUX= can take one of these three values:

==# enforcing - SELinux security policy is enforced.==

==# permissive - SELinux prints warnings instead of enforcing.==

==# disabled - No SELinux policy is loaded.==

==#SELINUX=enforcing==

SELINUX=disabled

==# SELINUXTYPE= can take one of three two values:==

==# targeted - Targeted processes are protected,==

==# minimum - Modification of targeted policy. Only selected processes are protected.==

==# mls - Multi Level Security protection.==

SELINUXTYPE=targeted

在三台机器的hosts文件中分别加入以下内容

[root@cti-m kubernetes]# vi /etc/hosts
172.16.36.50   master
172.16.36.51   cti-1
172.16.36.54   cti-4

一.安装配置master 1.安装kubernetes和etcd

yum -y install kubernetes etcd

2.编辑/etc/etcd/etcd.conf，确保etcd监听所有的ip地址，这里配置etcd的方法参见《etcd集群配置》

3.配置ServiceAccount和Secret 使用openssl工具在master服务器上创建证书和私钥相关的文件，分别执行以下命令：

[root@cti-m kubernetes]# mkdir /var/run/kubernetes
[root@cti-m kubernetes]# cd /var/run/kubernetes
[root@cti-m kubernetes]# openssl genrsa -out ca.key 2048
[root@cti-m kubernetes]# openssl req -x509 -new -nodes -key ca.key -subj "/CN=wecloud.com" -days 5000 -out ca.crt
[root@cti-m kubernetes]# openssl genrsa -out server.key 2048
[root@cti-m kubernetes]# openssl req -new -key server.key -subj "/CN=cti-m" -out server.csr
[root@cti-m kubernetes]# openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 5000

注意：在生成server.csr时-subj参数中/CN指定的名字为master的主机名。另外，在生成ca.crt时-subj参数中/CN的名字最好与主机名不同，设置为相同可能导致对普通master的https访问认证失败

执行完成后会生成6个文件：ca.crt、ca.key、ca.srl、server.crt、server.csr、server.key

4.配置kube-apiserver，编辑/etc/kubernetes/apiserver，需要修改的配置如下：

[root@cti-m kubernetes]# egrep -v "^#|^$" /etc/kubernetes/apiserver
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
KUBE_API_PORT="--port=8080"
KUBELET_PORT="--kubelet-port=10250"
KUBE_ETCD_SERVERS="--etcd-servers=http://172.16.36.50:2379,http://172.16.36.51:2379,http://172.16.36.54:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_API_ARGS="--client_ca_file=/var/run/kubernetes/ca.crt --tls-private-key-file=/var/run/kubernetes/server.key --tls-cert-file=/var/run/kubernetes/server.crt"

注意：如果之前在创建kubernetes集群时因为不需要安全认证而去掉ServiceAccount，在这里需要添加回来,如果重启虚拟机后，这些证书需要重新生成，不然apiserver不能启动

5.配置kube-controller-manager，编辑/etc/kubernetes/controller-manager，需要修改的配置如下：

[root@cti-m kubernetes]# egrep -v "^#|^$" /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS="--service_account_private_key_file=/var/run/kubernetes/server.key --root_ca_file=/var/run/kubernetes/ca.crt"

6.启动etcd, kube-apiserver, kube-controller-manager和kube-scheduler

for SERVICE in etcd kube-apiserver kube-controller-manager kube-scheduler; do
    systemctl restart $SERVICE  
    systemctl enable $SERVICE
	systemctl status $SERVICE  
done

在kube-apiserver服务成功启动后，系统会自动为每个命名空间创建一个ServiceAccount和一个Secret（包含一个ca.crt和一个token）

[root@cti-m kubernetes]# kubectl get serviceaccount --all-namespaces
NAMESPACE   NAME      SECRETS   AGE
default     default   1         1h

[root@cti-m kubernetes]# kubectl get secrets --all-namespaces
NAMESPACE   NAME                  TYPE                                  DATA      AGE
default     default-token-fq3j8   kubernetes.io/service-account-token   3         1h

[root@cti-m kubernetes]# kubectl describe secret default-token-fq3j8
Name:           default-token-fq3j8
Namespace:      default
Labels:         
Annotations:    kubernetes.io/service-account.name=default,kubernetes.io/service-account.uid=4c27b13e-ad51-11e6-a4d0-000c298207a9

Type:   kubernetes.io/service-account-token

Data
====
token:          eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tZnEzajgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRjMjdiMTNlLWFkNTEtMTFlNi1hNGQwLTAwMGMyOTgyMDdhOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.mKTF9y7kUzWomc2GBwUvvQ0vZZvbQ0ojH_1NBzzOqH4kaYHE545xkpRqeHxCq91h19aXVMkT96mkxLsn75mDPAMoxUt238YqYrUTvsDZJz8NYknXZ18AMfylJYQsLi_6KO4aE1z8hDh-5R-y5jKhQoAOnNmK8uJKRfGoDLweHHqYCCeNPH-hAqdh7eisIvjpsFgUFsBtCJPrwoNVRboMZSqjItE2YEd_y0sgWxAoK1SQqg2JN3zOY3l2RHHj9y48FEDWI5Cf3nY4CTEqv5n97iggnNTi9JhGEEOkK9ZockvqAYMv4luVDqmCud2nZmuVV26Igdyp6IiHvC6WX8jvFQ
ca.crt:         1099 bytes
namespace:      7 bytes

之后ReplicationController在创建Pod时，会生成类型为Secret的volume存储卷，并将该volume挂载到Pod内的如下目录中：/var/run/secrets/kubernetes.io/serviceaccount。然后，容器内的应用程序就可以使用该Secret与master建立https连接了。Pod的volume设置和挂载操作由ReplicationController和kubelet自动完成，可以通过查看Pod的详细信息了解到

7.在etcd里配置flannel网络

[root@cti-m kubernetes]#  etcdctl mk /flannel/network/config '{"Network":"172.17.0.0/16","SubnetMin":"172.17.1.0","SubnetMax":"172.17.254.0"}'

[root@cti-m kubernetes]# etcdctl ls /flannel --recursive
/flannel/network
/flannel/network/config

[root@cti-m kubernetes]# etcdctl get /flannel/network/config
{"Network":"172.17.0.0/16","SubnetMin":"172.17.1.0","SubnetMax":"172.17.254.0"}

二.安装配置minion 1.安装kubernetes和flannel

yum -y install flannel kubernetes

2.为flannel配置etcd服务，编辑/etc/sysconfig/flanneld，修改如下内容

[root@cti-1 run]# egrep -v "^#|^$" /etc/sysconfig/flanneld
FLANNEL_ETCD="http://172.16.36.50:2379,http://172.16.36.51:2379,http://172.16.36.54:2379"
FLANNEL_ETCD_KEY="/flannel/network"

3.编辑kubernetes的全局配置文件/etc/kubernetes/config，修改如下内容：

[root@cti-1 run]# egrep -v "^#|^$" /etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://master:8080"

4.编辑/etc/kubernetes/kubelet，修改如下内容：

[root@cti-1 run]# egrep -v "^#|^$" /etc/kubernetes/kubelet
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=cti-1"
KUBELET_API_SERVER="--api-servers=http://master:8080"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause:0.8.0"
KUBELET_ARGS=""

5.修改docker的配置文件/etc/sysconfig/docker（vi /usr/lib/systemd/system/docker.service），修改内容如下：

# /etc/sysconfig/docker


# Modify these options if you want to change the way the docker daemon runs

#OPTIONS='--selinux-enabled --log-driver=journald'

OPTIONS='-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock'
DOCKER_CERT_PATH=/etc/docker

如果还是不能启动docker engine并且显示是如下错误：

Job for docker.service failed because the control process exited with error code
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor prese
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─flannel.conf
   Active: failed (Result: exit-code) since 五 2016-11-18 22:32:26 CST; 101ms ag
     Docs: http://docs.docker.com
 Main PID: 22472 (code=exited, status=1/FAILURE)

11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.82231893
11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.82233861
11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.86238249
11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.86242715
11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.86245195
11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.88479188
11月 18 22:32:26 cti-4 systemd[1]: docker.service: main process exited, code=exi
11月 18 22:32:26 cti-4 systemd[1]: Failed to start Docker Application Container
11月 18 22:32:26 cti-4 systemd[1]: Unit docker.service entered failed state.
11月 18 22:32:26 cti-4 systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

解决方法：removing /var/lib/docker/network

6.启动flanneld, kube-proxy, kubelet和docker服务

for SERVICE in flanneld kube-proxy kubelet docker; do  
    systemctl restart $SERVICE  
    systemctl enable $SERVICE  
    systemctl status $SERVICE  
done

7.在master上查看节点信息：

[root@cti-m kubernetes]# kubectl get node
NAME      STATUS    AGE
cti-1     Ready     1h
cti-4     Ready     1h

8.判断节点是否能连接master：

[root@cti-4 ~]# curl -s -L http://172.16.36.50:2379/version
{"etcdserver":"2.3.7","etcdcluster":"2.3.0"}[root@cti-4 ~]#

9.在master节点查看flannel子网分配情况：

[root@cti-m kubernetes]# etcdctl ls /flannel/network/subnets
/flannel/network/subnets/172.17.58.0-24
/flannel/network/subnets/172.17.70.0-24