k8s环境部署及使用方式
安装k8s+docker集群环境（基于centos7系统）
集群机器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67
1.配置yum源
[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo
[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0
- 1
- 2
- 3
- 4
加载安装包：
[centos-master]:yum repolist
virt7-docker-common-release virt7-docker-common-release 15
2.安装集群必要软件—-etcd/flannel/kubernetes
Etcd服务在k8s集群中用于配置共享和服务发现。
Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。
[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel
3.如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：
echo "192.168.121.9 centos-master
192.168.121.65 centos-minion-1
192.168.121.66 centos-minion-2
192.168.121.67 centos-minion-3" >> /etc/hosts
- 1
- 2
- 3
- 4
4.修改配置master节点的kubernetes配置文件/etc/kubernetes/config
#表示错误日志记录到文件还是输出到stderr
KUBE_LOGTOSTDERR="--logtostderr=true"
#日志等级
KUBE_LOG_LEVEL="--v=0"
#允许运行特权容器
KUBE_ALLOW_PRIV="--allow-privileged=false"
#apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9
KUBE_MASTER="--master=http://192.168.121.9:8080"
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
5.k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则，并需关闭selinux。为确保k8s集群的正常运行，我们可以直接关闭iptables与seliinux服务。
setenforce 0
systemctl stop firewalld.service
systemctl stop firewalld.service
- 1
- 2
- 3
6.修改配置master节点的etcd配置文件/etc/etcd/etcd.conf
etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：
# [member]
#etcd名称
ETCD_NAME=default
#etcd数据存储位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#监听的端口
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
#[cluster]
#集群监听的端口
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
7.修改配置master节点的apiserver配置文件/etc/kubernetes/apiserver
#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0
KUBE_API_ADDRESS="--address=0.0.0.0"
#apiserver的监听端口，默认8080
KUBE_API_PORT="--port=8080"
#minion上kubelet监听的端口，默认10250
KUBELET_PORT="--kubelet-port=10250"
#etcd服务地址，前面已经启动了etcd服务,端口为2379
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"
#kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
#需要额外添加的配置项，简单地启用一个集群无需配置
KUBE_API_ARGS=""
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
8.启动并且配置etcd的网段，此网段一定是未被使用的
systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"
- 1
- 2
- 3
9.修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld
#etcd的访问地址及端口
FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379"
#服务范围
FLANNEL_ETCD_PREFIX="/kube-centos/network"
#其他
FLANNEL_OPTIONS=""
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
10.启动k8s集群
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
- 1
- 2
- 3
- 4
- 5
以上就是k8s，master节点的安装及配置
接下来，配置nodes节点
11.修改配置nodes节点kubelet的配置文件/etc/kubernetes/kubelet
#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0
KUBELET_ADDRESS="--address=0.0.0.0"
#监听的端口
KUBELET_PORT="--port=10250"
#apiserver的访问地址及端口
KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080"
#额外增加的参数
KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
12.修改配置nodes节点flanneld的配置文件/etc/sysconfig/flanneld
#etcd的访问地址及端口
FLANNEL_ETCD="http://192.168.121.9:2379"
#etcd服务范围
FLANNEL_ETCD_KEY="/kube-centos/network"
- 1
- 2
- 3
- 4
- 5
- 6
13.启动k8s集群服务
for SERVICES in kube-proxy kubelet flanneld docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
- 1
- 2
- 3
- 4
- 5
14.设置kubectl的配置文件
kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context
- 1
- 2
- 3
15.检查集群状态
[centos-master]:kubectl get nodes
NAME STATUS AGE VERSION
centos-minion-1 Ready 3d v1.5.0+fff5156
centos-minion-2 Ready 3d v1.5.0+fff5156
centos-minion-3 Ready 3d v1.5.0+fff5156
- 1
- 2
- 3
- 4
- 5
至此，集群构建完毕
搭建私有库
私有库用于系统内部存储成品镜像，能够快速进行下载及被k8s调度。
1.下载并启动私有库
[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry
#--name 表示启动的容器后名称，此处为registry
#-v 表示挂载路径 格式为宿主机路径:容器内路径
#-p 表示映射端口 格式为宿主机端口:容器内端口
#-itd docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry
- 1
- 2
- 3
- 4
- 5
- 6
2.创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务
[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker [email protected]
[centos-master]:kubectl get secret
NAME TYPE DATA AGE
registrykey kubernetes.io/dockercfg 1 6s
- 1
- 2
- 3
- 4
- 5
- 6
此时登录时会提示认证错误
[centos-master]:docker login -u docker -p docker -e [email protected] registry.evehicle.cn
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized
- 1
- 2
- 3
- 4
这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用tls。这样我们就需要为Registry配置tls所需的key和crt文件了
3.配置nginx反向代理
[centos-master]: cat registry.evehicle.cn.conf
# For versions of nginx > 1.3.9 that include chunked transfer encoding support
# Replace with appropriate values where necessary
upstream docker-registry {
server 192.168.121.9:5000;
#server 10.44.170.95:5000;
}
# uncomment if you want a 301 redirect for users attempting to connect
# on port 80
# NOTE: docker client will still fail. This is just for convenience
# server {
# listen *:80;
# server_name my.docker.registry.com;
# return 301 https://$server_name$request_uri;
# }
server {
listen 443;
server_name registry.evehicle.cn;
ssl on;
ssl_certificate ssl/registry.evehicle.cn.crt;
ssl_certificate_key ssl/registry.evehicle.cn.key;
client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location / {
auth_basic "Restricted";
auth_basic_user_file passwd;
add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
location /_ping {
auth_basic off;
include docker-registry.conf;
}
location /v1/_ping {
auth_basic off;
include docker-registry.conf;
}
location /v2/_ping {
auth_basic off;
include docker-registry.conf;
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
将key及crt证书文件放到../ssl目录下。使用htpasswd生成密码放于./上一级目录
htpasswd -bcm passwd docker docker
#-c：创建一个加密文件
#-m：md5加密，默认可不填写
#-b：表示用户名密码在命令行中一并输入，不用分别填写
- 1
- 2
- 3
- 4
4.再次登录
[centos-master]:docker login -u docker -p docker -e [email protected] registry.evehicle.cn
Login Succeeded
表示成功，此时再pull\push既在私有库中进行
- 1
- 2
- 3
- 4
构建服务
docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：
1.从docker的公有库里下载centos7的原生镜像
[centos-master]:docker pull centos
Using default tag: latest
Trying to pull repository docker.io/library/centos ...
latest: Pulling from docker.io/library/centos
d9aaf4d82f24: Downloading [> ] 540 kB/73.39 MB
d9aaf4d82f24: Pulling fs layer
Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e
Status: Downloaded newer image for centos:latest
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
2.编写Dockerfile制造apache基础镜像
######httpd####
FROM centos
MAINTAINER lienhua lienhua@zhongchuangsanyou.com
RUN yum -y install epel-release
RUN yum -y install httpd php php-mysql php-memcache* php-mbstring
ADD httpd.conf /etc/httpd/conf/httpd.conf
EXPOSE 80
CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
其中httpd.conf文件需要在当前目录下真实存在，此处其内容为
ServerRoot "/etc/httpd"
Listen 80
Listen 8080
Include conf.modules.d/*.conf
Include zcsy/*.conf
User apache
Group apache
ServerAdmin root@localhost
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）
3.将制作好的镜像上传到私有库
docker push registry.evehicle.cn/httpd
- 1
4.编写启动apache服务的yaml文件
[centos-master]:cat 13-rc-httpd.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: 13-rc-httpd
labels:
name: 13-rc-httpd
spec:
replicas: 2
selector:
name: 13-rc-httpd
template:
metadata:
labels:
name: 13-rc-httpd
spec:
containers:
- name: 13-rc-httpd
image: registry.evehicle.cn/httpd
env:
- name: LANG
value: en_US.UTF-8
ports:
- containerPort: 80
hostPort: 80
volumeMounts:
- name: time
mountPath: /etc/localtime
- name: zcsy
mountPath: /etc/httpd/zcsy
- name: deploy
mountPath: /docker/httpd/deploy
- name: log
mountPath: /var/log/httpd
volumes:
- name: time
hostPath:
path: /etc/localtime
- name: zcsy
hostPath:
path: /docker/httpd/zcsy
- name: deploy
hostPath:
path: /docker/httpd/deploy
- name: log
hostPath:
path: /docker/httpd/log
nodeSelector:
slave: "13"
imagePullSecrets:
- name: registrykey
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
5.给其中一个node加上标签为“13”
kubectl label nodes centos-minion-1 slave=13
- 1
6.此时拥有标签“13”的nodes应具备的条件
/docker/httpd/zcsy下需要有官网的配置文件
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
以及/docker/httpd/deploy下需要有官网的代码
7.运行yaml文件启动容器
[centos-master]: kuberctl create -f 13-rc-httpd.yaml
8.查看服务
[centos-master]: kuberctl get rc
NAME DESIRED CURRENT AGE
13-rc-httpd 2 2 168d
- 1
- 2
9.程序中涉及的mysql\redis\memcache等服务也需使用容器运行起来
[centos-master]: docker pull redis
[centos-master]: docker tag registry.evehicle.cn/redis redis
[centos-master]: docker push registry.evehicle.cn/redis
[centos-master]: kubectl create -f rc-redis.yaml
[centos-master]: cat rc-redis.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: redis
labels:
name: redis
spec:
replicas: 2
selector:
name: redis
template:
metadata:
labels:
name: redis
spec:
containers:
- name: redis
image: registry.evehicle.cn/redis
ports:
- containerPort: 6379
hostPort: 6379
volumeMounts:
- name: data
mountPath: /data
- name: time
mountPath: /etc/localtime
volumes:
- name: data
hostPath:
path: /docker/redis/6379
- name: time
hostPath:
path: /etc/localtime
nodeSelector:
slave: "13"
imagePullSecrets:
- name: registrykey
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
启动memcache
[centos-master]: docker pull memcache
[centos-master]: docker tag registry.evehicle.cn/memcached memcache
[centos-master]: docker push registry.evehicle.cn/memcached
[centos-master]: kubectl create -f rc-memcached.yaml
[centos-master]: cat rc-memcached.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: memcached
labels:
name: memcached
spec:
replicas: 3
selector:
name: memcached
template:
metadata:
labels:
name: memcached
spec:
containers:
- name: memcached
image: registry.evehicle.cn/memcached
ports:
- containerPort: 11211
hostPort: 11211
#nodeSelector:
# slave: "13"
imagePullSecrets:
- name: registrykey
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
制造mysql镜像
[centos-master]: cat Dockerfile
FROM alpine
COPY startup.sh /startup.sh
RUN addgroup mysql && \
adduser -H -D -s /bin/false -G mysql mysql && \
apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \
mkdir /data && \
chown -R mysql:mysql /data /etc/mysql && \
chmod 755 /startup.sh \
;
WORKDIR /data
VOLUME /data
VOLUME /etc/mysql
EXPOSE 3306
CMD ["/startup.sh"]
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
启动mysql（建议mysql在宿主机启动）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: 13-rc-mysql
labels:
name: 13-rc-mysql
spec:
replicas: 2
selector:
name: 13-rc-mysql
template:
metadata:
labels:
name: 13-rc-mysql
spec:
containers:
- name: 13-rc-mysql
image: registry.evehicle.cn/mysql
env:
- name: MYSQL_DATABASE
value: admin
- name: MYSQL_USER
value: tony
- name: MYSQL_PASSWORD
value: 456
- name: MYSQL_ROOT_PASSWORD
value: 123
ports:
- containerPort: 3306
hostPort: 3306
volumeMounts:
- name: time
mountPath: /etc/localtime
- name: data
mountPath: /data
- name: etc
mountPath: /etc/mysql
- name: run
mountPath: /run/mysqld
volumes:
- name: time
hostPath:
path: /etc/localtime
- name: data
hostPath:
path: /docker/mysql/data
- name: etc
hostPath:
path: /docker/mysql/etc
- name: run
hostPath:
path: /docker/mysql/run
nodeSelector:
slave: "13"
imagePullSecrets:
- name: registrykey
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
为方便代码编写及统一管理，应提前做好内部DNS解析。将所负责的应用规整到对应的机器上。