ClusterIp：通过VIP来访问,
NodePort: 需要自己搭建负载据衡器
LoadBalancer：仅仅用于特定的云提供商 和 Google Container Engine
https://www.nginx.com/blog/load-balancing-kubernetes-services-nginx-plus/
port：相当于服务端口（对及集群内客户访问）
targetPort: 相当于pods端口
nodePort； 宿主机端口（也是服务端口，只不过是对集群外客户访问）
hostPort: 相当于docker run 中的 -p (localhost 端口)
访问流程：
client ——》nodePort ——》node local random Port ——》Cluster-Ip Port ——》targetPort ——》containerPort
服务发现有两种：
环境变量：默认支持，
DNS：需要以插件的形式安装skyDNS
注：
环境变量方式存在限制：Pod必须在Service之后创建，DNS则没有这个限制
kubenetes（ppc64le）:
0. 关闭firewalld服务器
a. systemctl disable firewalld
b. systemctl stop firewalld
1. 在所有上安装at9.0:
# yum install -y advance-toolchain-at9.0-runtime \
advance-toolchain-at9.0-devel \
advance-toolchain-at9.0-perf \
advance-toolchain-at9.0-mcore-libs
# echo "export PATH=/opt/at9.0/bin:/opt/at9.0/sbin:$PATH" >> /etc/profile.d/at9.sh
# source /etc/profile.d/at9.sh
# /opt/at9.0/sbin/ldconfig
2. master节点：
a. # git clone https://github.com/Pensu/pause.git
# cd pause
# make
b. yum install kubernetes-client kubernetes-master etcd
c. 配置/etc/kubernetes/config:
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://172.16.9.158:8080"
d. 配置/etc/kubernetes/apiserver ：
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
#KUBE_API_PORT="--port=8080"
# KUBELET_PORT="--kubelet-port=10250"
KUBE_ETCD_SERVERS="--etcd-servers=http://172.16.9.158:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""
e. 配置/etc/etcd/etcd.conf：
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
f. 启动服务：
# for SERVICES in kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
3. 配置Minion：
a. yum install docker-io kubernetes-client kubernetes-node
b. 配置 /etc/kubernetes/kubelet：
KUBELET_ADDRESS="--address=0.0.0.0"
# KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=172.16.9.158"
KUBELET_API_SERVER="--api-servers=http://172.16.9.158:8080"
KUBELET_ARGS="--pod-infra-container-p_w_picpath=docker.repo:5000/kube/pause:0.8.0"
c. 启动服务：
for SERVICES in kube-proxy kubelet flanneld docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
4. 应用：
a. 编写replicationControl：
cat > httprc.yml < apiVersion: v1 kind: ReplicationController metadata: name: my-httpd spec: replicas: 2 template: metadata: labels: app: httpd spec: containers: - name: httpd p_w_picpath: ppc64le/httpd ports: - containerPort: 80 EOF b. 编写Service： cat > httpsvc-nodeport.yml < apiVersion: v1 kind: Service metadata: name: httpdsvc-nodeport spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP nodePort: 30080 selector: app: httpd EOF c. 分别创建replicationControl和Service： kubectl create -f httprc.yml kubectl create -f httpsvc-nodeport.yml d. 查看应用： kubectl get pod kubectl get rc kubectl get svc 5. flannel安装与配置： http://blog.shippable.com/kubernetes-cluster-with-flannel-overlay-network a. yum -y install flannel (在minion节点) b. 设置通讯子网 etcdctl set /atomic.io/network/config '{"Network":"10.10.0.0/16", "Backend":{"Type":"vxlan"}}' (在master节点) curl http://172.16.0.204:4001/v2/keys//atomic.io/network/config -XPUT -d value='{"Network":"10.10.0.0/16", "Backend":{"Type":"vxlan"}}' c. 配置/etc/sysconfig/flanneld FLANNEL_ETCD="http://172.16.9.158:2379" FLANNEL_ETCD_KEY="/atomic.io/network" #FLANNEL_OPTIONS=""
d. service flanneld start; e. 修改flannel.conf为以下内容： cat >/usr/lib/systemd/system/docker.service.d/flannel.conf< [Service] EnvironmentFile=-/run/flannel/docker ExecStart= ExecStart=/usr/bin/docker daemon $DOCKER_NETWORK_OPTIONS -H fd:// EOF 注：添加此行的目的主要是要添加 $DOCKER_NETWORK_OPTIONS 此变量，由于默认安装的docker-io(1.9.1，此版本在/etc/sysconfig/下也仅有docker文件，并没有像其它的rpm安装包一样，包含docker-network和docker-storage等等文件，因此也没有定义相关的变量，当然这些文件必须在docker.service里面使用EnvironmentFile来引用)里面的启动服务脚本中并没有添加此变量，所以要重新进行添加，而此处的这个变量是由flannel（具体是由mk-docker-opts.sh）产生的. f. systemctl deamon-reload; service docker restart 防火墙（公共）要开启端口的服务有： kube-apiserver: 默认是8080 kube-proxy: 对应nodePort etcd: 默认是2379 flanneld：默认bankend，使用UDP(8285)来封装数据报文，建议使用另一种后端类型：vxlan(8472)，有公司评测过，不管是在兼容性和性能上都比较好，评测网址：https://blog.tingyun.com/web/article/detail/406 glusterfs: glusterd(24007)、 glusterfsd(49152)、 rpcbind(111) cAdvisor: 4194 skydns: 53 kubelet: 10250 kube2sky: 8081 skydns和kube2sky: 1. 拷贝skydns和kube2sky二进制程序到/usr/local/bin cp skydns kube2sky /usr/local/bin 2. 在etcd里配置skydns: curl http://172.16.0.204:4001/v2/keys/skydns/config -XPUT -d value='{"addr_dns": "192.168.33.67:53", "ttl": 3600, "domain": "cluster.local.", "nameservers": ["219.141.136.10:53", "114.114.114.114:53"]}' 3. 分别配置 skydns和kube2sky的服务启动脚本 a. /usr/lib/systemd/system/skydns.service: [Unit] Description=SkyDNS service [Service] ExecStart=/usr/local/bin/skydns -machines=http://172.16.0.204:4001 Restart=on-failure [Install] WantedBy=multi-user.target b. /usr/lib/systemd/system/kube2sky.service: [Unit] Description=Kube2sky service After=skydns.service kube-apiserver.service [Service] ExecStart=/usr/local/bin/kube2sky \ --domain=cluster.local \ --etcd-server=http://172.16.0.204:4001 \ --kube-master-url=http://172.16.1.27:8080 Restart=on-failure [Install] WantedBy=multi-user.target 管理node的标签 使用kubectl label nodes {nodename} {key=value} 进行标签的添加。如： kubectl label nodes 10.126.72.31 points=test Kubernetes的持久存储Glusterfs： 在minion节点上: 1. 更改/etc/kubernetes/config为如下内容： KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=0" KUBE_ALLOW_PRIV="--allow-privileged=true"(此项只有在glusterfs作为kubeneters的pod时使用) KUBE_MASTER="--master=http://172.16.9.158:8080" 2. 更改/etc/kubernetes/kubelet为如下内容： KUBELET_ADDRESS="--address=0.0.0.0" KUBELET_PORT="--port=10250" KUBELET_HOSTNAME="--hostname-override=172.16.9.158" KUBELET_API_SERVER="--api-servers=http://172.16.9.158:8080" KUBELET_ARGS="--pod-infra-container-p_w_picpath=docker.repo:5000/kube/pause:0.8.0 \ --register-node=true \ --host-network-sources=*"(此项只有在glusterfs作为kubeneters的pod时使用) 3. 配置glusterfs的yum repo： cat > glusterfs.repo < # Place this file in your /etc/yum.repos.d/ directory [glusterfs-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/epel-$releasever/$basearch/ enabled=1 skip_if_unavailable=1 gpgcheck=1 gpgkey=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/rsa.pub [glusterfs-noarch-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/epel-$releasever/noarch enabled=1 skip_if_unavailable=1 gpgcheck=1 gpgkey=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/rsa.pub [glusterfs-source-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. - Source baseurl=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/epel-$releasever/SRPMS enabled=0 skip_if_unavailable=1 gpgcheck=1 gpgkey=http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/rsa.pub EOF 4. glusterfs服务端的软件安装： yum -y install glusterfs-server 5. glusterfs客户端的软件安装： yum -y install glusterfs glusterfs-fuse 6. 在glusterfs服务端启动glusterd服务: service glusterd start 7. 配置hosts文件： a. 在gluster1节点上，配置hosts文件如下： cat >> /etc/hosts < 127.0.0.1 gluster1 172.16.12.10gluster2 EOF b. 在gluster2节点上，配置hosts文件如下： cat >> /etc/hosts < 172.16.12.9gluster1 127.0.0.1gluster2 EOF 8. 组建glusterfs集群： a. gluster peer probe gluster2 b. 查看集群节点状态： gluster peer status 9. 需要一个独立的分区，用于glusterfs创建数据卷,此处假设为/dev/vdb a. mkdir /brick；mkfs.xfs /dev/vdb1 b. mount /dev/vdb1 /brick c. 创建glusterfs卷： gluster volume create gv1 replica 2 gluster1:/brick/gv1 gluster2:/brick/gv1 d. 启动gv1卷： gluster volume start gv1 c. 查看gv1卷的状态： gluster volume info 10. 客户端挂载测试： a. 客户端先配置好hosts cat >> /etc/hosts < 172.16.12.9gluster1 172.16.12.10gluster2 EOF b. mount -t glusterfs gluster1:/gv1 /mnt 11. 与k8s集成： a. 创建glusterfs的endpoints(相当于glusterfs文件系统的入口)： cat > glusterfs-endpoints.yml < apiVersion: v1 kind: Endpoints metadata: name: glusterfs-cluster subsets: - addresses: - ip: 172.16.12.9 ports: - port: 1 - addresses: - ip: 172.16.12.10 ports: - port: 1 EOF b. 配置glusterfs-endpoints的services： cat > glusterfs-service.yml < apiVersion: v1 kind: Service metadata: name: glusterfs-cluster spec: ports: - port: 1 EOF c. 部署用glusterfs做后端存储的应用，此处用ppc64le/httpd来举例： cat > apache-deployment.yml < apiVersion: extensions/v1beta1 kind: Deployment metadata: name: httpd-deployment spec: replicas: 2 template: metadata: labels: app: apache spec: containers: - name: apache p_w_picpath: ppc64le/httpd p_w_picpathPullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - mountPath: /usr/local/apache2/htdocs name: glusterfsvol volumes: - name: glusterfsvol glusterfs: endpoints: glusterfs-cluster path: gv1 EOF d. 定义ppc64le/httpd应用对应的Service cat > apache-service.yml < apiVersion: v1 kind: Service metadata: name: apache-service spec: type: NodePort ports: - port: 80 targetPort: 80 nodePort: 30080 selector: app: apache EOF e. 依次执行以下创建： kubectl create -f glusterfs-endpoints.yml kubectl create -f glusterfs-service.yml kubectl create -f apache-deployment.yml kubectl create -f apache-service.yml Kubernetes集群的容器化部署方法：
master节点： 1. 安装kubernetes-node、kubernetes-client、etcd和flannel a. yum -y install etcd flannel kubernetes-node kubernetes-client b. 配置flannel 和 docker： 同普通方式的kube的部署方法，如果docker版本是>1.11.* 那么在docker.service 里面还要加上： --exec-root=/var/lib/docker 2. 修改/etc/kubernetes/kubelet如下： KUBELET_ADDRESS="--address=0.0.0.0" KUBELET_HOSTNAME="--hostname-override=172.16.8.68" KUBELET_API_SERVER="--api-servers=http://172.16.8.68:8080" KUBELET_ARGS="--pod-infra-container-p_w_picpath=docker.repo:5000/kube/pause:0.8.0 \ --config=/etc/kubernetes/manifests \ --register-schedulable=false \ --cluster-dns=10.254.53.53 \ --cluster-domain=cluster.local" 3. 修改/etc/kubernetes/config如下： KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=0" KUBE_ALLOW_PRIV="--allow-privileged=true" KUBE_MASTER="--master=http://172.16.8.68:8080" 4. 在/usr/lib/systemd/system/kubelet.service的[Service]段添加如下内容： ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests 5. 配置/etc/kubernetes/manifests/kube-apiserver.yaml:
apiVersion: v1 kind: Pod metadata: name: kube-apiserver namespace: kube-system spec: hostNetwork: true containers: - name: kube-apiserver p_w_picpath: index.tenxcloud.com/google_containers/kube-apiserver-ppc64le:v1.3.0 command: - /usr/local/bin/kube-apiserver - --insecure-bind-address=0.0.0.0 - --etcd-servers=http://172.16.11.244:4001 - --advertise-address=172.16.11.244 - --allow-privileged=true - --service-cluster-ip-range=10.254.0.0/16 - --admission-control=NamespaceLifecycle,LimitRanger,ResourceQuota - --runtime-config=extensions/v1beta1=true, extensions/v1beta1/thirdpartresources=true ports: - containerPort: 8080 hostPort: 8080 name: local 6. 配置/etc/kubernetes/manifests/kube-proxy.yaml:
apiVersion: v1 kind: Pod metadata: name: kube-proxy namespace: kube-system spec: hostNetwork: true containers: - name: kube-proxy p_w_picpath: index.tenxcloud.com/google_containers/kube-proxy-ppc64le:v1.3.0 command: - /usr/local/bin/kube-proxy - --master=http://172.16.11.244:8080 - --proxy-mode=iptables securityContext: privileged: true 7. 配置/etc/kubernetes/manifests/kube-controller-manager.yaml: apiVersion: v1 kind: Pod metadata: name: kube-controller-manager namespace: kube-system spec: hostNetwork: true containers: - name: kube-controller-manager p_w_picpath: index.tenxcloud.com/google_containers/kube-controller-manager-ppc64le:v1.3.0 command: - /usr/local/bin/kube-controller-manager - --master=http://172.16.11.244:8080 - --leader-elect=true 8. 配置/etc/kubernetes/manifests/kube-scheduler.yaml: apiVersion: v1 kind: Pod metadata: name: kube-scheduler namespace: kube-system spec: hostNetwork: true containers: - name: kube-scheduler p_w_picpath: index.tenxcloud.com/google_containers/kube-scheduler-ppc64le:v1.3.0 command: - /usr/local/bin/kube-scheduler - --master=http://172.16.11.244:8080 - --leader-elect=true 9. 启动kubelet服务： systemctl start kubelet 10. 验证apiserver端口8080是否开启： ss -an src :8080 minion节点： 1. 安装kubernetes-node、kubernetes-client和flannel yum -y install flannel kubernetes-node kubernetes-client 2. 修改/etc/kubernetes/kubelet如下： KUBELET_ADDRESS="--address=0.0.0.0" KUBELET_HOSTNAME="--hostname-override=` inventory_hostname `" KUBELET_API_SERVER="--api-servers=http://172.16.8.68:8080" KUBELET_ARGS="--pod-infra-container-p_w_picpath=docker.repo:5000/kube/pause:0.8.0 \ --config=/etc/kubernetes/manifests \ --register-node=true \ --cluster-dns=10.254.53.53 \ --cluster-domain=cluster.local" 3. 修改/etc/kubernetes/config如下： 同master节点 4. 在/usr/lib/systemd/system/kubelet.service的[Service]段添加如下内容： 同master节点 5. 配置/etc/kubernetes/manifests/kube-proxy.yaml: apiVersion: v1 kind: Pod metadata: name: kube-proxy namespace: kube-system spec: hostNetwork: true containers: - name: kube-proxy p_w_picpath: index.tenxcloud.com/google_containers/kube-proxy-ppc64le:v1.3.0 command: - /usr/local/bin/kube-proxy - --master=http://172.16.11.244:8080 - --proxy-mode=iptables securityContext: privileged: true 6. 启动服务： a. systemctl daemon-reload b. for i in flanneld docker kubelet; do systemctl enable $i systemctl restart $i systemctl status $i done 7. 在master节点查看minion节点是否注册： kubectl get nodes