ingress做为k8s集群的入口非常重要，能实现ingress功能的软件很多，可根据自身需求选择。本篇博客主要使用nginx官方提供的nginx-ingress完成了http/https7层代理和tcp四层代理的环境配置。
系统环境
1，k8s的版本为1.8.2
2，docker ce的版本为19.03.8-3
3，五台主机操作系统版本为centos7，kernel版本3.10.0-957
4，使用五台主机部署，3台master节点+2台work节点
5，Ingress测试域名t.myk8s.com
部署nginx和apache应用
使用Deployment部署nginx
nginx服务的manifest：
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.17
livenessProbe: # 设置存活探针，如果检测失败则重启pod
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
httpGet:
path: /
port: 80
scheme: HTTP
ports:
- containerPort: 80 # 设置为pod中应用监听的端口
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
ports:
- name: http
port: 80 # service服务访问的端口
protocol: TCP
targetPort: 80 # 设置为pod中应用监听的端口
selector:
app: nginxapache服务的manifests：
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: httpd
spec:
selector:
matchLabels:
app: httpd
replicas: 1 # tells deployment to run 2 pods
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: httpd:2.4
livenessProbe: # 设置存活探针，如果检测失败则重启pod
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
httpGet:
path: /
port: 80
scheme: HTTP
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: httpd
namespace: default
labels:
app: httpd
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: httpd通过manifest配置k8s，使用--record选项记录命令和版本信息方便后续的升级或者回滚等操作：
# kubectl apply -f nginx-dp.yaml --record
# kubectl apply -f httpd-dp.yaml --record查看应用状态：
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-7c456f6cf9-gb7gk 1/1 Running 0 3m15s 10.107.199.77 work2
nginx-5577575ddd-499wr 1/1 Running 0 46m 10.109.125.196 work3
nginx-5577575ddd-7zwmv 1/1 Running 0 48m 10.99.1.85 work1 查看services状态，通过services暴露出来的固定ip可以实现对应应用的访问：
# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
httpd ClusterIP 10.104.235.200 80/TCP 45m
kubernetes ClusterIP 10.96.0.1 443/TCP 6d5h
nginx ClusterIP 10.111.5.31 80/TCP 57m Ingress配置http访问应用
Ingress控制器的部署方式使用daemon-set+hostNetwork方式，且使用nginx官方提供的nginx-ingress控制器（https://blog.51cto.com/leejia/2495558）。
生成配置Ingress的manifests：
# vim http-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: http-ingress
spec:
rules:
- host: t.myk8s.com
http:
paths:
- path: /index.html
backend:
serviceName: nginx
servicePort: 80
- path: /
backend:
serviceName: httpd
servicePort: 80
# kubectl apply -f http-ingress.yaml查看状态：
# kubectl get ingress -o wide
NAME CLASS HOSTS ADDRESS PORTS AGE
http-ingress t.myk8s.com 80 102m master集群中其中一台master的ip为172.18.2.175，在本地pc测试访问:
~ curl -H 'Host: t.myk8s.com' 172.18.2.175
It works!
➜ ~ curl -H 'Host: t.myk8s.com' 172.18.2.175/index.html
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
Ingress配置https访问应用
我们使用自签名证书来实现https的访问：
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ingress.key -out ingress.crt -subj "/CN=t.myk8s.com/O=t.myk8s.com"通过secret存储证书:
# kubectl create secret tls ingress-secret --key ingress.key --cert ingress.crt
# kubectl get secrets
NAME TYPE DATA AGE
ingress-secret kubernetes.io/tls 2 2s注意如下ingress资源要和上面的secret资源在同一个namespace下面：
# vim nginx-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: http-ingress
spec:
tls:
- hosts:
- t.myk8s.com
secretName: ingress-secret
rules:
- host: t.myk8s.com
http:
paths:
- path: /index.html
backend:
serviceName: nginx
servicePort: 80
- path: /
backend:
serviceName: httpd
servicePort: 80使用本地pc测试访问，默认http访问也会强制转https：
➜ ~ curl https://t.myk8s.com -k
It works!
➜ ~ curl http://t.myk8s.com -I -k
HTTP/1.1 301 Moved Permanently
Server: nginx/1.17.10
Date: Thu, 21 May 2020 03:53:06 GMT
Content-Type: text/html
Content-Length: 170
Connection: keep-alive
Location: https://t.myk8s.com:443/
➜ ~ curl https://t.myk8s.com/index.html -k
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
可以通过配置nginx-ingress控制器实现http和https同时支持访问，config map的name以及namespace要和nginx -ingress控制器使用的相同：
# vim nginx-cf.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
data:
ssl-redirect: "false"
# kubectl apply -f nginx-cf.yaml -n nginx-ingress
# kubectl get configmap -n nginx-ingress
NAME DATA AGE
nginx-config 1 6d19h默认nginx-ingress控制器不提供tcp和udp代理功能，但是nginx是支持的，故也可以通过调整nginx-ingress控制器来提供tcp和udp代理服务
配置nginx-ingress做4层tcp代理
打开nginx官方的nginx-ingress仓库，然后修改mainifests，使nginx加载tcp和udp代理的相关配置（即在nginx-ingress.yaml的args下面添加global-configuration配置）：
# git clone https://github.com/nginxinc/kubernetes-ingress/
# cd kubernetes-ingress/deployments
# git checkout v1.7.0
# vim daemon-set/nginx-ingress.yaml
- -global-configuration=$(POD_NAMESPACE)/nginx-configuration
# kubectl apply -f daemon-set/nginx-ingress.yaml生成tcp代理相关的manifests：
# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
name: nginx-configuration
namespace: nginx-ingress
spec:
listeners:
- name: nginx-tcp
port: 8000
protocol: TCP
# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
name: nginx-configuration
namespace: nginx-ingress
spec:
listeners:
- name: nginx-tcp
port: 8000
protocol: TCP
[root@master1 manifests]# cat transport-server-passthrough.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
name: nginx-tcp
spec:
listener:
name: nginx-tcp
protocol: TCP
upstreams:
- name: nginx-app
service: nginx
port: 80
action:
pass: nginx-app
# kubectl apply -f global-configuration.yaml
# kubectl apply -f transport-server-passthrough.yaml在本地pc测试访问
➜ ~ curl http://172.18.2.175:8000/index.html
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
手动对nginx-ingress的后端nginx服务扩缩容和升级回滚
由2台水平扩容为3台，扩容期间服务无影响：
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-7c456f6cf9-gb7gk 1/1 Running 0 27h 10.107.199.77 work2
nginx-5577575ddd-pz59n 1/1 Running 0 18s 10.99.1.91 work1
nginx-5577575ddd-qsrb2 1/1 Running 0 18s 10.109.125.207 work3
# kubectl scale deployment nginx --replicas 3
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-7c456f6cf9-gb7gk 1/1 Running 0 27h 10.107.199.77 work2
nginx-5577575ddd-j4nq4 1/1 Running 0 13s 10.107.199.84 work2
nginx-5577575ddd-pz59n 1/1 Running 0 68s 10.99.1.91 work1
nginx-5577575ddd-qsrb2 1/1 Running 0 68s 10.109.125.207 work3 由3台水平缩容为1台，缩容期间服务无影响：
# kubectl scale deployment nginx --replicas 1
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-7c456f6cf9-gb7gk 1/1 Running 0 27h 10.107.199.77 work2
nginx-5577575ddd-pz59n 1/1 Running 0 102s 10.99.1.91 work1 平滑升级，先启动一个新版本的pod，然后流量切到新pod之后，再关闭老版本的pod
# kubectl set image deployment nginx nginx=nginx:1.18 --record
# kubectl rollout status deployment nginx
Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
deployment "nginx" successfully rolled out
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-7c456f6cf9-gb7gk 1/1 Running 0 27h 10.107.199.77 work2
nginx-648f9b4559-m2b8m 1/1 Running 0 19s 10.109.125.208 work3
# kubectl get deployment nginx -o yaml|grep '\- image:'
- image: nginx:1.18 查看历史操作deployment的命令
# kubectl rollout history deployment nginx
deployment.apps/nginx
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-dp.yaml --record=true
2 kubectl set image deployment nginx nginx=nginx:1.18 --record=true可以通过两种方式回滚：
第一种：回滚最近一次操作
# kubectl rollout undo deployment nginx
# kubectl get deployment nginx -o yaml|grep '\- image:'
- image: nginx:1.17第二种：回滚到指定的命令版本，即执行对应的命令
# kubectl rollout undo deployment nginx --to-revision=1参考文档
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps
https://kubernetes.github.io/ingress-nginx/user-guide/tls/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/
https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/