kubernetes（六）--Ingress高可用
一、Ingress高可用
上篇文章我们只是解决了集群对外提供服务的功能，并没有对ingress进行高可用的部署，接下来将实现Ingress的高可用，我们可以通过修改deployment的副本数来实现高可用，但是由于ingress承载着整个集群流量的接入，所以生产环境中，建议把ingress通过DaemonSet的方式部署集群中，而且该节点打上污点不允许业务pod进行调度，以避免业务应用与Ingress服务发生资源争抢。然后通过SLB把ingress节点主机添为后端服务器，进行流量转发。
# 修改mandatory.yaml
# 主要修改pod相关
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
#replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
serviceAccountName: nginx-ingress-serviceaccount
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
---
修改参数如下：
kind: Deployment #修改为DaemonSet
replicas: 1 #注销此行，DaemonSet不需要此参数
hostNetwork: true #添加该字段让pod使用物理机网络，在物理机暴露服务端口80，注意：物理机80端口不能被占用
dnsPolicy: ClusterFirstWithHostNet #使用hostNetwork后容器会使用物理机网络包括DNS，会无法解析内部service，使用此参数让容器使用K8S集群内部的DNS
创建ingress资源
[root@master ingress]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx unchanged
configmap/nginx-configuration unchanged
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/nginx-ingress-serviceaccount unchanged
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole unchanged
role.rbac.authorization.k8s.io/nginx-ingress-role unchanged
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding unchanged
daemonset.apps/nginx-ingress-controller created
# 查看资源分布情况
# 可以看到两个ingress-controller已经根据我们选择，部署在2个node节点上，使用宿主机的网络
kubectl get pod -n ingress-nginx -o wide
[root@master ingress]# kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-7mkfq 1/1 Running 0 13m 192.168.147.133 node1 <none> <none>
nginx-ingress-controller-lkfkl 1/1 Running 0 13m 192.168.147.134 node2 <none> <none>
测试
这里直接使用上篇文章中创建的pod资源及对应service资源，同时ingress规则也直接使用上篇文章的。
另外注意一点，因为我们创建的ingress-controller采用的时hostnetwork模式，所以无需再为ingress-controller创建service服务来把端口映射到节点主机上。
[root@master ingress]# kubectl get pod
NAME READY STATUS RESTARTS AGE
myapp-test-ingress-554b88bc89-9ng78 1/1 Running 0 21h
myapp-test-ingress-554b88bc89-n296x 1/1 Running 0 21h
myapp-test-ingress-554b88bc89-ndt9z 1/1 Running 0 21h
[root@master ingress]# kubectl get svc | grep ingress
myapp-ingress-svc ClusterIP 10.99.110.120 <none> 80/TCP 21h
[root@master ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> www.cwj.com 80 20h
二、扩展
上篇文章说道后端的deploy资源的svc服务，只用于收集pod资源并进行分类。因此我们可以创建多个后端不同的pod资源并创建不同的service资源，然后前面通过不同的虚拟主机进行访问。
创建新的deploy资源及其相关的svc资源
[root@master kube_manifest]# vim web-ingress-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: web-ingress-svc
namespace: default
spec:
selector:
app: web
env: test
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-test-ingress
spec:
replicas: 2
selector:
matchLabels:
app: web
env: test
template:
metadata:
labels:
app: web
env: test
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
ports:
- name: http
containerPort: 80
[root@master kube_manifest]# kubectl apply -f web-ingress-deploy.yaml
service/web-ingress-svc unchanged
deployment.apps/web-test-ingress created
查看pod运行状况。
[root@master kube_manifest]# kubectl get pod
NAME READY STATUS RESTARTS AGE
myapp-test-ingress-554b88bc89-9ng78 1/1 Running 0 24h
myapp-test-ingress-554b88bc89-n296x 1/1 Running 0 24h
myapp-test-ingress-554b88bc89-ndt9z 1/1 Running 0 24h
web-test-ingress-6cf54d8c9b-4x95t 1/1 Running 0 65s
web-test-ingress-6cf54d8c9b-w75bm 1/1 Running 0 65s
查看service的运行状况。
[root@master kube_manifest]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18d
myapp-ingress-svc ClusterIP 10.99.110.120 <none> 80/TCP 24h
web-ingress-svc ClusterIP 10.111.51.79 <none> 80/TCP 129m
修改ingress规则，使其能关联到后端两个不同的pod资源。
[root@master ingress]# vim ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: www.cwj.com
http:
paths:
- path:
backend:
serviceName: myapp-ingress-svc
servicePort: 80
- host: www.chenwj.com
http:
paths:
- path:
backend:
serviceName: web-ingress-svc
servicePort: 80
[root@master ingress]# kubectl apply -f ingress-myapp.yaml
ingress.extensions/ingress-myapp configured
[root@master ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> www.cwj.com,www.chenwj.com 80 24h
[root@master kube_manifest]# kubectl describe ingress ingress-myapp
Name: ingress-myapp
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www.cwj.com
myapp-ingress-svc:80 (10.244.1.34:80,10.244.2.26:80,10.244.2.27:80)
www.chenwj.com
web-ingress-svc:80 (10.244.1.35:80,10.244.2.28:80)
Annotations: kubernetes.io/ingress.class: nginx
Events: <none>
交互式进入到nginx-ingress-controller中，查看nginx的配置信息。ingress规则会注入到nginx中
[root@master ~]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-lkfkl -- /bin/sh
$ ls
fastcgi.conf geoip mime.types nginx.conf scgi_params uwsgi_params.default
fastcgi.conf.default koi-utf mime.types.default nginx.conf.default scgi_params.default win-utf
fastcgi_params koi-win modsecurity opentracing.json template
fastcgi_params.default lua modules owasp-modsecurity-crs uwsgi_params
server {
server_name www.chenwj.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
location / {
set $namespace "default";
set $ingress_name "ingress-myapp";
set $service_name "web-ingress-svc";
set $service_port "80";
set $location_path "/";
...
server {
server_name www.cwj.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
location / {
set $namespace "default";
set $ingress_name "ingress-myapp";
set $service_name "myapp-ingress-svc";
set $service_port "80";
set $location_path "/";
...
在本地主机配置hosts域名解析，这里使用的是windows主机。
编辑hosts文件，windows\system32\drivers\etc\hosts。
访问测试
