官网参考:https://kubernetes.github.io/ingress-nginx/
Ingress是一种全局的、为了代理不同后端 Service 而设置的负载均衡服务,就是 Kubernetes 里的 Ingress 服务。
Ingress由两部分组成:Ingress controller和Ingress服务。
Ingress Controller 会根据你定义的 Ingress 对象,提供对应的代理能力。业界常用的各 种反向代理项目,比如 Nginx、HAProxy、Envoy、Traefik 等,都已经为Kubernetes 专门维护了对应的 Ingress Controller。
参考官网:https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml
下载所需的镜像到镜像仓库
[root@harbor ~]# docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0
[root@harbor ~]# docker pull docker.io/jettech/kube-webhook-certgen:v1.2.0
[root@harbor ~]# docker tag quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0 reg.red.org/library/nginx-ingress-controller:0.33.0
[root@harbor ~]# docker tag docker.io/jettech/kube-webhook-certgen:v1.2.0 reg.red.org/library/kube-webhook-certgen:v1.2.0
[root@harbor ~]# docker push reg.red.org/library/nginx-ingress-controller:0.33.0
[root@harbor ~]# docker push reg.red.org/library/kube-webhook-certgen:v1.2.0
kubectl get all -n ingress-nginx
查看ingress-nginx下的所有信息资源
kubectl get pod -n ingress-nginx -o wide
查看pod运行在那些节点
kubectl get svc -n ingress-nginx
获取到端口 31181
kubectl describe svc -n ingress-nginx ingress-nginx-controller
查看服务详细信息
[kubeadm@server1 mainfest]$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
[kubeadm@server1 mainfest]$ kubectl get namespaces
NAME STATUS AGE
default Active 7d18h
ingress-nginx Active 35s
kube-node-lease Active 7d18h
kube-public Active 7d18h
kube-system Active 7d18h
[kubeadm@server1 mainfest]$ kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-xfcsd 0/1 Completed 0 57s
ingress-nginx-admission-patch-k9dcm 0/1 Completed 0 57s
ingress-nginx-controller-77b5fc5746-dq7hn 1/1 Running 0 67s
[kubeadm@server1 mainfest]$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.100.163.143 <none> 80:31904/TCP,443:32203/TCP 2m21s
ingress-nginx-controller-admission ClusterIP 10.101.228.137 <none> 443/TCP 2m21s
[kubeadm@server1 mainfest]$ kubectl describe svc -n ingress-nginx ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.33.0
helm.sh/chart=ingress-nginx-2.9.0
Annotations: Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: NodePort
IP: 10.100.163.143
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31904/TCP
Endpoints: 10.244.2.82:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 32203/TCP
Endpoints: 10.244.2.82:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
参考官网:https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/
示例一
[kubeadm@server1 mainfest]$ vim ingress.yml
[kubeadm@server1 mainfest]$ cat ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: www1.red.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
[kubeadm@server1 mainfest]$ kubectl apply -f ingress.yml
ingress.extensions/ingress1 created
[kubeadm@server1 mainfest]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d22h
myservice NodePort 10.102.1.239 <none> 80:31334/TCP 6h47m
[kubeadm@server1 mainfest]$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress1 <none> www1.westos.org 172.25.1.3 80 78s
[kubeadm@server1 mainfest]$ kubectl -n ingress-nginx get pod
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-xfcsd 0/1 Completed 0 3h20m
ingress-nginx-admission-patch-k9dcm 0/1 Completed 0 3h20m
ingress-nginx-controller-77b5fc5746-dq7hn 1/1 Running 0 3h20m
[kubeadm@server1 mainfest]$ kubectl -n ingress-nginx exec -it ingress-nginx-controller-77b5fc5746-dq7hn -- sh
/etc/nginx $ ls
fastcgi.conf koi-utf modsecurity owasp-modsecurity-crs uwsgi_params.default
fastcgi.conf.default koi-win modules scgi_params win-utf
fastcgi_params lua nginx.conf scgi_params.default
fastcgi_params.default mime.types nginx.conf.default template
geoip mime.types.default opentracing.json uwsgi_params
/etc/nginx $ vi nginx.conf
示例二
[kubeadm@server1 mainfest]$ vim service.yml
[kubeadm@server1 mainfest]$ cat service.yml
kind: Service
apiVersion: v1
metadata:
name: myservice
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: myappv1
type: ClusterIP
---
kind: Service
apiVersion: v1
metadata:
name: myservice2
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: myappv2
type: ClusterIP
[kubeadm@server1 mainfest]$ kubectl apply -f service.yml
service/myservice created
service/myservice2 created
[kubeadm@server1 mainfest]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d22h
myservice ClusterIP 10.110.45.54 <none> 80/TCP 6s
myservice2 ClusterIP 10.103.62.115 <none> 80/TCP 6s
[kubeadm@server1 mainfest]$ vim pod2.yml
[kubeadm@server1 mainfest]$ cat pod2.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-example
spec:
replicas: 2
selector:
matchLabels:
app: myappv1
template:
metadata:
labels:
app: myappv1
spec:
containers:
- name: myappv1
image: myapp:v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-example2
spec:
replicas: 2
selector:
matchLabels:
app: myappv2
template:
metadata:
labels:
app: myappv2
spec:
containers:
- name: myappv2
image: myapp:v2
[kubeadm@server1 mainfest]$ kubectl apply -f pod2.yml
deployment.apps/deployment-example created
deployment.apps/deployment-example2 created
[kubeadm@server1 mainfest]$ kubectl describe svc myservice
Name: myservice
Namespace: default
Labels: <none>
Annotations: Selector: app=myappv1
Type: ClusterIP
IP: 10.110.45.54
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.56:80,10.244.2.84:80
Session Affinity: None
Events: <none>
[kubeadm@server1 mainfest]$ kubectl describe svc myservice2
Name: myservice2
Namespace: default
Labels: <none>
Annotations: Selector: app=myappv2
Type: ClusterIP
IP: 10.103.62.115
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.57:80,10.244.2.83:80
Session Affinity: None
Events: <none>
[kubeadm@server1 mainfest]$ vim ingress.yml
[kubeadm@server1 mainfest]$ cat ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: www1.red.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
- host: www2.red.org
http:
paths:
- path: /
backend:
serviceName: myservice2
servicePort: 80
[kubeadm@server1 mainfest]$ kubectl apply -f ingress.yml
ingress.extensions/ingress1 created
[kubeadm@server1 mainfest]$ kubectl describe ingress ingress1
Name: ingress1
Namespace: default
Address: 172.25.1.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www1.red.org
/ myservice:80 (10.244.1.56:80,10.244.2.84:80)
www2.red.org
/ myservice2:80 (10.244.1.57:80,10.244.2.83:80)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 2m17s nginx-ingress-controller Ingress default/ingress1
Normal UPDATE 83s nginx-ingress-controller Ingress default/ingress1
用DaemonSet结合nodeselector来部署ingress-controller到特定的node上,然后使用 HostNetwork直接把该pod与宿主机node的网络打通,直接使用宿主机的80/433端口就 能访问服务。
优点:是整个请求链路最简单,性能相对NodePort模式更好。
缺点:是由于直接利用宿主机节点的网络和端口,一个node只能部署一个ingresscontroller pod。• 比较适合大并发的生产环境使用。
[kubeadm@server1 mainfest]$ vim deploy.yaml
kind: DaemonSet
spec:
hostNetwork: true
nodeSelector:
kubernetes.io/hostname: server3
[kubeadm@server1 mainfest]$ kubectl -n ingress-nginx delete deployments.apps ingress-nginx-controller
Error from server (NotFound): deployments.apps "ingress-nginx-controller" not found
[kubeadm@server1 mainfest]$ kubectl apply -f deploy.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
configmap/ingress-nginx-controller configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
service/ingress-nginx-controller-admission unchanged
service/ingress-nginx-controller unchanged
daemonset.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
serviceaccount/ingress-nginx-admission unchanged
[kubeadm@server1 mainfest]$ kubectl -n ingress-nginx get all
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-xfcsd 0/1 Completed 0 4h36m
pod/ingress-nginx-admission-patch-k9dcm 0/1 Completed 0 4h36m
pod/ingress-nginx-controller-pvr28 1/1 Running 0 29s
kubectl get all -n ingress-nginx
查看ingress-nginx下所有信息,可以看到控制器类型已经改为DaemonSet
官网参考:https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/
[kubeadm@server1 mainfest]$ vim ingress.yml
[kubeadm@server1 mainfest]$ cat ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: www1.red.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress2
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/affinity: cookie
spec:
rules:
- host: www2.red.org
http:
paths:
- path: /
backend:
serviceName: myservice2
servicePort: 80
[kubeadm@server1 mainfest]$ kubectl apply -f ingress.yml
ingress.extensions/ingress1 configured
ingress.extensions/ingress2 created
[kubeadm@server1 mainfest]$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress1 <none> www1.red.org 172.25.1.3 80 72m
ingress2 <none> www2.red.org 172.25.1.3 80 2m2s
[kubeadm@server1 mainfest]$ kubectl describe ingress ingress1
Name: ingress1
Namespace: default
Address: 172.25.1.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www1.red.org
/ myservice:80 (10.244.1.56:80,10.244.2.84:80)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 13m nginx-ingress-controller Ingress default/ingress1
Normal UPDATE 2m12s (x2 over 13m) nginx-ingress-controller Ingress default/ingress1
[kubeadm@server1 mainfest]$ kubectl describe ingress ingress2
Name: ingress2
Namespace: default
Address: 172.25.1.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www2.red.org
/ myservice2:80 (10.244.1.57:80,10.244.2.83:80)
Annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 2m14s nginx-ingress-controller Ingress default/ingress2
Normal UPDATE 82s nginx-ingress-controller Ingress default/ingress2
官网参考:
https://kubernetes.github.io/ingress-nginx/examples/tls-termination/
https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/#tls-certificates
[kubeadm@server1 mainfest]$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" ##生成证书和key
[kubeadm@server1 mainfest]$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt ##对证书和key加密
[kubeadm@server1 mainfest]$ kubectl get secrets
NAME TYPE DATA AGE
default-token-5qqxc kubernetes.io/service-account-token 3 7d23h
tls-secret kubernetes.io/tls 2 66s
[kubeadm@server1 mainfest]$ vim tls.yml
[kubeadm@server1 mainfest]$ cat tls.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx-tls
spec:
tls:
- hosts:
- www1.red.org
# This assumes tls-secret exists and the SSL
# certificate contains a CN for foo.bar.com
secretName: tls-secret
rules:
- host: www1.red.org
http:
paths:
- path: /
backend:
# This assumes http-svc exists and routes to healthy endpoints
serviceName: myservice
servicePort: 80
[kubeadm@server1 mainfest]$ kubectl apply -f tls.yml
ingress.networking.k8s.io/nginx-tls created
[kubeadm@server1 mainfest]$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress1 <none> www1.red.org 172.25.1.3 80 86m
ingress2 <none> www2.red.org 172.25.1.3 80 16m
nginx-tls <none> www1.red.org 80, 443 34s
[kubeadm@server1 mainfest]$ kubectl describe ingress nginx-tls
Name: nginx-tls
Namespace: default
Address: 172.25.1.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
tls-secret terminates www1.red.org
Rules:
Host Path Backends
---- ---- --------
www1.red.org
/ myservice:80 (10.244.1.56:80,10.244.2.84:80)
Annotations: Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 68s nginx-ingress-controller Ingress default/nginx-tls
Normal UPDATE 8s nginx-ingress-controller Ingress default/nginx-tls