部署kubernetes的ingress-nginx使用(含实际操作以及所有yaml文件)

一、Ingress简介

Ingress是对外服务到集群内的Service之间规则的集合：允许进入集群的请求被转发至集群内的Service。

Ingress能把Service配置成外网能够访问的url，流量负载均衡，终止ssl，提供基于域名访问的虚拟主机等，用户通过访问url访问Service。

Ingress-controller负责处理所有Ingress的请求流量，它通常是一个负载均衡器。

 

在Kubernetes中，服务和Pod的IP地址仅可以在集群网络内部使用，对于集群外的应用是不可见的。为了使外部的应用能够访问集群内的服务，在Kubernetes 目前 提供了以下几种方案：

NodePort

LoadBalancer

Ingress

Ingress组成

ingress controller

　　将新加入的Ingress转化成Nginx的配置文件并使之生效

ingress服务

　　将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可

Ingress工作原理

1.ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化，

2.然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段nginx配置，

3.再写到nginx-ingress-control的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中，

4.然后reload一下使配置生效。以此达到域名分配置和动态更新的问题。

 

Ⅰ、ingress策略本质是转发的规则；

Ⅱ、ingress-controller基于ingress策略将客户端的请求转发到service对应的后端endpoint，即Pod上；实现了为所有后端service提供统一入口，基于不同的http url向后转发负载分发规则，并可以灵活设置7层的负载分发策略的功能；一般由nginx实现。

Ingress可以解决什么问题

1.动态配置服务

　　如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作.

2.减少不必要的端口暴露

　　配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式

 

二、部署配置Ingress controller

1、准备docker images

由于下载的yaml文件中需要用到的两个镜像 ingress-nginx和defaultbackend，但是国内是无法直接下载，因此需要提前准备好，方式和部署kubernetes集群的docker镜像是一样的，以下是操作方式以及实际操作流程

# 【如果镜像pull之后，不想改tag，则在对应的yaml文件中进行调整】

# 由于下载的yaml部署文件中，使用的镜像版本是0.30.0，因此找这个版本的

docker pull suisrc/ingress-nginx:0.30.0

docker tag suisrc/ingress-nginx:0.30.0 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

docker rmi suisrc/ingress-nginx:0.30.0

 

# 由于在部署的时候，如果删除了阿里云的那个镜像，部署会出现无法拉取镜像出现，因此这里就不删除

docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

docker tag registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend:1.5

# 实际操作如下

root@k8s-master:~/k8s/ingress# docker pull suisrc/ingress-nginx:0.30.0

0.30.0: Pulling from suisrc/ingress-nginx

c9b1b535fdd9: Pull complete

45ba4c948320: Pull complete

70c24c20a569: Pull complete

58acda238271: Pull complete

7873cb07ba91: Pull complete

3572b831a7ad: Pull complete

2e4b94d88c7a: Pull complete

73d054fe6162: Pull complete

72107c0475b3: Pull complete

0920fa00bdaf: Pull complete

bbc4231b0eed: Pull complete

1a0d8e7b84e8: Pull complete

b2947a77ebb4: Pull complete

Digest: sha256:d22b600d98649dd06a128fbf0c8bf8d1aaf25512f0d57770079b7211e1d4e64a

Status: Downloaded newer image for suisrc/ingress-nginx:0.30.0

docker.io/suisrc/ingress-nginx:0.30.0

root@k8s-master:~/k8s/ingress# docker images

REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE

nginx                                latest              ed21b7a8aee9        9 days ago          127MB

k8s.gcr.io/kube-proxy                v1.18.0             43940c34f24f        2 weeks ago         117MB

k8s.gcr.io/kube-controller-manager   v1.18.0             d3e55153f52f        2 weeks ago         162MB

k8s.gcr.io/kube-scheduler            v1.18.0             a31f78c7c8ce        2 weeks ago         95.3MB

k8s.gcr.io/kube-apiserver            v1.18.0             74060cea7f70        2 weeks ago         173MB

suisrc/ingress-nginx                 0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/coreos/flannel               v0.12.0-amd64       4e9f801d2217        3 weeks ago         52.8MB

quay.io/coreos/flannel               v0.12.0-arm64       4e9f801d2217        3 weeks ago         52.8MB

k8s.gcr.io/pause                     3.2                 80d28bedfe5d        7 weeks ago         683kB

k8s.gcr.io/coredns                   1.6.7               67da37a9a360        2 months ago        43.8MB

k8s.gcr.io/etcd                      3.4.3-0             303ce5db0e90        5 months ago        288MB

root@k8s-master:~/k8s/ingress# docker tag suisrc/ingress-nginx:0.30.0 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

root@k8s-master:~/k8s/ingress# docker images

REPOSITORY                                                                TAG                 IMAGE ID            CREATED             SIZE

nginx                                                                     latest              ed21b7a8aee9        9 days ago          127MB

k8s.gcr.io/kube-proxy                                                     v1.18.0             43940c34f24f        2 weeks ago         117MB

k8s.gcr.io/kube-scheduler                                                 v1.18.0             a31f78c7c8ce        2 weeks ago         95.3MB

k8s.gcr.io/kube-controller-manager                                        v1.18.0             d3e55153f52f        2 weeks ago         162MB

k8s.gcr.io/kube-apiserver                                                 v1.18.0             74060cea7f70        2 weeks ago         173MB

suisrc/ingress-nginx                                                      0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller            0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/coreos/flannel                                                    v0.12.0-amd64       4e9f801d2217        4 weeks ago         52.8MB

quay.io/coreos/flannel                                                    v0.12.0-arm64       4e9f801d2217        4 weeks ago         52.8MB

k8s.gcr.io/pause                                                          3.2                 80d28bedfe5d        7 weeks ago         683kB

k8s.gcr.io/coredns                                                        1.6.7               67da37a9a360        2 months ago        43.8MB

k8s.gcr.io/etcd                                                           3.4.3-0             303ce5db0e90        5 months ago        288MB

root@k8s-master:~/k8s/ingress# docker rmi suisrc/ingress-nginx:0.30.0

root@k8s-master:~/k8s/ingress# docker images

REPOSITORY                                                                TAG                 IMAGE ID            CREATED             SIZE

nginx                                                                     latest              ed21b7a8aee9        9 days ago          127MB

k8s.gcr.io/kube-proxy                                                     v1.18.0             43940c34f24f        2 weeks ago         117MB

k8s.gcr.io/kube-scheduler                                                 v1.18.0             a31f78c7c8ce        2 weeks ago         95.3MB

k8s.gcr.io/kube-controller-manager                                        v1.18.0             d3e55153f52f        2 weeks ago         162MB

k8s.gcr.io/kube-apiserver                                                 v1.18.0             74060cea7f70        2 weeks ago         173MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller            0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/coreos/flannel                                                    v0.12.0-amd64       4e9f801d2217        4 weeks ago         52.8MB

quay.io/coreos/flannel                                                    v0.12.0-arm64       4e9f801d2217        4 weeks ago         52.8MB

k8s.gcr.io/pause                                                          3.2                 80d28bedfe5d        7 weeks ago         683kB

k8s.gcr.io/coredns                                                        1.6.7               67da37a9a360        2 months ago        43.8MB

k8s.gcr.io/etcd                                                           3.4.3-0             303ce5db0e90        5 months ago        288MB

 

 

root@k8s-master:~/k8s/ingress# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

1.5: Pulling from kubernetes_xingej/defaultbackend-amd64

Digest: sha256:2cdff48ab9b20ca5f9b0ee48bf3c139c51d6fb1a15245966583bc371c121c238

Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

root@k8s-master:~/k8s/ingress# docker images

REPOSITORY                                                                TAG                 IMAGE ID            CREATED             SIZE

nginx                                                                     latest              ed21b7a8aee9        9 days ago          127MB

k8s.gcr.io/kube-proxy                                                     v1.18.0             43940c34f24f        2 weeks ago         117MB

k8s.gcr.io/kube-controller-manager                                        v1.18.0             d3e55153f52f        2 weeks ago         162MB

k8s.gcr.io/kube-scheduler                                                 v1.18.0             a31f78c7c8ce        2 weeks ago         95.3MB

k8s.gcr.io/kube-apiserver                                                 v1.18.0             74060cea7f70        2 weeks ago         173MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller            0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/coreos/flannel                                                    v0.12.0-amd64       4e9f801d2217        4 weeks ago         52.8MB

quay.io/coreos/flannel                                                    v0.12.0-arm64       4e9f801d2217        4 weeks ago         52.8MB

k8s.gcr.io/pause                                                          3.2                 80d28bedfe5d        7 weeks ago         683kB

k8s.gcr.io/coredns                                                        1.6.7               67da37a9a360        2 months ago        43.8MB

k8s.gcr.io/etcd                                                           3.4.3-0             303ce5db0e90        5 months ago        288MB

registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64   1.5                 d8f37b8cdaf4        18 months ago       5.13MB

root@k8s-master:~/k8s/ingress# docker tag registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend:1.5

root@k8s-master:~/k8s/ingress# docker images

REPOSITORY                                                                TAG                 IMAGE ID            CREATED             SIZE

nginx                                                                     latest              ed21b7a8aee9        9 days ago          127MB

k8s.gcr.io/kube-proxy                                                     v1.18.0             43940c34f24f        2 weeks ago         117MB

k8s.gcr.io/kube-controller-manager                                        v1.18.0             d3e55153f52f        2 weeks ago         162MB

k8s.gcr.io/kube-scheduler                                                 v1.18.0             a31f78c7c8ce        2 weeks ago         95.3MB

k8s.gcr.io/kube-apiserver                                                 v1.18.0             74060cea7f70        2 weeks ago         173MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller            0.30.0              0118b9637e9b        2 weeks ago         393MB

quay.io/coreos/flannel                                                    v0.12.0-amd64       4e9f801d2217        4 weeks ago         52.8MB

quay.io/coreos/flannel                                                    v0.12.0-arm64       4e9f801d2217        4 weeks ago         52.8MB

k8s.gcr.io/pause                                                          3.2                 80d28bedfe5d        7 weeks ago         683kB

k8s.gcr.io/coredns                                                        1.6.7               67da37a9a360        2 months ago        43.8MB

k8s.gcr.io/etcd                                                           3.4.3-0             303ce5db0e90        5 months ago        288MB

k8s.gcr.io/defaultbackend                                                 1.5                 d8f37b8cdaf4        18 months ago       5.13MB

registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64   1.5                 d8f37b8cdaf4        18 months ago       5.13MB

2、下载和设置相关的yaml文件

下载地址： https://github.com/kubernetes/ingress-nginx/blob/master/deploy/static/mandatory.yaml

这个文件包含了需要部署的全部信息，但是没有 defaultbackend这个的yaml文件内容，我按照顺序逐个进行拆分，拆分的文件有：

01-namespace.yaml、02-configmap.yaml、04-rbac.yaml、05-with-rbac.yaml、06-limit.yaml

03-http-backend.yaml 就是部署defaultbackend服务

07-node-port.yaml 就是部署service，暴露出http和https的端口给外部使用【在没有loadbalance的情况下暂时这样实现】

 

这里只列举调整的地方， 至于具体详细的说明见附录代码说明 【红色部分是我调整的内容】

Ⅰ、主要是把namespace由ingress-nginx改成了kube-ingress

Ⅱ、由于最新的mandatory.yaml并没有和defaultbackend结合起来，因此在部署nginx-ingress-controller对应deployment的时候添加了结合

Ⅲ、defaultbackend的yaml文件部署，我放在了configmap之后

 

3、按照顺序进行部署yaml文件

kubectl apply -f 01-namespace.yaml

kubectl apply -f 02-configmap.yaml

kubectl apply -f 03-http-backend.yaml

kubectl apply -f 04-rbac.yaml

kubectl apply -f 05-with-rbac.yaml

kubectl apply -f 06-limit.yaml

kubectl apply -f 07-node-port.yaml

root@k8s-master:~/k8s/ingress# ll

total 36

drwxr-xr-x 2 root   root   4096 Apr 10 03:26 ./

drwxrwxr-x 3 master master 4096 Apr  9 10:37 ../

-rw-r--r-- 1 root   root    236 Apr  9 10:36 01-namespace.yaml

-rw-r--r-- 1 root   root   1602 Apr  9 10:36 02-configmap.yaml

-rw-r--r-- 1 root   root   1510 Apr 10 03:23 03-http-backend.yaml

-rw-r--r-- 1 root   root   3289 Apr 10 03:25 04-rbac.yaml

-rw-r--r-- 1 root   root   3458 Apr 10 02:12 05-with-rbac.yaml

-rw-r--r-- 1 root   root    300 Apr  9 10:32 06-limit.yaml

-rw-r--r-- 1 root   root    591 Apr  9 10:55 07-node-port.yaml

root@k8s-master:~/k8s/ingress# kubectl apply -f 01-namespace.yaml

namespace/kube-ingress created

root@k8s-master:~/k8s/ingress# kubectl apply -f 02-configmap.yaml

configmap/nginx-configuration created

configmap/tcp-services created

configmap/udp-services created

root@k8s-master:~/k8s/ingress# kubectl apply -f 03-http-backend.yaml

deployment.apps/default-http-backend created

service/default-http-backend created

root@k8s-master:~/k8s/ingress# kubectl apply -f 04-rbac.yaml

serviceaccount/nginx-ingress-serviceaccount created

clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created

role.rbac.authorization.k8s.io/nginx-ingress-role created

rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created

clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created

root@k8s-master:~/k8s/ingress# kubectl apply -f 05-with-rbac.yaml

deployment.apps/nginx-ingress-controller created

root@k8s-master:~/k8s/ingress# kubectl apply -f 06-limit.yaml

limitrange/ingress-nginx created

root@k8s-master:~/k8s/ingress# kubectl apply -f 07-node-port.yaml

service/ingress-nginx created

root@k8s-master:~/k8s/ingress# kubectl get all -n kube-ingress

NAME                                            READY   STATUS    RESTARTS   AGE

pod/default-http-backend-59c5fc7f59-5t9gp       1/1     Running   0          68s

pod/nginx-ingress-controller-845cdf96c7-2xmhp   1/1     Running   0          60s

pod/nginx-ingress-controller-845cdf96c7-xgwgj   1/1     Running   0          60s

 

 

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE

service/default-http-backend   ClusterIP   10.104.118.160           80/TCP                       68s

service/ingress-nginx          NodePort    10.105.13.73             80:32080/TCP,443:32443/TCP   53s

 

 

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/default-http-backend       1/1     1            1           68s

deployment.apps/nginx-ingress-controller   2/2     2            2           60s

 

 

NAME                                                  DESIRED   CURRENT   READY   AGE

replicaset.apps/default-http-backend-59c5fc7f59       1         1         1       68s

replicaset.apps/nginx-ingress-controller-845cdf96c7   2         2         2       60s

root@k8s-master:~/k8s/ingress# kubectl get secret -n kube-ingress

NAME                                       TYPE                                  DATA   AGE

default-token-bj7hg                        kubernetes.io/service-account-token   3      119s

nginx-ingress-serviceaccount-token-t49gf   kubernetes.io/service-account-token   3      106s

root@k8s-master:~/k8s/ingress# kubectl get configmap -n kube-ingress

NAME                              DATA   AGE

ingress-controller-leader-nginx   0      105s

nginx-configuration               0      2m

tcp-services                      0      2m

udp-services                      0      2m

 

4、 访问ingress-nginx服务，查看是否配置成功

         

        可以看到，提示404，这个因为当前ingress-nginx服务现在还没有后端服务，这是正常的

 

 

 

三、部署配置Ingress

1、部署一个后端服务(首先创建Deployment，然后创建对应Service，最后设置对应的Ingress)

这里以一个简单的nginx作为后端测试服务，部署的namespace=default，首先创建一个deployment和service，yaml文件具体内容如下

# 文件名：08-nginx.yaml

# 创建后端服务的nginx的服务

# Deployment的name=nginx，labels=web，使用nginx最新的镜像，部署两个副本

---

apiVersion: apps/v1

kind: Deployment

metadata:

  # Deployment的全局唯一名称

  name: nginx

spec:

  replicas: 2

  selector:

    matchLabels:

      app: web

  template:

    metadata:

      labels:

        app: web

    spec:

      containers:

      - name: nginx

        image: nginx:latest

        # 这个nginx默认就80端口，可以不指定这个ports

        ports:

        - containerPort: 80

 

 

---

apiVersion: v1

kind: Service

metadata:

  # Service的全局唯一名称

  name: nginx

spec:

  # Service关联定义了相应标签的Pod

  # 由于上面定义的deployment的pod的app的标签是labels app=web

  selector:

    app: web

  ports:

    # 端口名称(非必须)

    - name: nginx

      # Service服务的协议，默认是TCP

      protocol: TCP

      # Service服务监听的端口号【即：对其他namespace的访问端口号】

      port: 80

      # 后端服务Pod提供的端口号【即：上面Deployment的containers中设置的containerPort】

      targetPort: 80

 

接着是对应的ingress的yaml文件设置，需要和上面的service对应，部署的namespace与上面一致，都在default namespace下

# 文件名：09-nginx-svc-ingress.yaml

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  # Ingress的全局唯一名称

  name: nginx

  # 部署的namepsace

  namespace: default

  annotations:

    kubernetes.io/ingress.class: "nginx"

spec:

  rules:

  # 主机域名，需要在本地绑定节点IP

  - host: domain.com

    http:

      # 如果paths下有具体的路径，如/demo，需要与后端提供真实服务的path一致，这里即nginx下需要有/demo路径

      paths:

      - path:

        backend:

          # 后端服务名，就是kind=Service中的metadata中的name

          serviceName: nginx

          # 后端服务监听端口，区别于提供真实服务的容器监听端口

          servicePort: 80

 

最后配置域名解析，当前测试环境我们使用hosts文件进行解析，将下面信息写入hosts里面 【其中domain.com就是上面ingress中的host】

192.168.89.133 domain.com

root@k8s-master:~/k8s/ingress# kubectl apply -f 08-nginx.yaml

deployment.apps/nginx created

service/nginx created

root@k8s-master:~/k8s/ingress# kubectl apply -f 09-nginx-svc-ingress.yaml

ingress.extensions/nginx created

root@k8s-master:~/k8s/ingress# kubectl get all

NAME                         READY   STATUS              RESTARTS   AGE

pod/nginx-7dc7b7dbf5-kfnhb   0/1     ContainerCreating   0          12s

pod/nginx-7dc7b7dbf5-sms78   0/1     ContainerCreating   0          12s

 

 

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE

service/kubernetes   ClusterIP   10.96.0.1               443/TCP   2d1h

service/nginx        ClusterIP   10.107.193.92           80/TCP    12s

 

 

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/nginx   0/2     2            0           12s

 

 

NAME                               DESIRED   CURRENT   READY   AGE

replicaset.apps/nginx-7dc7b7dbf5   2         2         0       12s

root@k8s-master:~/k8s/ingress# kubectl get all

NAME                         READY   STATUS    RESTARTS   AGE

pod/nginx-7dc7b7dbf5-kfnhb   1/1     Running   0          5m53s

pod/nginx-7dc7b7dbf5-sms78   1/1     Running   0          5m53s

 

 

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE

service/kubernetes   ClusterIP   10.96.0.1               443/TCP   2d1h

service/nginx        ClusterIP   10.107.193.92           80/TCP    5m53s

 

 

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/nginx   2/2     2            2           5m53s

 

 

NAME                               DESIRED   CURRENT   READY   AGE

replicaset.apps/nginx-7dc7b7dbf5   2         2         2       5m53s

root@k8s-master:~/k8s/ingress# kubectl get ingress

NAME    CLASS    HOSTS        ADDRESS        PORTS   AGE

nginx      domain.com   10.105.13.73   80      5m54s

 

2、对部署的后端服务添加https服务

首先创建私有证书以及secret，然后将私有证书应用于nginx服务中，具体操作如下

# 创建私有证书key

root@k8s-master:~/k8s/ingress# openssl genrsa -out tls.key 2048

Generating RSA private key, 2048 bit long modulus (2 primes)

.......................................+++++

.+++++

e is 65537 (0x010001)

 

# 用私有证书key生成crt证书，注意域名要和服务的域名一致

root@k8s-master:~/k8s/ingress# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Guangdong/L=Shenzhen/O=DevOps/CN=domain.com

Can't load /home/master/.rnd into RNG

140440815153600:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/master/.rnd

root@k8s-master:~/k8s/ingress# ll

total 52

drwxr-xr-x 2 root   root   4096 Apr 10 10:59 ./

drwxrwxr-x 3 master master 4096 Apr  9 10:37 ../

-rw-r--r-- 1 root   root    236 Apr  9 10:36 01-namespace.yaml

-rw-r--r-- 1 root   root   1602 Apr  9 10:36 02-configmap.yaml

-rw-r--r-- 1 root   root   1510 Apr 10 03:23 03-http-backend.yaml

-rw-r--r-- 1 root   root   3289 Apr 10 03:25 04-rbac.yaml

-rw-r--r-- 1 root   root   3458 Apr 10 02:12 05-with-rbac.yaml

-rw-r--r-- 1 root   root    300 Apr  9 10:32 06-limit.yaml

-rw-r--r-- 1 root   root    591 Apr  9 10:55 07-node-port.yaml

-rw-r--r-- 1 root   root   1248 Apr 10 06:28 08-nginx.yaml

-rw-r--r-- 1 root   root    743 Apr 10 06:29 09-nginx-svc-ingress.yaml

-rw-r--r-- 1 root   root   1302 Apr 10 10:59 tls.crt

-rw------- 1 root   root   1671 Apr 10 10:58 tls.key

 

# 在namespace=defalut里面创建一个tls的secret

root@k8s-master:~/k8s/ingress# kubectl create secret tls nginx-svc-ingress-secret --cert=tls.crt --key=tls.key

secret/nginx-svc-ingress-secret created

root@k8s-master:~/k8s/ingress# kubectl get secret

NAME                       TYPE                                  DATA   AGE

default-token-xtzh2        kubernetes.io/service-account-token   3      2d6h

nginx-svc-ingress-secret   kubernetes.io/tls                     2      12s

root@k8s-master:~/k8s/ingress# kubectl describe secret nginx-svc-ingress-secret

Name:         nginx-svc-ingress-secret

Namespace:    default

Labels:       

Annotations:  

 

 

Type:  kubernetes.io/tls

 

 

Data

====

tls.crt:  1302 bytes

tls.key:  1671 bytes

 

#################################################

# 将证书添加至ingress中，红色部分

# 文件名：09-nginx-svc-ingress.yaml

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  # Ingress的全局唯一名称

  name: nginx

  # 部署的namepsace

  namespace: default

  annotations:

    kubernetes.io/ingress.class: "nginx"

spec:

  tls:

    - hosts:

        # 与secret证书的域名需要保持一致

        - domain.com

      # secret证书的名称

      secretName: nginx-svc-ingress-secret

  rules:

    # 主机域名，需要在本地绑定节点IP

    - host: domain.com

      http:

        # 如果paths下有具体的路径，如/demo，需要与后端提供真实服务的path一致，这里即nginx下需要有/demo路径

        paths:

        - path:

          backend:

            # 后端服务名，就是kind=Service中的metadata中的name

            serviceName: nginx

            # 后端服务监听端口，区别于提供真实服务的容器监听端口

            servicePort: 80

 

 

root@k8s-master:~/k8s/ingress# kubectl apply -f 09-nginx-svc-ingress.yaml

ingress.extensions/nginx configured

             

 

 

参考资料：

【kubernetes/ingress-nginx，版本：0.30.0】 https://github.com/kubernetes/ingress-nginx

 

附录：

 

# 文件名：01-namespace.yaml
# 创建一个独立的命名空间 kube-ingress
---
apiVersion: v1
kind: Namespace
metadata:
  # name: ingress-nginx【这里调整成了kube-ingress】
  name: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

 

 

 

# 文件名：02-configmap.yaml
# ConfigMap是存储通用的配置变量的，类似于配置文件，使用户可以将分布式系统中用于不同模块的环境变量统一到一个对象中管理；而它与配置文件的区别在于它是存在集群的“环境”中的，并且支持K8S集群中所有通用的操作调用方式。
# 从数据角度来看，ConfigMap的类型只是键值组，用于存储被Pod或者其他资源对象（如RC）访问的信息。这与secret的设计理念有异曲同工之妙，主要区别在于ConfigMap通常不用于存储敏感信息，而只存储简单的文本信息。
# ConfigMap可以保存环境变量的属性，也可以保存配置文件。
# 创建pod时，对configmap进行绑定，pod内的应用可以直接引用ConfigMap的配置。相当于configmap为应用/运行环境封装配置。
# pod使用ConfigMap，通常用于：设置环境变量的值、设置命令行参数、创建配置文件。
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


# 文件名：03-http-backend.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  # namespace: ingress-nginx
  namespace: kube-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        # 这里镜像路径进行调整了【之前docker pull好之后的】
        image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          # 这里调整了cpu和memory的大小，可能不同集群限制的最小值不同，看部署失败的原因就清楚
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi


---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

 

 

 

# 文件名：04-rbac.yaml
# 负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding
# ingress-controller需要监听apiserver，获取ingress定义，通过rbac授权；
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "-"
      # Here: "-"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    # namespace: ingress-nginx
    namespace: kube-ingress


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    # namespace: ingress-nginx
    namespace: kube-ingress

 

 

 

# 文件名：04-with-rbac.yaml
# Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置
# Ingress-Controller以Pod的形式运行，监控apiserver的/ingress接口后端的backend services，如果service发生变化，则Ingress-Controller自动更新转发规则。
# 基本逻辑如下：
#    1、监听apiserver，获取全部ingress定义；
#    2、基于ingress定义，生成nginx的配置文件/etc/nginx/nginx.conf；
#    3、执行nginx -s reload，重新加载nginx.conf配置文件的内容。
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  # replicas: 1
  # 部署改成2个副本
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          # 镜像路径调整【原先docker pull和docker tag好的】
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            # 增加default-backend-service，原先是没有，与defaultbackend结合起来
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

 

 

 

# 文件名：06-limit.yaml
---
apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container

 

 

 

# 文件名：07-node-port.yaml
# 添加NodePort端口，这里对外开放HTTP端口为：32080，HTTPS端口为：32443
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  # namespace: ingress-nginx
  namespace: kube-ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      # HTTP
      nodePort: 32080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      # HTTPS
      nodePort: 32443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

 

 

 

# 完整的mandatory.yaml内容

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---


kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "-"
      # Here: "-"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx


---


apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown


---


apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container