kubernetes 1.11配置使用nginx ingress

[TOC]

1. 实验环境说明

lab1: etcd master haproxy keepalived 192.168.105.92
lab2: etcd master haproxy keepalived 192.168.105.93
lab3: etcd master haproxy keepalived 192.168.105.94
lab4: node  192.168.105.95
lab4: node  192.168.105.96

vip(loadblancer ip): 192.168.105.99

2. 暴露服务的方式

由于Pod和Service是kubernetes集群范围内的虚拟概念,所以集群外的客户端系统无法通过Pod的IP地址或者Service的虚拟IP地址和虚拟端口号访问到它们。为了让外部客户端能够访问到这些服务,可以将Pod或Service的端口号映射到宿主机。

  1. 将容器应用的端口号映射到物理机
    1)设置容器级别的hostPort,将容器应用的端口号映射到物理机
    2)设置Pod级别的hostNetwork=true,该Pod中所有容器的端口号都将被直接映射到物理机上

  2. 将Service的端口号映射到物理机
    1)设置nodePort映射到物理机,同时设置Service的类型为NodePort
    2)设置LoadBalancer映射到云服务商提供的LoadBalancer地址

如果设置了Service的nodePort,那么集群会在每一个节点都监听设置的nodePort,外部客户端可以通过任意 nodeIP:Port的方式对集群服务进行访问。但是当集群中服务较多,那么需要管理的端口也会比较多,各个端口之间不能冲突,比较麻烦;另外,因为方式访问形式为nodeIP:Port的方式,那么对于一些HTTP服务,这种方式是无法做到根据URL路径进行转发的。

  1. ingress是kubernetes V1.1版本之后新增的资源对象,用于实现HTTP层业务路由机制。
    实现ingress路由机制主要包括3个组件:
    1)ingress是kubernetes的一个资源对象,用于编写定义规则。
    2)反向代理负载均衡器,通常以Service的Port方式运行,接收并按照ingress定义的规则进行转发,通常为nginx,haproxy,traefik等,本文使用nginx。
    3)ingress-controller,监听apiserver,获取服务新增,删除等变化,并结合ingress规则动态更新到反向代理负载均衡器上,并重载配置使其生效。

3. 安装nginx ingress

一般是以DaemonSet形式在每个node上创建一个资源,然后以hostNetwork的方式开放一个端口。

下面是根据官方只修改了image地址和添加了hostNetwork: true的ingress-control的yaml文件。

#---
#
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "-"
      # Here: "-"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default 
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true  # 添加这行
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

4. 使用DaemonSet将ingress control固定pod

DaemonSet能够让所有(或者特定)的节点运行同一个pod。

当节点加入到K8S集群中,pod会被(DaemonSet)调度到该节点上运行,当节点从K8S集群中被移除,被DaemonSet调度的pod会被移除,如果删除DaemonSet,所有跟这个DaemonSet相关的pods都会被删除。

在某种程度上,DaemonSet承担了RC的部分功能,它也能保证相关pods持续运行,如果一个DaemonSet的Pod被杀死、停止、或者崩溃,那么DaemonSet将会重新创建一个新的副本在这台计算节点上。

一般应用于日志收集、监控采集、分布式存储守护进程、ingress等。

Deployment和DaemonSet的区别
Deployment 部署的副本 Pod 会分布在各个 Node 上,每个 Node 都可能运行好几个副本。DaemonSet 的不同之处在于:每个 Node 上最多只能运行一个副本。

主要区别:

  1. The scalability is much better when using a Deployment, because you will have a Single-Pod-per-Node model when using the DaemonSet.
  2. It is possible to exclusively run a Service on a dedicated set of machines using taints and tolerations with a DaemonSet.
  3. On the other hand the DaemonSet allows you to access any Node directly on Port 80 and 443, where you have to setup a Service object with a Deployment.

以下是只使用标签方式,将ingress control固定node。

将要使用的node节点打上标签

[root@lab1 nginx-ingress]# kubectl label nodes lab4 LB=ingress
node/lab4 labeled

修改ingress control的DeploymentDaemonSet,并根据标签使用nodeSelector

#---
#
## 有需要创建namespace,取消该段注释
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "-"
      # Here: "-"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---

apiVersion: extensions/v1beta1
kind: DaemonSet  # Deployment修改成Daemonset
metadata:
  name: nginx-ingress-controller
  namespace: default 
spec:
  #replicas: 2  # 注释该段
  #selector:
  #  matchLabels:
  #    app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      nodeSelector:  # 节点标签
        LB: ingress
      #tolerations:  # 声明可接受的污点
      #- key: "LB"
      #  operator: "Equal"
      #  value: "NIC"
      #  effect: "NoSchedule"
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true  # 添加这行,绑定到主机
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---

## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

5. 配置nginx url转发

以下是配置nginx域名ingress.test.com的url转发规则。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-nginx-ingress
  namespace: default
spec:
  rules:
  - host: ingress.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

  #    - path: /nginx
  #      backend:
  #        serviceName: nginx
  #        servicePort: 80

配置好后,kubectl get ing查看结果。

外部通过域名绑定Nginx Ingress controller所在的节点的IP即可访问。

参数资料:
[1] https://kubernetes.io/docs/concepts/services-networking/ingress/
[2] https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
[3] https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
[4] https://blog.csdn.net/waltonwang/article/details/77587003
[5] https://blog.csdn.net/tiger435/article/details/73650174
[6] https://blog.51cto.com/newfly/2067531