《kubernetes-1.8.0》12-addon-ingress-nginx

《kubernetes-1.8.0》12-addon-ingress-nginx

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-29

一、ingress介绍

关于nginx-ingress：

官网上摘抄几句：

Typically, services and pods have IPs only routable by the cluster network. All traffic that ends up at an edge router is either dropped or forwarded elsewhere. Conceptually, this might look like:
通常，services和Pod只能通过集群内网络访问。 所有在边界路由器上的流量都被丢弃或转发到别处。 从概念上讲，这可能看起来像：

    internet
        |
  ------------
  [ Services ]

An Ingress is a collection of rules that allow inbound connections to reach the cluster services.
Ingress是允许入站连接到达集群服务的一组规则。即介于物理网络和群集svc之间的一组转发规则。
其实就是实现L4 L7的负载均衡

    internet
        |
   [ Ingress ]
   --|-----|--
   [ Services ]

出来ingress以外，对外的服务发现还可以使用nodeport的方式，但这样会导致节点上的端口占用很多，且不利于管理。

It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting, and more. Users request ingress by POSTing the Ingress resource to the API server. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic in an HA manner.
它可以配置为提供外部可访问的URL，负载平衡流量，终止SSL，提供基于名称的虚拟主机，等等。 用户请求通过将Ingress resource发送到API server。 一个Ingress controller负责响应入口请求，通常是负载均衡方式，它也可能通过配置你的外部路由器或额外的前端，以帮助处理HA方式的流量。

二、ingress-nginx部署

ingress-nginx组件有几个部分组成：

• configmap.yaml:提供configmap可以在线更行nginx的配置
• default-backend.yaml:提供一个缺省的后台错误页面 404
• namespace.yaml:创建一个独立的命名空间 ingress-nginx
• rbac.yaml:创建对应的role rolebinding 用于rbac
• tcp-services-configmap.yaml:修改L4负载均衡配置的configmap
• udp-services-configmap.yaml:修改L4负载均衡配置的configmap
• with-rbac.yaml:有应用rbac的nginx-ingress-controller组件

相关yaml文件如下：

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

default-backend.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.mirrors.ustc.edu.cn/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

• github上 image: gcr.io/google_containers/defaultbackend:1.4 更换为 gcr.mirrors.ustc.edu.cn/google_containers/defaultbackend:1.4

configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx

tcp-services-configmap.yaml

apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx

udp-services-configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "-"
      # Here: "-"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

with-rbac.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.19
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
      nodeSelector:
        custom/ingress-controller-ready: "true"

这里做了一些改动，如下：

• kind: DaemonSet：官档给的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。测试环境考虑防止单点故障，改为DaemonSet，配合亲和性部署在制定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，后续将这些节点加入到外部硬件负载均衡组实现高可用性。
• hostNetwork: true：添加该字段，暴露nginx-ingress-controller pod的服务端口（80）
• nodeSelector: 加入亲和性部署，有custom/ingress-controller-ready label的才会部署该DaemonSet

测试环境中将三台master打上label

kubectl label nodes node.131 custom/ingress-controller-ready=true
kubectl label nodes node.132 custom/ingress-controller-ready=true
kubectl label nodes node.133 custom/ingress-controller-ready=true

加载对应yaml文件

kubectl create -f namespace.yaml
kubectl create -f default-backend.yaml
kubectl create -f configmap.yaml
kubectl create -f tcp-services-configmap.yaml
kubectl create -f udp-services-configmap.yaml
kubectl create -f rbac.yaml
kubectl create -f with-rbac.yaml

查看pod创建情况：

[root@node-131 deploy]#  kubectl get pods --all-namespaces -l app=ingress-nginx --watch
NAMESPACE       NAME                             READY     STATUS    RESTARTS   AGE
ingress-nginx   nginx-ingress-controller-5jxp5   1/1       Running   0          1h
ingress-nginx   nginx-ingress-controller-845z5   1/1       Running   0          1h
ingress-nginx   nginx-ingress-controller-wprgh   1/1       Running   0          1h

都running了就可以ctrl-c了

三、测试ingress

##查看default命名空间里的svc：
[root@node-131 ~]# kubectl get svc
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
default-http-backend   NodePort    10.254.196.136           80:45135/TCP   1d
demo-service           NodePort    10.254.23.220            80:31001/TCP   5d
kubernetes             ClusterIP   10.254.0.1               443/TCP        6d
nginx-service          NodePort    10.254.197.49            80:31000/TCP   5d
##之前案例中留下了 demo-service  、 nginx-service，直接拿来做测试：

通过域名进行url转发，对应yaml：

demo-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  namespace: default
spec:
  rules:
  - host: demo.bs.ingress
    http:
      paths:
      - path: /
        backend:
          serviceName: demo-service
          servicePort: 80
  - host: nginx.bs.ingress
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

• host: 对应的域名
• path: url上下文
• backend:后向转发 到对应的 serviceName: servicePort:

应用yaml文件：

kubectl apply -f demo-ingress.yaml 

##查看ingress resource：
[root@node-131 ingress]# kubectl get ing
NAME           HOSTS                              ADDRESS   PORTS     AGE
demo-ingress   demo.bs.ingress,nginx.bs.ingress             80        5h

修改hosts文件并测试访问：

echo '172.18.169.131 demo.bs.ingress' >> /etc/hosts
echo '172.18.169.132 nginx.bs.ingress' >> /etc/hosts

###ip为启动nginx-controller的节点


##测试访问：
[root@node-131 ingress]# curl nginx.bs.ingress

<html>
<head>
<title>Welcome to nginx!title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
style>
head>
<body>
<h1>Welcome to nginx!h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.orga>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.coma>.p>

<p><em>Thank you for using nginx.em>p>
body>
html>

------

[root@node-131 ingress]# curl demo.bs.ingress

<html lang="zh">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <title>Running!title>
    <style type="text/css">
        body {
            width: 100%;
            min-height: 100%;
            background: linear-gradient(to bottom, #fff 0, #b8edff 50%, #83dfff 100%);
            background-attachment: fixed;
        }
    style>
head>
<body class=" hasGoogleVoiceExt">
<div align="center">
    <h1>Your container is running!h1>
    <img src="./docker.png" alt="docker">
div>
body>
html>

更多ingress写法，查看官方文档：ingress

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考资料

https://github.com/kubernetes/ingress-nginx/tree/master/deploy

https://kubernetes.io/docs/concepts/services-networking/ingress/