Kubernetes之服务发现ingress & ingress controller

Service虽然解决了服务发现和负载均衡的问题,但它在使用上还是有一些限制:

  • 只支持4层负载均衡,没有7层功能
  • 对外访问时,NodePort类型需要在外部搭建额外的负载均衡,而LoadBalancer要求kubernetes必须跑在支持的cloud provider上

一、ingress引入历程

我们已经了解到Kubernetes暴露服务的方式目前只有三种:LoadBalancer Service、ExternalName、NodePort Service;那还有没有更好的方式呢?我们先来了解一下以下几个问题:

1、Pod漂移问题

我们知道Pod是有生命周期的,Pod宕机之后,其IP必然会改变,那我们如何把这个动态的Pod IP暴露出去呢?总所周知,我们可以借助Kubernetes的Service机制,在Service中以标签的形式选定一组带有指定标签的Pod,并监控和自动负载到它们的Pod IP,那么我们向外只需要暴露Service IP就行了,这就是我们常使用的NodePort模式:即在每个节点上开启一个端口,然后转发到内部的Pod IP上。此时的访问方式为:http: //nodeip:nodeport/
Kubernetes之服务发现ingress & ingress controller_第1张图片

2、端口管理问题

采用NodePort方式暴露服务面临的问题是:服务一旦躲起来,NodePort在每个节点上开启的端口会极其庞大,而且难以维护。想一想,鉴于nginx的能力,我们能够使用一个nginx直接对内进行转发呢?由于Pod与Pod之间是可以互相通信的,而Pod是可以共享宿主机的网络名称空间的,此时Pod上所监听的就是Node上的端口。那这又该如何实现呢?简单的实现就是使用DaemonSet在每个Node上监听80端口,然后写好规则,因为nginx外面绑定了宿主机80端口(就像NodePort),本身又在集群内,那么向后直接转发到相应Service IP就行了
Kubernetes之服务发现ingress & ingress controller_第2张图片
Kubernetes之服务发现ingress & ingress controller_第3张图片

3. 域名分配及动态更新问题

用上面的方法,我们采用nginx Pod似乎已经解决了问题,但是当每次有新服务加入又该如何修改nginx配置呢?nginx可以通过虚拟主机域名区分不同的服务,而每个服务通过upstream进行定义不同的负载均衡池,再加上location进行负载均衡的反向代理,在日常使用中只需要修改nginx.conf即可实现,但是在k8s中又该如何实现这种方式调度呢?
假设后端的服务开始只有ecshop,后面增加了bbs和member服务,那么又该如何将这2个服务加入到nginx Pod进行调度呢?总不能每次手动改后者Rolling update前端nginx pod吧!所以k8s引入了ingress,ingress就是为了解决这些问题而生的,ingress包含两大组件:ingress controller和ingress。

  • ingress
    简单理解就是你原来需要修改nginx配置,然后配置各种域名对应哪个Service,现在把这个动作抽象出来,变成一个Ingress对象,你可以用yaml创建,每次不要去修改nginx了,直接改yaml然后创建/更新就行了,那么问题又来了:nginx该如何处理呢?
  • ingress controller
    这个东西就是解决nginx如何处理这个问题的,ingress controller通过与kubernetes API交互,动态的去感知集群中Ingress规则变化,然后读取它,按照它自己的模板生成一段nginx配置,再写到nginx Pod中,最后reload以下,工作流程如下图
    Kubernetes之服务发现ingress & ingress controller_第4张图片
    图表说明:
    首先我们有一个外部的负载均衡器externalLB把请求调度到一个nodePort类型的Service(ingress-nginx)上,然后nodePort类型的Service(ingress-nginx)又把它调度到内部的叫做ingressController的Pod上,ingressCtroller根据ingress中的定义(虚拟主机还是URL),每一组主机名或者URL都对应后端的Pod资源,并且用Service分组(图中的site1/site2只是用来做分组用的)

二、Ingress Nginx部署

使用Ingress功能步骤:

  1. 安装部署ingressCtroller Pod

  2. 部署ingress-nginx service

  3. 部署后端服务

  4. 部署ingress
    从前面的描述我们知道,Ingress可以使用yaml的方式进行创建,从而得知Ingress也是标准的K8S资源,其定义的方式也可以使用kubectl explain进行查看:

     [root@master manifests]# kubectl explain ingress
     KIND:     Ingress
     VERSION:  extensions/v1beta1
    
     DESCRIPTION:
     Ingress is a collection of rules that allow inbound connections to reach
     the endpoints defined by a backend. An Ingress can be configured to give
     services externally-reachable urls, load balance traffic, terminate SSL,
     offer name based virtual hosting etc.
    
     FIELDS:
     apiVersion	
      APIVersion defines the versioned schema of this representation of an
      object. Servers should convert recognized schemas to the latest internal
      value, and may reject unrecognized values. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
    
     kind	
      Kind is a string value representing the REST resource this object
      represents. Servers may infer this from the endpoint the client submits
      requests to. Cannot be updated. In CamelCase. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds
    
     metadata	
      Standard object's metadata. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
    
     spec	
      Spec is the desired state of the Ingress. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
    
     status	
      Status is the current state of the Ingress. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
      
       
      

    2.1. 部署ingress controller

    ingress-nginx在github上的地址

    2.1.1. 下载ingress相关yaml文件

    [root@master ~]# mkdir ingress-nginx
    [root@master ~]# cd ingress-nginx/
    [root@master ingress-nginx]# for filr in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml default-backend.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
    [root@master ingress-nginx]# ls -l | awk '{print $9}'
    
    configmap.yaml						#用于为nginx从外部注入配置的
    default-backend.yaml				#配置默认后端服务
    namespace.yaml						#创建独立的名称空间
    rbac.yaml							#用于集群角色授权
    tcp-services-configmap.yaml
    udp-services-configmap.yaml
    with-rbac.yaml
    

    2.1.2. 创建ingress-nginx名称空间

    [root@k8s-master ingress-nginx]# cat namespace.yaml 
    apiVersion: v1
    kind: Namespace
    metadata:
      name: ingress-nginx
    
    ---
    [root@k8s-master ingress-nginx]# kubectl apply -f namespace.yaml 
    namespace/ingress-nginx created
    

    2.1.3. 创建ingress controller的Pod

    [root@master ingress-nginx]# kubectl apply -f .
    configmap/nginx-configuration created
    deployment.extensions/default-http-backend created
    service/default-http-backend created
    namespace/ingress-nginx configured
    serviceaccount/nginx-ingress-serviceaccount created
    clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured
    role.rbac.authorization.k8s.io/nginx-ingress-role created
    rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
    clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured
    configmap/tcp-services created
    configmap/udp-services created
    deployment.extensions/nginx-ingress-controller created
    

    此处遇到一个问题,新版本的Kubernetes在安装部署中,需要从k8s.grc.io仓库拉去所需镜像文件,但是由于国内防火墙问题导致无法正常拉去。
    辛亏docker.io仓库对google的容器做了mirror,可以通过下列命令拉取相关镜像:

    [root@node01 ~]# docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5
    1.5: Pulling from mirrorgooglecontainers/defaultbackend-amd64
    9ecb1e82bb4a: Pull complete 
    Digest: sha256:d08e129315e2dd093abfc16283cee19eabc18ae6b7cb8c2e26cc26888c6fc56a
    Status: Downloaded newer image for mirrorgooglecontainers/defaultbackend-amd64:1.5
    
    [root@node01 ~]# docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5
    [root@node01 ~]# docker image ls
    REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
    mirrorgooglecontainers/defaultbackend-amd64   1.5                 b5af743e5984        34 hours ago        5.13MB
    k8s.gcr.io/defaultbackend-amd64               1.5                 b5af743e5984        34 hours ago        5.13MB   
    

    2.2. 部署ingress-nginx service

    通过ingress-controller对外提供服务,现在还需要手动给ingress-controller建立一个servcie,接收集群外部流量。

    2.2.1 下载ingress-controller service的yaml文件

    [root@master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
    [root@master ingress-nginx]# cat service-nodeport.yaml 
    apiVersion: v1
    kind: Service
    metadata:
      name: ingress-nginx
      namespace: ingress-nginx
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      type: NodePort
      ports:
        - name: http
          port: 80
          targetPort: 80
          protocol: TCP
        - name: https
          port: 443
          targetPort: 443
          protocol: TCP
      selector:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    
    ---
    

    2.2.2. 创建ingress-controller的service

    [root@master ingress-nginx]# kubectl apply -f service-nodeport.yaml 
    service/ingress-nginx created
    [root@master ingress-nginx]# kubectl get svc -n ingress-nginx
    NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
    default-http-backend   ClusterIP   10.107.114.219           80/TCP                       27m
    ingress-nginx          NodePort    10.108.12.235            80:31875/TCP,443:32020/TCP   38s
    

    2.3. 部署ingress

    2.3.1. 查看ingress的配置清单选项

    [root@master ingress-nginx]# kubectl explain ingress.spec
    KIND:     Ingress
    VERSION:  extensions/v1beta1
    
    RESOURCE: spec 
    
    DESCRIPTION:
         Spec is the desired state of the Ingress. More info:
         https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
    
         IngressSpec describes the Ingress the user wishes to exist.
    
    FIELDS:
       backend       #定义后端有哪几个主机
         A default backend capable of servicing requests that don't match any rule.
         At least one of 'backend' or 'rules' must be specified. This field is
         optional to allow the loadbalancer controller or defaulting logic to
         specify a global default.
    
       rules    <[]Object>    #定义规则
         A list of host rules used to configure the Ingress. If unspecified, or no
         rule matches, all traffic is sent to the default backend.
    
       tls  <[]Object>
         TLS configuration. Currently the Ingress only supports a single TLS port,
         443. If multiple members of this list specify different hosts, they will be
         multiplexed on the same port according to the hostname specified through
         the SNI TLS extension, if the ingress controller fulfilling the ingress
         supports SNI.
     
      

    2.3.2. 编写ingress的配置清单并创建ingress

    [root@master ingress-nginx]# vim ingress-default-backend.yaml
    apiVersion: extensions/v1beta1				#api版本
    kind: Ingress								#清单类型
    metadata:									#元数据
      name: ingress-default-backend				#ingress名称
      namespace: ingress-nginx					#所属命名空间
      annotations:								#注解信息
        kubernetes.io/ingress.class: "nginx"
    spec:										#规格
      rules:									#定义后端转发规则
      - host: default.backend.com				#通过域名进行转发
        http:
          paths:
            - path:								#配置访问路径
              backend:							#配置后端服务
                serviceName: default-http-backend
                servicePort: 80 
    [root@master ingress-nginx]# kubectl apply -f ingress-default-backend.yaml 
    ingress.extensions/ingress-default-backend created
    

    查看ingress-default-backend的详细信息:

    [root@master ingress-nginx]# kubectl describe ingress -n ingress-nginx ingress-default-backend
    Name:             ingress-default-backend
    Namespace:        ingress-nginx
    Address:          
    Default backend:  default-http-backend:80 ()
    Rules:
      Host                 Path  Backends
      ----                 ----  --------
      default.backend.com  
                              default-http-backend:80 ()
    Annotations:
      kubernetes.io/ingress.class:                       nginx
      kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}}
    
    Events:
      Type    Reason  Age   From                      Message
      ----    ------  ----  ----                      -------
      Normal  CREATE  9m    nginx-ingress-controller  Ingress ingress-nginx/ingress-default-backend
    

    进入nginx-ingress-controller进行查看是否注入了nginx配置:

    [root@master ingress-nginx]# kubectl get pods -n ingress-nginx
    NAME                                        READY     STATUS    RESTARTS   AGE
    default-http-backend-7db7c45b69-znpxp       1/1       Running   0          1h
    nginx-ingress-controller-6bd7c597cb-fgdz8   1/1       Running   0          1h
    [root@master ingress-nginx]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-6bd7c597cb-fgdz8 -- /bin/sh
    $ cat nginx.conf
    ....
    ....
    ## start server default.backend.com
    server {
    	server_name default.backend.com ;
    	
    	listen 80;
    	
    	set $proxy_upstream_name "-";
    	
    	location / {
    		
    		set $namespace      "ingress-nginx";
    		set $ingress_name   "ingress-default-backend";
    		set $service_name   "default-http-backend";
    		set $service_port   "80";
    		set $location_path  "/";
    		
    		rewrite_by_lua_block {
    			
    			balancer.rewrite()
    			
    		}
    		
    		log_by_lua_block {
    			
    			balancer.log()
    			
    			monitor.call()
    		}
    		
    		port_in_redirect off;
    		
    		set $proxy_upstream_name "ingress-nginx-default-http-backend-80";
    		
    		client_max_body_size                    "1m";
    		
    		proxy_set_header Host                   $best_http_host;
    		
    		# Pass the extracted client certificate to the backend
    		
    		# Allow websocket connections
    		proxy_set_header                        Upgrade           $http_upgrade;
    		
    		proxy_set_header                        Connection        $connection_upgrade;
    		
    		proxy_set_header X-Request-ID           $req_id;
    		proxy_set_header X-Real-IP              $the_real_ip;
    		
    		proxy_set_header X-Forwarded-For        $the_real_ip;
    		
    		proxy_set_header X-Forwarded-Host       $best_http_host;
    		proxy_set_header X-Forwarded-Port       $pass_port;
    		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
    		
    		proxy_set_header X-Original-URI         $request_uri;
    		
    		proxy_set_header X-Scheme               $pass_access_scheme;
    		
    		# Pass the original X-Forwarded-For
    		proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
    		
    		# mitigate HTTPoxy Vulnerability
    		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
    		proxy_set_header Proxy                  "";
    		
    		# Custom headers to proxied server
    		
    		proxy_connect_timeout                   5s;
    		proxy_send_timeout                      60s;
    		proxy_read_timeout                      60s;
    		
    		proxy_buffering                         "off";
    		proxy_buffer_size                       "4k";
    		proxy_buffers                           4 "4k";
    		proxy_request_buffering                 "on";
    		
    		proxy_http_version                      1.1;
    		
    		proxy_cookie_domain                     off;
    		proxy_cookie_path                       off;
    		
    		# In case of errors try the next upstream server before returning an error
    		proxy_next_upstream                     error timeout;
    		proxy_next_upstream_tries               3;
    		
    		proxy_pass http://upstream_balancer;
    		
    		proxy_redirect                          off;
    		
    	}
    	
    }
    ## end server default.backend.com
    ....
    ....
    

    可以发现我们刚刚部署的ingress的的确确已经注入到nginx-ingress-controller的nginx配置中了

    2.3.3. 通过ingress访问服务

    修改本地host文件:

    192.168.116.128 default.backend.com
    192.168.116.129 default.backend.com
    

    Kubernetes之服务发现ingress & ingress controller_第5张图片

    三、构建TLS站点

    3.1. 添加tomcat服务

    [root@master ingress-nginx]# vim tomcat-demo.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: tomcat
      namespace: default
    spec:
      selector:
        app: tomcat
        release: stable
      ports:
      - name: http
        port: 8080
        targetPort: 8080
      - name: ajp
        port: 8009
        targetPort: 8009
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: tomcat-demo
      namespace: default
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: tomcat
          release: stable
      template:
        metadata:
          labels:
            app: tomcat
            release: stable
        spec:
          containers:
          - name: tomcat
            image: tomcat:8.5.34-jre8-alpine
            ports:
            - name: http
           	  containerPort: 8080
            - name: ajp
          	  containerPort: 8009
    [root@master ingress-nginx]# kubectl create -f tomcat-demo.yaml 
    service/tomcat created
    deployment.apps/tomcat-demo created
    [root@master ingress-nginx]# kubectl get svc
    NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
    kubernetes   ClusterIP   10.96.0.1               443/TCP             12d
    tomcat       ClusterIP   10.108.64.162           8080/TCP,8009/TCP   8s
    [root@master ingress-nginx]# kubectl get deployments
    NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    tomcat-demo   3         3         3            0           16s
    [root@master ingress-nginx]# kubectl get pods
    NAME                           READY     STATUS    RESTARTS   AGE
    tomcat-demo-798cdf69b9-qk9pd   1/1       Running   0          7m
    tomcat-demo-798cdf69b9-t642t   1/1       Running   0          7m
    tomcat-demo-798cdf69b9-tlfq6   1/1       Running   0          7m
    

    3.2. 准备证书并生成secret

    准备证书:

    [root@master ingress-nginx]# openssl genrsa -out tls.key 2048
    Generating RSA private key, 2048 bit long modulus
    ............................+++
    ..................+++
    e is 65537 (0x10001)
    [root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.zwx.com
    

    生成secret:

    [root@master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
    secret/tomcat-ingress-secret created
    [root@master ingress-nginx]# kubectl get secret
    NAME                    TYPE                                  DATA      AGE
    default-token-l9jws     kubernetes.io/service-account-token   3         12d
    tomcat-ingress-secret   kubernetes.io/tls                     2         33s
    [root@master ingress-nginx]# kubectl describe secret tomcat-ingress-secret
    Name:         tomcat-ingress-secret
    Namespace:    default
    Labels:       
    Annotations:  
    
    Type:  kubernetes.io/tls
    
    Data
    ====
    tls.crt:  1285 bytes
    tls.key:  1679 bytes
    

    3.3. 创建ingress

    [root@master ingress-nginx]# vim ingress-tomcat-tls.yaml
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-tomcat-tls
      namespace: default
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - tomcat.zwx.com
        secretName: tomcat-ingress-secret
      rules:
      - host: tomcat.zwx.com
        http:
          paths:
          - backend:
              serviceName: tomcat
              servicePort: 8080
    [root@master ingress-nginx]# kubectl apply -f ingress-tomcat-tls.yaml 
    ingress.extensions/ingress-tomcat-tls created
    

    3.4. 访问测试

    查看ingress-nginx 服务

    [root@master ingress-nginx]# kubectl get svc -n ingress-nginx -o wide
    NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE       SELECTOR
    default-http-backend   ClusterIP   10.107.114.219           80/TCP                       4h        app.kubernetes.io/name=default-http-backend,app.kubernetes.io/part-of=ingress-nginx
    ingress-nginx          NodePort    10.108.12.235            80:31875/TCP,443:32020/TCP   3h        app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
    

    所以访问https://tomcat.zwx.com:32020
    Kubernetes之服务发现ingress & ingress controller_第6张图片

    你可能感兴趣的:(kubernetes)