Nginx Ingress TCP代理实现

一前言

一般使用ingress都是代理http流量，但是有些场景希望代理tcp流量，例如：不想占用过多的公网IP。

开源的ingress对tcp支持不是很好，主要原因在于k8s的Ingress没有给tcp留下插入点，可以通过ingress定义 kubectl explain ingress.spec.rules 证实。

ingress http代理简单来说，暴露一个http服务，根据host和path转发用户请求到真正的svc（用户请求带有host）。tpc代理就是暴露一堆端口号，不同的端口对应不同的后端svc。

二 nginx ingress使用

官网暴露TCP服务章节，介绍可以通过--tcp-services-configmap暴露tcp服务，具体怎么使用没有实践之前一直不是很理解。

2.1 启动参数

通过chart安装包可以获取nginx-ingress-controller deployment启动配置。

cat < tmpconfig.yaml
tcp:
  8080: "default/apple-service:5678"
EOF
#helm template tcpproxy k8s/nginx-ingress -f tmpconfig.yaml
#安装
helm install  tcpproxy k8s/nginx-ingress -f tmpconfig.yaml

deploy脚本示例

# Source: nginx-ingress/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.34.2
    heritage: Helm
    release: tcpproxy
    app.kubernetes.io/component: controller
  name: tcpproxy-nginx-ingress-controller
  annotations:
    {}
spec:
  selector:
    matchLabels:
      app: nginx-ingress
      release: tcpproxy
  replicas: 1
  revisionHistoryLimit: 10
  strategy:
    {}
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app: nginx-ingress
        release: tcpproxy
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      containers:
        - name: nginx-ingress-controller
          image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - /nginx-ingress-controller
            - --default-backend-service=default/tcpproxy-nginx-ingress-default-backend
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=default/tcpproxy-nginx-ingress-controller
            - --tcp-services-configmap=default/tcpproxy-nginx-ingress-tcp

2.2 查看ConfigMap配置

tcp的相关配置通过configmap存储，需要注意data属性，controller会解析它。

[root@test ~]# kubectl get cm tcpproxy-nginx-ingress-tcp -o yaml
apiVersion: v1
data:
  "8080": default/apple-service:5678
kind: ConfigMap
metadata:
  creationTimestamp: "2020-03-24T02:24:29Z"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.34.2
    component: controller
    heritage: Helm
    release: tcpproxy
  name: tcpproxy-nginx-ingress-tcp

查看nginx配置

登陆controller的Pod,直接查看nginx.conf，在最后一行，可以看到nginx代理配置。直接通过curl localhost:8080，可以正常访问服务。

kubectl exec -it tcpproxy-nginx-ingress-controller-7ff6b85d96-h58ww  /bin/sh

nginx.conf示例

    # TCP services
    
    server {
        preread_by_lua_block {
            ngx.var.proxy_upstream_name="tcp-default-apple-service-5678";
        }
        
        listen                  8080;
        
        proxy_timeout           600s;
        proxy_pass              upstream_balancer;
        
    }
    
    # UDP services
    
}

三实现分析

整体架构可以参考
https://blog.csdn.net/shida_csdn/article/details/84032019

3.1 同步

NGINXController有个channel,所有更新事件通过watch传到这个channel；同时channel通过queue绑定NGINXController的syncIngress，用于处理变更事件。

func (n *NGINXController) syncIngress(interface{}) error {
    n.syncRateLimiter.Accept()

    if n.syncQueue.IsShuttingDown() {
        return nil
    }

    ings := n.store.ListIngresses(nil)
    // pcfg里包含tcpendpoints
    hosts, servers, pcfg := n.getConfiguration(ings)

func (n *NGINXController) getConfiguration(ingresses []*ingress.Ingress) (sets.String, []*ingress.Server, *ingress.Configuration) {
    upstreams, servers := n.getBackendServers(ingresses)
    var passUpstreams []*ingress.SSLPassthroughBackend

    hosts := sets.NewString()
    // ...
    return hosts, servers, &ingress.Configuration{
        Backends:              upstreams,
        Servers:               servers,
        //获取tcp的代理服务
        TCPEndpoints:          n.getStreamServices(n.cfg.TCPConfigMapName, apiv1.ProtocolTCP),
        UDPEndpoints:          n.getStreamServices(n.cfg.UDPConfigMapName, apiv1.ProtocolUDP),
        PassthroughBackends:   passUpstreams,
        BackendConfigChecksum: n.store.GetBackendConfiguration().Checksum,
        ControllerPodsCount:   n.store.GetRunningControllerPodsCount(),
    }
}

func (n *NGINXController) getStreamServices(configmapName string, proto apiv1.Protocol) []ingress.L4Service {
    // 禁止业务上使用的端口，防止跟ingress的内部服务冲突
    rp := []int{
        n.cfg.ListenPorts.HTTP,
        n.cfg.ListenPorts.HTTPS,
        n.cfg.ListenPorts.SSLProxy,
        n.cfg.ListenPorts.Health,
        n.cfg.ListenPorts.Default,
        nginx.ProfilerPort,
        nginx.StatusPort,
        nginx.StreamPort,
    }

    reserverdPorts := sets.NewInt(rp...)
    // 解析tcp configmap的data字段
    for port, svcRef := range configmap.Data {
        externalPort, err := strconv.Atoi(port)
        //。。。
        if reserverdPorts.Has(externalPort) {
            klog.Warningf("Port %d cannot be used for %v stream services. It is reserved for the Ingress controller.", externalPort, proto)
            continue
        }
        nsSvcPort := strings.Split(svcRef, ":")
        //...
        // 获取ns
        svcNs, svcName, err := k8s.ParseNameNS(nsName)
      // 获取svc
        svc, err := n.store.GetService(nsName)
        var endps []ingress.Endpoint
        targetPort, err := strconv.Atoi(svcPort)
        
        svcs = append(svcs, ingress.L4Service{
            Port: externalPort,
            Backend: ingress.L4Backend{
                Name:          svcName,
                Namespace:     svcNs,
                Port:          intstr.FromString(svcPort),
                Protocol:      proto,
                ProxyProtocol: svcProxyProtocol,
            },
            Endpoints: endps,
            Service:   svc,
        })
    }
    // Keep upstream order sorted to reduce unnecessary nginx config reloads.
    sort.SliceStable(svcs, func(i, j int) bool {
        return svcs[i].Port < svcs[j].Port
    })
    return svcs
}

3.2 监听

有关watch的初始化在store.go中实现，当key的名称为tpcconfigmap时，会触发更新。
internal/ingress/controller/store/store.go

    // tcp = tcpproxy-nginx-ingress-tcp
    changeTriggerUpdate := func(name string) bool {
        return name == configmap || name == tcp || name == udp
    }
    
    handleCfgMapEvent := func(key string, cfgMap *corev1.ConfigMap, eventName string) {
        // updates to configuration configmaps can trigger an update
        triggerUpdate := false
        if changeTriggerUpdate(key) {
        // 设置触发更新
            triggerUpdate = true
            recorder.Eventf(cfgMap, corev1.EventTypeNormal, eventName, fmt.Sprintf("ConfigMap %v", key))
            if key == configmap {
                store.setConfig(cfgMap)
            }
        }

        ings := store.listers.IngressWithAnnotation.List()
        for _, ingKey := range ings {
            key := k8s.MetaNamespaceKey(ingKey)
            ing, err := store.getIngress(key)
            if err != nil {
                klog.Errorf("could not find Ingress %v in local store: %v", key, err)
                continue
            }

            if parser.AnnotationsReferencesConfigmap(ing) {
                store.syncIngress(ing)
                continue
            }
            // 触发同步
            if triggerUpdate {
                store.syncIngress(ing)
            }
        }

        if triggerUpdate {
            updateCh.In() <- Event{
                Type: ConfigurationEvent,
                Obj:  cfgMap,
            }
        }
    }

四问题

1）在tcp configmap手动新增配置，ingress contorller svc会不会动态改变？

更改configmap后，nginx.conf会更新，当然服务也可以访问通。
svc，pod不会动态新增端口。

更多文章见:http://huiwq1990.github.io/

Nginx Ingress TCP代理实现

一 前言