kubernetes中部署ingress-nginx访问k8s内部服务

一、系统环境

CentOS Linux release 7.9.2009 (Core)
kubectl-1.20.4-0.x86_64
kubelet-1.20.4-0.x86_64
kubeadm-1.20.4-0.x86_64
kubernetes-cni-0.8.7-0.x86_64

二、k8s架构

| 用途 | ip地址 | 主机名 |
| master | 192.168.10.127 | minio-4 |
| node01 | 192.168.10.124 | minio-1 |
| node02 | 192.168.10.125 | minio-2 |
| node03 | 192.168.10.126 | minio-3 |
| nfs存储 | 192.168.10.143 | |

三、Ingress Controller原理

具体可以参考：深入理解ingress-nginx - 赵SIR - 博客园

Ingress Controller有很多实现，我们这里采用官方维护的Nginx控制器。

部署文档：https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md

四、具体安装方法

1、下载官方nignx版本

wget -O nginx-ingress.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml

image.gif

2、修改下载后的nginx-ingress.yaml

vim nginx-ingress.yaml

image.gif

3、增加行 replicas: 2 副本数

spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  minReadySeconds: 0
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirst

image.gif

4、修改镜像为国内镜像

增加行 hostNetwork: true
image: pollyduan/ingress-nginx-controller:v0.41.2

minReadySeconds: 0
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: pollyduan/ingress-nginx-controller:v0.41.2
          #image: k8s.gcr.io/ingress-nginx/controller:v0.48.1@sha256:e9fb216ace49dfa4a5983b183067e97496e7a8b307d2093f4278cd550c303899
          imagePullPolicy: IfNotPresent
          lifecycle:

image.gif

5、节点服务器上添加标签 和 ingress-deploy.yaml 中保持一致否则部署时会出现错误

image

image.gif

在每个节点执行打标签命令

#kubectl label nodes master ingress-ready=true
#kubectl label nodes master kubernetes.io/os=linux --overwrite

image.gif

查看label,应该就有刚才执行的两个标签

# kubectl get nodes --show-labels
NAME      STATUS   ROLES                  AGE    VERSION   LABELS
minio-1   Ready    master                 118d   v1.20.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress-ready=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=minio-1,kubernetes.io/os=linux,node-role.kubernetes.io/master=true
minio-2   Ready    master                 118d   v1.20.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress-ready=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=minio-2,kubernetes.io/os=linux,node-role.kubernetes.io/master=true
minio-3   Ready    master                 101d   v1.20.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress-ready=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=minio-3,kubernetes.io/os=linux,node-role.kubernetes.io/master=true
minio-4   Ready    control-plane,master   118d   v1.20.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress-ready=false,kubernetes.io/arch=amd64,kubernetes.io/hostname=minio-4,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=

image.gif

image

image.gif

6、部署ingress-nginx服务

#kubectl create -f nginx-ingress.yaml

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

image.gif

查看运行情况

# kubectl get pod,service -n ingress-nginx -o wide
NAME                                            READY   STATUS      RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-bwg5x        0/1     Completed   0          72d   10.244.2.39      minio-2              
pod/ingress-nginx-admission-patch-958vt         0/1     Completed   0          72d   10.244.1.31      minio-1              
pod/ingress-nginx-controller-6ff975b7f7-2dcjh   1/1     Running     1          72d   192.168.10.126   minio-3              
pod/ingress-nginx-controller-6ff975b7f7-fwlql   1/1     Running     1          72d   192.168.10.124   minio-1              
pod/ingress-nginx-controller-6ff975b7f7-wdhkw   1/1     Running     1          72d   192.168.10.125   minio-2              

NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
service/ingress-nginx-controller             NodePort    10.10.26.171           80:30096/TCP,443:30113/TCP   72d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP   10.10.3.13             443/TCP                      72d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

image.gif

#kubectl get pods -n ingress-nginx

NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-bwg5x        0/1     Completed   0          72d
ingress-nginx-admission-patch-958vt         0/1     Completed   0          72d
ingress-nginx-controller-6ff975b7f7-2dcjh   1/1     Running     1          72d
ingress-nginx-controller-6ff975b7f7-fwlql   1/1     Running     1          72d
ingress-nginx-controller-6ff975b7f7-wdhkw   1/1     Running     1          72d

image.gif

如果pod状态有异样请使用kubectl describe pod pod名称 -n ingress-nginx 查看详情

#kubectl describe pod ingress-nginx-controller-6ff975b7f7-2dcjh -n ingress-nginx
Name:         ingress-nginx-controller-6ff975b7f7-2dcjh
Namespace:    ingress-nginx
Priority:     0
Node:         minio-3/192.168.10.126
Start Time:   Wed, 11 Aug 2021 13:52:07 +0800
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=6ff975b7f7
Annotations:  
Status:       Running
IP:           192.168.10.126
IPs:
  IP:           192.168.10.126
Controlled By:  ReplicaSet/ingress-nginx-controller-6ff975b7f7
Containers:
  controller:
    Container ID:  docker://9fcedbe86046d9dda1f3f26778efcd38ede942e341fc65dec1a83a07c5822e26
    Image:         pollyduan/ingress-nginx-controller:v0.41.2
    Image ID:      docker-pullable://pollyduan/ingress-nginx-controller@sha256:c7f8c2eaa157cc18f267c6c89a9e659f65ee7b347547a37c6471aff78ca22021
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 8443/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --publish-status-address=localhost
    State:          Running
      Started:      Wed, 15 Sep 2021 09:20:42 +0800
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6ff975b7f7-2dcjh (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-sjjs6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  ingress-nginx-token-sjjs6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-token-sjjs6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  ingress-ready=true
                 kubernetes.io/os=linux
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          

image.gif

6、测试一个服务

#前提你的nginx服务已经部署好

vim pvc.test.cn-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pvc.test.cn
  # 命名空间和代理的serviceName 所属命名空间保存一致，否则访问是会出现503错误
  namespace: default  
spec:
  rules:
  - host: pvc.test.cn
    http:
      paths:
      - backend:
          # 注册的服务名称
          serviceName: nginx-pvc-php74
          # 服务端口
          servicePort: 80

image.gif

#kubectl create -f pvc.test.cn-ingress.yaml

image.gif

查看状态

kubectl get ingress --all-namespaces
NAMESPACE    NAME                  CLASS    HOSTS                 ADDRESS     PORTS   AGE
default      pvc.test.cn             pvc.test.cn          localhost   80      44d

image.gif

在dns做一下域名解析

因为我这3台的node都装了ingress服务，所以dns上就解析了3条记录，这样就可以轮询

pvc     A   192.168.10.124
pvc     A   192.168.10.125
pvc     A   192.168.10.126

image.gif

打开浏览器输入pvc.test.cn