从零开始搭建K8S--搭建K8S Ingress
Ingress是个什么鬼，网上资料很多（推荐官方），大家自行研究。简单来讲，就是一个负载均衡的玩意，其主要用来解决使用NodePort暴露Service的端口时Node IP会漂移的问题。同时，若大量使用NodePort暴露主机端口，管理会非常混乱。
好的解决方案就是让外界通过域名去访问Service，而无需关心其Node IP及Port。那为什么不直接使用Nginx？这是因为在K8S集群中，如果每加入一个服务，我们都在Nginx中添加一个配置，其实是一个重复性的体力活，只要是重复性的体力活，我们都应该通过技术将它干掉。
Ingress就可以解决上面的问题，其包含两个组件Ingress Controller和Ingress：
- Ingress
将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可 - Ingress Controller
将新加入的Ingress转化成Nginx的配置文件并使之生效
好了，废话不多，走你~
准备操作
官方文档
人生苦短，不造轮子，本文将以官方的标准脚本为基础进行搭建，参考请戳官方文档。官方文档中要求依次执行如下命令：
-
curl
https:/
/raw.githubusercontent.com/kubernetes
/ingress-nginx/master
/deploy/namespace.yaml \
-
| kubectl apply -f -
-
-
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
-
| kubectl apply -f -
-
-
curl
https:/
/raw.githubusercontent.com/kubernetes
/ingress-nginx/master
/deploy/configmap.yaml \
-
| kubectl apply -f -
-
-
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
-
| kubectl apply -f -
-
-
curl
https:/
/raw.githubusercontent.com/kubernetes
/ingress-nginx/master
/deploy/udp-services-configmap.yaml \
-
| kubectl apply -f -
-
以上yaml文件创建Ingress用到的Namespace、ConfigMap，以及默认的后端default-backend。最关键的一点是，由于之前我们基于Kubeadm创建了K8S集群，则还必须执行：
-
curl
https:/
/raw.githubusercontent.com/kubernetes
/ingress-nginx/master
/deploy/rbac.yaml \
-
| kubectl apply -f -
-
-
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
-
| kubectl apply -f -
这是由于Kubeadm创建的集群默认开启了RABC，因此Ingress也必须创建相应的RABC权限控制。
导入镜像
但是，直接按照上述方式执行，我们的Ingress很可能会无法使用。所以，我们需要将上述Yaml文件全部wget下来，经过一些修改后才能执行kubectl apply -f创建。另外需要注意的是，这些yaml文件中提到的一些镜像，国内目前无法下载，如：
本人已经提前下载好，大家请戳：
-
地址：https:
//pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA
-
密码：
1a8a
拿到镜像后，在每个节点上执行如下命令导入镜像：
-
docker load < quay.io
#kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tar
-
docker tag
452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:
0.14
.0
-
docker load < gcr.io
#google_containers#defaultbackend.tar
-
docker tag
452a96d81c30 gcr.io/google_containers/defaultbackend
如上所示，导入镜像后，别忘记给打tag，否则镜像名称为
image.png
主要文件介绍
这里，我们先对一些重要的文件进行简单介绍。
default-backend.yaml
default-backend的作用是，如果外界访问的域名不存在的话，则默认转发到default-http-backend这个Service，其会直接返回404：
-
apiVersion: extensions/v1beta1
-
kind: Deployment
-
metadata:
-
name:
default-http-backend
-
labels:
-
app:
default-http-backend
-
namespace: ingress-nginx
-
spec:
-
replicas:
1
-
selector:
-
matchLabels:
-
app:
default-http-backend
-
template:
-
metadata:
-
labels:
-
app:
default-http-backend
-
spec:
-
terminationGracePeriodSeconds:
60
-
containers:
-
-
name:
default-http-backend
-
# Any image
is permissible
as long
as:
-
#
1. It serves a
404 page at /
-
#
2. It serves
200
on a /healthz endpoint
-
image: gcr.io/google_containers/defaultbackend:
1.4
-
livenessProbe:
-
httpGet:
-
path: /healthz
-
port:
8080
-
scheme: HTTP
-
initialDelaySeconds:
30
-
timeoutSeconds:
5
-
ports:
-
- containerPort:
8080
-
resources:
-
limits:
-
cpu:
10m
-
memory:
20Mi
-
requests:
-
cpu:
10m
-
memory:
20Mi
-
---
-
-
apiVersion: v1
-
kind: Service
-
metadata:
-
name:
default-http-backend
-
namespace: ingress-nginx
-
labels:
-
app:
default-http-backend
-
spec:
-
ports:
-
- port:
80
-
targetPort:
8080
-
selector:
-
app:
default-http-backend
rbac.yaml
rbac.yaml负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文《从零开始搭建Kubernetes集群》中，我们已对这些概念进行了简单介绍。
-
apiVersion: v1
-
kind: ServiceAccount
-
metadata:
-
name: nginx-ingress-serviceaccount
-
namespace: ingress-nginx
-
-
---
-
-
apiVersion: rbac.authorization.k8s.io/v1beta1
-
kind: ClusterRole
-
metadata:
-
name: nginx-ingress-clusterrole
-
rules:
-
- apiGroups:
-
- ""
-
resources:
-
- configmaps
-
- endpoints
-
- nodes
-
- pods
-
- secrets
-
verbs:
-
- list
-
- watch
-
- apiGroups:
-
- ""
-
resources:
-
- nodes
-
verbs:
-
- get
-
- apiGroups:
-
- ""
-
resources:
-
- services
-
verbs:
-
- get
-
- list
-
- watch
-
- apiGroups:
-
- "extensions"
-
resources:
-
- ingresses
-
verbs:
-
- get
-
- list
-
- watch
-
- apiGroups:
-
- ""
-
resources:
-
- events
-
verbs:
-
-
create
-
-
patch
-
- apiGroups:
-
-
"extensions"
-
resources:
-
- ingresses/
status
-
verbs:
-
-
update
-
-
---
-
-
apiVersion: rbac.authorization.k8s.io/v1beta1
-
kind:
Role
-
metadata:
-
name: nginx-ingress-
role
-
namespace: ingress-nginx
-
rules:
-
- apiGroups:
-
-
""
-
resources:
-
- configmaps
-
- pods
-
- secrets
-
- namespaces
-
verbs:
-
-
get
-
- apiGroups:
-
-
""
-
resources:
-
- configmaps
-
resourceNames:
-
#
Defaults
to
"
-"
-
# Here:
"
-"
-
# This has
to be adapted
if you
change either parameter
-
#
when launching the nginx-ingress-controller.
-
-
"ingress-controller-leader-nginx"
-
verbs:
-
-
get
-
-
update
-
- apiGroups:
-
-
""
-
resources:
-
- configmaps
-
verbs:
-
-
create
-
- apiGroups:
-
-
""
-
resources:
-
- endpoints
-
verbs:
-
-
get
-
-
---
-
-
apiVersion: rbac.authorization.k8s.io/v1beta1
-
kind: RoleBinding
-
metadata:
-
name: nginx-ingress-
role-nisa-binding
-
namespace: ingress-nginx
-
roleRef:
-
apiGroup: rbac.authorization.k8s.io
-
kind:
Role
-
name: nginx-ingress-
role
-
subjects:
-
- kind: ServiceAccount
-
name: nginx-ingress-serviceaccount
-
namespace: ingress-nginx
-
-
---
-
-
apiVersion: rbac.authorization.k8s.io/v1beta1
-
kind: ClusterRoleBinding
-
metadata:
-
name: nginx-ingress-clusterrole-nisa-binding
-
roleRef:
-
apiGroup: rbac.authorization.k8s.io
-
kind: ClusterRole
-
name: nginx-ingress-clusterrole
-
subjects:
-
- kind: ServiceAccount
-
name: nginx-ingress-serviceaccount
-
namespace: ingress-nginx
with-rbac.yaml
with-rbac.yaml是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置。
-
apiVersion: extensions/v1beta1
-
kind: Deployment
-
metadata:
-
name: nginx-ingress-controller
-
namespace: ingress-nginx
-
spec:
-
replicas:
1
-
selector:
-
matchLabels:
-
app: ingress-nginx
-
template:
-
metadata:
-
labels:
-
app: ingress-nginx
-
annotations:
-
prometheus.io/port:
'10254'
-
prometheus.io/scrape:
'true'
-
spec:
-
serviceAccountName: nginx-ingress-serviceaccount
-
containers:
-
-
name: nginx-ingress-controller
-
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:
0.14.
0
-
args:
-
- /nginx-ingress-controller
-
- --
default-backend-service=$(POD_NAMESPACE)/
default-http-backend
-
- --configmap=$(POD_NAMESPACE)/nginx-configuration
-
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
-
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
-
- --annotations-prefix=nginx.ingress.kubernetes.io
-
env:
-
-
name: POD_NAME
-
valueFrom:
-
fieldRef:
-
fieldPath: metadata.
name
-
-
name: POD_NAMESPACE
-
valueFrom:
-
fieldRef:
-
fieldPath: metadata.namespace
-
ports:
-
-
name: http
-
containerPort:
80
-
-
name: https
-
containerPort:
443
-
livenessProbe:
-
failureThreshold:
3
-
httpGet:
-
path: /healthz
-
port:
10254
-
scheme: HTTP
-
initialDelaySeconds:
10
-
periodSeconds:
10
-
successThreshold:
1
-
timeoutSeconds:
1
-
readinessProbe:
-
failureThreshold:
3
-
httpGet:
-
path: /healthz
-
port:
10254
-
scheme: HTTP
-
periodSeconds:
10
-
successThreshold:
1
-
timeoutSeconds:
1
-
securityContext:
-
runAsNonRoot: false
如上，可以看到nginx-ingress-controller启动时传入了参数，分别为前面创建的default-backend-service以及configmap。
创建Ingress
1.创建Ingress-controller
需要注意的是，官方提供的with-rbac.yaml文件不能直接使用，我们必须修改两处：
加入hostNetwork配置
如下，在serviceAccountName上方添加hostNetwork: true:
-
spec:
-
hostNetwork: true
-
serviceAccountName: nginx-ingress-serviceaccount
-
containers:
-
- name: nginx-ingress-controller
-
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
-
args:
-
- /nginx-ingress-controller
-
-
--default-backend-service=$(POD_NAMESPACE)/default-http-backend
-
-
--configmap=$(POD_NAMESPACE)/nginx-configuration
-
-
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
-
-
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
-
-
--annotations-prefix=nginx.ingress.kubernetes.io
配置hostNetwork: true是一种直接定义Pod网络的方式。定义后，Ingress-controller的IP就与宿主机k8s-node1一样(192.168.56.101)，并且端口80也是宿主机上的端口。这样，我们通过该192.168.56.101:80，就可以直接访问到Ingress-controller（实际上就是nginx），然后Ingress-controller则会转发我们的请求到相应后端。
加入环境变量
在其env部分加入如下环境变量：
-
env:
-
-
name: POD_NAME
-
valueFrom:
-
fieldRef:
-
fieldPath: metadata.
name
-
-
name: POD_NAMESPACE
-
valueFrom:
-
fieldRef:
-
fieldPath: metadata.namespace
-
-
name: KUBERNETES_MASTER
-
value: http:
//192.168.56.101:8080
否则，创建后会提示如下错误：
-
[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller
-9fbd7596d-rt9sf -n ingress-nginx
-
省略前面...
-
Events:
-
Type Reason Age From Message
-
---- ------ ---- ---- -------
-
Normal Scheduled
30s
default-scheduler Successfully assigned nginx-ingress-controller
-9fbd7596d-rt9sf to k8s-node1
-
Normal SuccessfulMountVolume
30s kubelet, k8s-node1 MountVolume.SetUp succeeded
for volume
"nginx-ingress-serviceaccount-token-lq2dt"
-
Warning BackOff
21s kubelet, k8s-node1 Back-off restarting failed container
-
Normal Pulled
11s (x3 over
29s) kubelet, k8s-node1 Container image
"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine
-
Normal Created
11s (x3 over
29s) kubelet, k8s-node1 Created container
-
Warning Failed
10s (x3 over
28s) kubelet, k8s-node1 Error: failed to start container
"nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux.
go:
348: starting container process caused
"exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown
修改with-rbac.yaml后，使用kubectl -f create命令分别执行如下yaml文件，即可创建Ingress-controller：
创建成功后如下所示：
-
[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wide
-
NAME READY STATUS RESTARTS AGE IP NODE
-
default-http-backend
-5c6d95c48-pdjn9
1/
1 Running
0
23s
192.168
.36
.81 k8s-node1
-
nginx-ingress-controller
-547cd7d9cb-jm
1/
1 Running
0
8s
192.168
.36
.82 k8s-node1
2.创建自定义Ingress
有了ingress-controller，我们就可以创建自定义的Ingress了。这里已提前搭建好了Kibana服务，我们针对Kibana创建一个Ingress：
-
apiVersion: extensions/v1beta1
-
kind: Ingress
-
metadata:
-
name: kibana-ingress
-
namespace:
default
-
-
spec:
-
rules:
-
- host: myk8s.com
-
http:
-
paths:
-
- path: /
-
backend:
-
serviceName: kibana
-
servicePort:
5601
其中：
- rules中的host必须为域名，不能为IP，表示Ingress-controller的Pod所在主机域名，也就是Ingress-controller的IP对应的域名。
- paths中的path则表示映射的路径。如映射
/表示若访问myk8s.com，则会将请求转发至Kibana的service，端口为5601。
创建成功后，查看：
-
[root@k8s-node1 ingress]#
kubectl
get
ingress
-o
wide
-
NAME
HOSTS
ADDRESS
PORTS
AGE
-
kibana-ingress
myk8s
.com 80 6
s
我们再执行kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx，可以看到生成nginx配置，篇幅较长，各位自行筛选：
-
## start server myk8s.com
-
server {
-
server_name myk8s.com ;
-
-
listen 80;
-
-
listen [::]:80;
-
-
set
$proxy_upstream_name
"-";
-
-
location /kibana {
-
-
log_by_lua_block {
-
-
}
-
-
port_in_redirect off;
-
-
set
$proxy_upstream_name
"";
-
-
set
$namespace
"kube-system";
-
set
$ingress_name
"dashboard-ingress";
-
set
$service_name
"kibana";
-
-
client_max_body_size
"1m";
-
-
proxy_set_header Host
$best_http_host;
-
-
# Pass the extracted client certificate to the backend
-
-
# Allow websocket connections
-
proxy_set_header Upgrade
$http_upgrade;
-
-
proxy_set_header Connection
$connection_upgrade;
-
-
proxy_set_header X-Real-IP
$the_real_ip;
-
-
proxy_set_header X-Forwarded-For
$the_real_ip;
-
-
proxy_set_header X-Forwarded-Host
$best_http_host;
-
proxy_set_header X-Forwarded-Port
$pass_port;
-
proxy_set_header X-Forwarded-Proto
$pass_access_scheme;
-
-
proxy_set_header X-Original-URI
$request_uri;
-
-
proxy_set_header X-Scheme
$pass_access_scheme;
-
-
# Pass the original X-Forwarded-For
-
proxy_set_header X-Original-Forwarded-For
$http_x_forwarded_for;
-
-
# mitigate HTTPoxy Vulnerability
-
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
-
proxy_set_header Proxy
"";
-
-
# Custom headers to proxied server
-
-
proxy_connect_timeout 5s;
-
proxy_send_timeout 60s;
-
proxy_read_timeout 60s;
-
-
proxy_buffering
"off";
-
proxy_buffer_size
"4k";
-
proxy_buffers 4
"4k";
-
proxy_request_buffering
"on";
-
-
proxy_http_version 1.1;
-
-
proxy_cookie_domain off;
-
proxy_cookie_path off;
-
-
# In case of errors try the next upstream server before returning an error
-
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
-
proxy_next_upstream_tries 0;
-
-
# No endpoints available for the request
-
return 503;
-
-
}
-
-
location / {
-
-
log_by_lua_block {
-
-
}
-
-
port_in_redirect off;
-
-
set
$proxy_upstream_name
"";
-
-
set
$namespace
"default";
-
set
$ingress_name
"kibana-ingress";
-
set
$service_name
"kibana";
-
-
client_max_body_size
"1m";
-
-
proxy_set_header Host
$best_http_host;
-
-
# Pass the extracted client certificate to the backend
-
-
# Allow websocket connections
-
proxy_set_header Upgrade
$http_upgrade;
-
-
proxy_set_header Connection
$connection_upgrade;
-
-
proxy_set_header X-Real-IP
$the_real_ip;
-
-
proxy_set_header X-Forwarded-For
$the_real_ip;
-
-
proxy_set_header X-Forwarded-Host
$best_http_host;
-
proxy_set_header X-Forwarded-Port
$pass_port;
-
proxy_set_header X-Forwarded-Proto
$pass_access_scheme;
-
-
proxy_set_header X-Original-URI
$request_uri;
-
-
proxy_set_header X-Scheme
$pass_access_scheme;
-
-
# Pass the original X-Forwarded-For
-
proxy_set_header X-Original-Forwarded-For
$http_x_forwarded_for;
-
-
# mitigate HTTPoxy Vulnerability
-
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
-
proxy_set_header Proxy
"";
-
-
# Custom headers to proxied server
-
-
proxy_connect_timeout 5s;
-
proxy_send_timeout 60s;
-
proxy_read_timeout 60s;
-
-
proxy_buffering
"off";
-
proxy_buffer_size
"4k";
-
proxy_buffers 4
"4k";
-
proxy_request_buffering
"on";
-
-
proxy_http_version 1.1;
-
-
proxy_cookie_domain off;
-
proxy_cookie_path off;
-
-
# In case of errors try the next upstream server before returning an error
-
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
-
proxy_next_upstream_tries 0;
-
-
# No endpoints available for the request
-
return 503;
-
-
}
-
-
}
-
## end server myk8s.com
3.设置host
首先，我们需要在Ingress-controller的Pod所在主机上（这里为k8s-node1），将上面提到的域名myk8s.com追加入/etc/hosts文件：
192.168.56.101 myk8s.com
除此之外，如果想在自己的Windows物理机上使用浏览器访问kibana，也需要在C:\Windows\System32\drivers\etc\hosts文件内加入上述内容。设置后，分别在k8s-node1和物理机上测试无误即可：
测试
在Windows物理机上，使用Chrome访问myk8s.com，也就是相当于访问了192.168.56.101:80：
随意访问一个错误的地址myk8s.com/abc，返回预期的404：
