metallb简介
官方网站：https://metallb.universe.tf/
- MetalLB是使用标准路由协议的裸机Kubernetes集群的软负载均衡器，目前处于测试版本阶段。
私有云裸金属架构的kubernetes集群不支持LoadBalance
Kubernetes没有为裸机群集提供网络负载均衡器（类型为LoadBalancer的服务）的实现，如果你的kubernetes集群没有在公有云的IaaS平台（GCP，AWS，Azure …）上运行，则LoadBalancers将在创建时无限期地保持“挂起”状态，也就是说只有公有云厂商自家的kubernetes支持LoadBalancer。
裸机群集运营商留下了两个较小的工具来将用户流量带入其集群，“NodePort”和“externalIPs”服务。这两种选择都对生产使用产生了重大影响，这使得裸露的金属集群成为Kubernetes生态系统中的二等公民。
部署metallb
metallb 目前有两种部署方式，支持 Helm 和 YAML 两种安装方法
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
第一次安装还需要运行以下：
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
查看：
# kubectl get pod -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-684f5d9b49-x859t 1/1 Running 0 6h3m
speaker-lmrgx 1/1 Running 0 6h3m
speaker-x9jk5 1/1 Running 0 6h3m
创建ConfigMap
- 下载 config.yaml 模版
wget https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/example-layer2-config.yaml
- 这里直接生成
cat >> config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: my-ip-space
protocol: layer2
addresses:
- 192.168.200.130-192.168.200.140
EOF
创建：
# kubectl apply -f config.yaml
查看 configMap 情况
# kubectl describe ConfigMap config -n metallb-system
Name: config
Namespace: metallb-system
Labels:
Annotations:
Data
====
config:
----
address-pools:
- name: my-ip-space
protocol: layer2
addresses:
- 192.168.200.130-192.168.200.140
Events:
至此，LoadBalancer的配置工作就完成
创建service测试 LoadBalancer
这里创建一个 nginx 服务，包含deployment和一个LoadBalancer类型的service
# vim nginx-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer
查看service分配的EXTERNAL-IP：
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 443/TCP 4d5h
nginx LoadBalancer 10.0.0.86 192.168.200.130 80:32295/TCP 6h1m
集群外的机器访问EXTERNAL-IP 192.168.200.130 成功即可
部署 ingress-nginx 控制器
1、安装
官方文档 https://kubernetes.github.io/ingress-nginx/deploy/
下载
# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml
# kubctl apply -f mandatory.yaml
创建 NodePort 模式的 service
# kubctl apply -f service-nodeport.yaml
集群有LB的可以创建 LoadBalarcer 模式 service
# cp service-nodeport.yaml service-lb.yaml
# vim service-lb.yaml
spec:
#type: NodePort
type: LoadBalancer
...
# kubctl apply -f service-lb.yaml
检查是否安装：
# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-69fb496d7d-t9n6g 1/1 Running 0 18m
2、部署 Ingress
这里创建一个nginx deployment 和 ingress
# vim ingress-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1
ports:
- name: http
containerPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
spec:
rules:
- host: k8s.alwooo.cn
http:
paths:
- path: /test
backend:
serviceName: nginx
servicePort: 80
创建：
# kubectl apply -f ingress-test.yaml
Error from server (InternalError): error when creating "ingress-test.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.0.0.171:443: connect: connection timed out
这里发现报错了
错误解决：
查看 node 节点的 kube-proxy
# systemctl status kube-proxy
6月 09 10:17:57 kub-node2 kube-proxy[28434]: E0609 10:17:57.325032 28434 node.go:125] Failed to retrieve node info: nodes "kube-node2" not found
发现其中一台节点 kube-proxy 报错，是配置文件 hostnameOverride 错误导致，修改即可
# vim /opt/kubernetes/cfg/kube-porxy-config.yaml
...
hostnameOverride: kub-node2 # 修改为正确的主机名
...
# systemctl restart kube-proxy
重新部署 ingress 还是报错。
判断是 master 节点无法访问 10.0.0.171:443
查看ingress 服务
# kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.0.0.196 80:31217/TCP,443:30348/TCP 12s
ingress-nginx-controller-admission ClusterIP 10.0.0.171 443/TCP 12s
svervice 为 NodePort 方式，修改为 LoadBalancer
# vim deploy.yaml
...
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-2.0.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.32.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: LoadBalancer # 修改此字段
ports:
- name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
...
重新部署后，还是一样。
经过多番查阅后发现，是master节点没有部署 kubelet, kube-proxy 导致。必须把master节点当成worker节点。
重新部署kubelet 和 kube-proxy 到master节点后。部署ingerss服务成功
# kubectl apply -f ingress-test.yaml
deployment.apps/nginx created
ingress.extensions/test-ingress created
# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
test-ingress k8s.alwooo.cn 192.168.200.120 80 41s
如果 service 为 NodePort 模式，则客户端域名指向的地址应为 ADDRESS
如使用是 LoadBalancer 则使用 LB 地址作为 VIP，映射到 ASSRESS
# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.0.0.207 192.168.200.131 80:30859/TCP,443:30756/TCP 57m
ingress-nginx-controller-admission ClusterIP 10.0.0.242 443/TCP 57m
其他坑 https://www.cnblogs.com/tchua/p/10844013.html
配置 ingress 处理 TLS 传输
- 创建私钥和证书
# openssl genrsa -out tls.key 2048
# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=GuangDong/L=ZhongShan/O=Anlewo/CN=tomcat.alwooo.cn
- 创建 k8s Secret
kubectl create secret tls ingress-secret --cert=tls.crt --key=tls.key
- 添加部署文件
# cat tomcat-ingress-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat-tls
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- tomcat.alwooo.cn
secretName: tomcat-ingress-secret
rules:
- host: tomcat.alwooo.cn
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
部署完后就可以用 https 访问了