Kubernetes Service 有3种暴露服务到外网的方式:
- NodePort 模式
- Ingress 模式
- LoadBalancer 模式
其中 NodePort 和 Ingress 模式均可在测试环境模拟, 但是 LoadBalancer 模式依赖于云厂商. 如果是在本地或者私有云中部署试用Kubernetes, 那么将无法直接使用.
MetalLB 提供了这样一种模拟环境. 他嵌入 k8s 中, 允许建立 LoadBalancer模式的 Service. 下面做一个简单 demo.
一. 安装
1.1 准备工作
首先必须已经成功安装了 kubernetes 集群.
其次, 如果 kube-proxy 使用了 IPVS 方式, 那么需要开启严格 ARP 模式.
1.2 安装
应用以下 Manifest:
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"以上文件 namespace.yaml 和 metallb.yaml 在墙内无法直接打开和下载, 可见文末附录部分.
准备和安装部分更多细节参考官方文档.
二. 配置
网络配置支持Layer 2 和 BGP, 测试环境一般使用 Layer 2. 支持2种地址池配置方式:
- 掩码方式: NN.NN.NN.NN/NN
- 地址范围: NN.NN.NN.NN-NN.NN.NN.NN
具体的地址范围根据 k8s 集群所在的网络决定. 比如集群所在局域网 IP 范围为 192.168.31.255/24, 那么可以设置为其中一部分 192.168.31.254/28 (或 192.168.31.240-192.168.31.254)
本地新建 metallb-l2-config.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.31.254/28应用该配置:
kubectl apply -f metallb-l2-config.yaml更多有关网络配置相关信息可参考官方文档 和 官方示例 (如无法打开, 见附录3).
三. 附录
3.1 MetalLB 的安装配置文件
1) namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
app: metallb2) metallb.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities: []
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 65535
min: 1
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities:
- NET_ADMIN
- NET_RAW
- SYS_ADMIN
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
rule: RunAsAny
hostIPC: false
hostNetwork: true
hostPID: false
hostPorts:
- max: 7472
min: 7472
privileged: true
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:controller
rules:
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- update
- apiGroups:
- ''
resources:
- services/status
verbs:
- update
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- controller
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:speaker
rules:
- apiGroups:
- ''
resources:
- services
- endpoints
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- speaker
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:controller
subjects:
- kind: ServiceAccount
name: controller
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:speaker
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:speaker
subjects:
- kind: ServiceAccount
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: config-watcher
subjects:
- kind: ServiceAccount
name: controller
- kind: ServiceAccount
name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-lister
subjects:
- kind: ServiceAccount
name: speaker
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: metallb
component: speaker
name: speaker
namespace: metallb-system
spec:
selector:
matchLabels:
app: metallb
component: speaker
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
component: speaker
spec:
containers:
- args:
- --port=7472
- --config=config
env:
- name: METALLB_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: METALLB_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: METALLB_ML_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: METALLB_ML_LABELS
value: "app=metallb,component=speaker"
- name: METALLB_ML_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: METALLB_ML_SECRET_KEY
valueFrom:
secretKeyRef:
name: memberlist
key: secretkey
image: metallb/speaker:v0.9.3
imagePullPolicy: Always
name: speaker
ports:
- containerPort: 7472
name: monitoring
resources:
limits:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_ADMIN
- NET_RAW
- SYS_ADMIN
drop:
- ALL
readOnlyRootFilesystem: true
hostNetwork: true
nodeSelector:
beta.kubernetes.io/os: linux
serviceAccountName: speaker
terminationGracePeriodSeconds: 2
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: metallb
component: controller
name: controller
namespace: metallb-system
spec:
revisionHistoryLimit: 3
selector:
matchLabels:
app: metallb
component: controller
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
component: controller
spec:
containers:
- args:
- --port=7472
- --config=config
image: metallb/controller:v0.9.3
imagePullPolicy: Always
name: controller
ports:
- containerPort: 7472
name: monitoring
resources:
limits:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
nodeSelector:
beta.kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: controller
terminationGracePeriodSeconds: 03) example-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
# The peers section tells MetalLB what BGP routers to connect too. There
# is one entry for each router you want to peer with.
peers:
- # The target IP address for the BGP session.
peer-address: 10.0.0.1
# The BGP AS number that MetalLB expects to see advertised by
# the router.
peer-asn: 64512
# The BGP AS number that MetalLB should speak as.
my-asn: 64512
# (optional) the TCP port to talk to. Defaults to 179, you shouldn't
# need to set this in production.
peer-port: 179
# (optional) The proposed value of the BGP Hold Time timer. Refer to
# BGP reference material to understand what setting this implies.
hold-time: 120s
# (optional) The router ID to use when connecting to this peer. Defaults
# to the node IP address. Generally only useful when you need to peer with
# another BGP router running on the same machine as MetalLB.
router-id: 1.2.3.4
# (optional) Password for TCPMD5 authenticated BGP sessions
# offered by some peers.
password: "yourPassword"
# (optional) The nodes that should connect to this peer. A node
# matches if at least one of the node selectors matches. Within
# one selector, a node matches if all the matchers are
# satisfied. The semantics of each selector are the same as the
# label- and set-based selectors in Kubernetes, documented at
# https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/.
# By default, all nodes are selected.
node-selectors:
- # Match by label=value
match-labels:
kubernetes.io/hostname: prod-01
# Match by 'key OP values' expressions
match-expressions:
- key: beta.kubernetes.io/arch
operator: In
values: [amd64, arm]
# The address-pools section lists the IP addresses that MetalLB is
# allowed to allocate, along with settings for how to advertise
# those addresses over BGP once assigned. You can have as many
# address pools as you want.
address-pools:
- # A name for the address pool. Services can request allocation
# from a specific address pool using this name, by listing this
# name under the 'metallb.universe.tf/address-pool' annotation.
name: my-ip-space
# Protocol can be used to select how the announcement is done.
# Supported values are bgp and layer2.
protocol: bgp
# A list of IP address ranges over which MetalLB has
# authority. You can list multiple ranges in a single pool, they
# will all share the same settings. Each range can be either a
# CIDR prefix, or an explicit start-end range of IPs.
addresses:
- 198.51.100.0/24
- 192.168.0.150-192.168.0.200
# (optional) If true, MetalLB will not allocate any address that
# ends in .0 or .255. Some old, buggy consumer devices
# mistakenly block traffic to such addresses under the guise of
# smurf protection. Such devices have become fairly rare, but
# the option is here if you encounter serving issues.
avoid-buggy-ips: true
# (optional, default true) If false, MetalLB will not automatically
# allocate any address in this pool. Addresses can still explicitly
# be requested via loadBalancerIP or the address-pool annotation.
auto-assign: false
# (optional) A list of BGP advertisements to make, when
# protocol=bgp. Each address that gets assigned out of this pool
# will turn into this many advertisements. For most simple
# setups, you'll probably just want one.
#
# The default value for this field is a single advertisement with
# all parameters set to their respective defaults.
bgp-advertisements:
- # (optional) How much you want to aggregate up the IP address
# before advertising. For example, advertising 1.2.3.4 with
# aggregation-length=24 would end up advertising 1.2.3.0/24.
# For the majority of setups, you'll want to keep this at the
# default of 32, which advertises the entire IP address
# unmodified.
aggregation-length: 32
# (optional) The value of the BGP "local preference" attribute
# for this advertisement. Only used with IBGP peers,
# i.e. peers where peer-asn is the same as my-asn.
localpref: 100
# (optional) BGP communities to attach to this
# advertisement. Communities are given in the standard
# two-part form :. You can also use
# alias names (see below).
communities:
- 64512:1
- no-export
# (optional) BGP community aliases. Instead of using hard to
# read BGP community numbers in address pool advertisement
# configurations, you can define alias names here and use those
# elsewhere in the configuration. The "no-export" community used
# above is defined below.
bgp-communities:
# no-export is a well-known BGP community that prevents
# re-advertisement outside of the immediate autonomous system,
# but people don't usually recognize its numerical value. :)
no-export: 65535:65281