千云物流 - 使用k8s负载均衡openelb
openelb的介绍
具体根据官方文档进行安装官方文档,这里作为测试环境的安装使用.
OpenELB 是一个开源的云原生负载均衡器实现，可以在基于裸金属服务器、边缘以及虚拟化的 Kubernetes 环境中使用 LoadBalancer 类型的 Service 对外暴露服务。OpenELB 项目最初由 KubeSphere 社区 发起，目前已作为 CNCF 沙箱项目 加入 CNCF 基金会，由 OpenELB 开源社区维护与支持。
与MetalLB类似，OpenELB也拥有两种主要工作模式：Layer2模式和BGP模式。OpenELB的BGP模式目前暂不支持IPv6。
-
layer2 Mode
-
BGP Mode
准备k8s的环境
千云物流测试环境部署使用openelb部署.
| 所需要的软件&版本 | 对应依赖软件版本 |
|---|---|
| linux [CentOS] | 7.9.2009 |
| kubernetes | v1.22.12 |
| docker [20.10.8] | 20.10.8 |
| openelb | kubesphere/openelb:v0.5.1 |
准备Layer2 Mode配置
- 配置ARP参数
部署Layer2模式需要把k8s集群中的ipvs配置打开strictARP，
strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
# 查看kube-proxy中的strictARP配置
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
#strictARP: false
# 手动修改strictARP配置为true
$ kubectl edit configmap -n kube-system kube-proxy
configmap/kube-proxy edited
# 使用命令直接修改并对比不同
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl diff -f - -n kube-system
# 确认无误后使用命令直接修改并生效
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl apply -f - -n kube-system
# 重启kube-proxy确保配置生效
$ kubectl rollout restart ds kube-proxy -n kube-system
# 确认配置生效
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
strictARP: true
开启之后k8s集群中的kube-proxy会停止响应kube-ipvs0网卡之外的其他网卡的arp请求，而由MetalLB接手处理。
strict ARP开启之后相当于把将arp_ignore设置为1;并将arp_announce设置为2启用严格的ARP，这个原理和LVS中的DR模式对RS的配置一样，可以参考之前的文章中的解释。
网卡配置
#多个网卡，需要指定master节点IP,一个网卡不需要
# kubectl annotate nodes k8s-master01 layer2.openelb.kubesphere.io/v1alpha1="masterip"
创建EIP
接下来我们需要配置loadbalancerIP所在的网段资源，这里我们创建一个Eip对象来进行定义，后面对IP段的管理也是在这里进行。
- 部署eip
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
# Eip 对象的名称。
name: layer2-eip
spec:
# Eip 对象的地址池
address: 10.0.0.122-10.0.0.123
# openELB的运行模式，默认为bgp
protocol: layer2
# OpenELB 在其上侦听 ARP/NDP 请求的网卡。该字段仅在protocol设置为时有效layer2。
interface: ens160
# 指定是否禁用 Eip 对象
# false表示可以继续分配
# true表示不再继续分配
disable: false
status:
# 指定 Eip 对象中的IP地址是否已用完。
occupied: false
# 指定 Eip 对象中有多少个 IP 地址已分配给服务。
# 直接留空，系统会自动生成
usage:
# Eip 对象中的 IP 地址总数。
poolSize: 2
# 指定使用的 IP 地址和使用 IP 地址的服务。服务以Namespace/Service name格式显示（例如，default/test-svc）。
# 直接留空，系统会自动生成
used:
# Eip 对象中的第一个 IP 地址。
firstIP: 10.0.0.122
# Eip 对象中的最后一个 IP 地址。
lastIP: 10.0.0.123
ready: true
# 指定IP协议栈是否为 IPv4。目前，OpenELB 仅支持 IPv4，其值只能是true.
v4: true
- 检查eip状态
kubectl apply -f openelb/openelb-eip.yaml
#部署完成后检查eip的状态
kubectl get eip
部署openelb
这里我们还是使用yaml进行部署，官方把所有部署的资源整合到了一个文件中，我们还是老规矩先下载到本地再进行部署
apiVersion: v1
kind: Namespace
metadata:
name: openelb-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.4.0
creationTimestamp: null
name: bgpconfs.network.kubesphere.io
spec:
group: network.kubesphere.io
names:
kind: BgpConf
listKind: BgpConfList
plural: bgpconfs
singular: bgpconf
scope: Cluster
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: BgpConf is the Schema for the bgpconfs API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: struct for container bgp:config. Configuration parameters
relating to the global BGP router.
properties:
as:
description: original -> bgp:as bgp:as's original type is inet:as-number.
Local autonomous system number of the router. Uses the 32-bit as-number
type from the model in RFC 6991.
format: int32
type: integer
port:
description: original -> gobgp:port
format: int32
maximum: 65535
minimum: 1
type: integer
routerID:
description: original -> bgp:router-id bgp:router-id's original type
is inet:ipv4-address. Router id of the router, expressed as an 32-bit
value, IPv4 address.
pattern: ^([0-9]{
1,3}\.){
3}[0-9]{
1,3}$
type: string
required:
- as
- port
- routerID
type: object
status:
description: BgpConfStatus defines the observed state of BgpConf
type: object
type: object
served: true
storage: false
- name: v1alpha2
schema:
openAPIV3Schema:
description: BgpConf is the Schema for the bgpconfs API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: Configuration parameters relating to the global BGP router.
properties:
as:
format: int32
type: integer
asPerRack:
additionalProperties:
format: int32
type: integer
type: object
families:
items:
format: int32
type: integer
type: array
gracefulRestart:
properties:
deferralTime:
format: int32
type: integer
enabled:
type: boolean
helperOnly:
type: boolean
localRestarting:
type: boolean
longlivedEnabled:
type: boolean
mode:
type: string
notificationEnabled:
type: boolean
peerRestartTime:
format: int32
type: integer
peerRestarting:
type: boolean
restartTime:
format: int32
type: integer
staleRoutesTime:
format: int32
type: integer
type: object
listenAddresses:
items:
type: string
type: array
listenPort:
format: int32
type: integer
policy:
type: string
routerId:
type: string
useMultiplePaths:
type: boolean
type: object
status:
description: BgpConfStatus defines the observed state of BgpConf
properties:
nodesConfStatus:
additionalProperties:
properties:
as:
format: int32
type: integer
routerId:
type: string
type: object
type: object
type: object
type: object
served: true
storage: true
subresources:
status: {
}
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.4.0
creationTimestamp: null
name: bgppeers.network.kubesphere.io
spec:
group: network.kubesphere.io
names:
kind: BgpPeer
listKind: BgpPeerList
plural: bgppeers
singular: bgppeer
scope: Cluster
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: BgpPeer is the Schema for the bgppeers API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
addPaths:
description: original -> bgp:add-paths Parameters relating to the
advertisement and receipt of multiple paths for a single NLRI (add-paths).
properties:
sendMax:
description: original -> bgp:send-max The maximum number of paths
to advertise to neighbors for a single NLRI.
type: integer
type: object
config:
description: original -> bgp:neighbor-address original -> bgp: