使用外部的 etcd 安装 cilium 在较大的运行环境中能够提供更好的性能。
Requirements
- Kubernetes >= 1.9
- Linux kernel >= 4.9
- ETCD >= 3.1.0
- kubernetes 环境中安装了 Helm 3
- Kubernetes in CNI mode
- 在所有 worker node 上挂载 BPF 文件系统
- 推荐：在 kube-controller-manager 上使能 PodCIDR allocation (--allocate-node-cidrs)
安装 helm 3
- 下载解压 helm 安装包
[root@k8s-master-01 ~]# wget https://get.helm.sh/helm-v3.1.2-linux-amd64.tar.gz
[root@k8s-master-01 ~]# tar -zxvf helm-v3.1.2-linux-amd64.tar.gz
[root@k8s-master-01 ~]# mv linux-amd64/helm /usr/local/bin/
- verify
[root@k8s-master-01 ~]# helm help
The Kubernetes package manager
Common actions for Helm:
- helm search: search for charts
- helm pull: download a chart to your local directory to view
- helm install: upload the chart to Kubernetes
- helm list: list releases of charts
Environment variables:
+------------------+-----------------------------------------------------------------------------+
| Name | Description |
+------------------+-----------------------------------------------------------------------------+
| $XDG_CACHE_HOME | set an alternative location for storing cached files. |
| $XDG_CONFIG_HOME | set an alternative location for storing Helm configuration. |
| $XDG_DATA_HOME | set an alternative location for storing Helm data. |
| $HELM_DRIVER | set the backend storage driver. Values are: configmap, secret, memory |
| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins. |
| $KUBECONFIG | set an alternative Kubernetes configuration file (default "~/.kube/config") |
+------------------+-----------------------------------------------------------------------------+
Helm stores configuration based on the XDG base directory specification, so
- cached files are stored in $XDG_CACHE_HOME/helm
- configuration is stored in $XDG_CONFIG_HOME/helm
- data is stored in $XDG_DATA_HOME/helm
Use "helm [command] --help" for more information about a command.
挂载 BPF 文件系统
- 在所有 k8s node 中挂载 bpf 文件系统
[root@k8s-master-01 ~]# mount bpffs /sys/fs/bpf -t bpf
- verify
[root@k8s-master-01 ~]# mount |grep bpf
bpffs on /sys/fs/bpf type bpf (rw,relatime)
# persistence configuration, don’t worry that ‘bpffs’ displaying as red, seems bpf was new commer, fastab desen’t update that feature.
[root@k8s-master-01 ~]# echo "bpffs /sys/fs/bpf bpf defaults 0 0" >> /etc/fstab
kubernetes 配置
在所有的 kubernetes worker node 中的 kubelet 配置使用 CNI 模式
kubelet.config 中添加--network-plugin=cni在 kube-controller-manager 中使能 PodCIDR
在 controller-manager.config 中添加--allocate-node-cidrs=true
Cilium 配置
当使用外部 etcd 作为 cilium 的 k-v 存储，etcd 的 IP 地址需要在 cilium 的 configmap 中配置。
- 使用 helm 安装 cilium
添加 helm cilium repo
[root@k8s-master-01 ~]# helm repo add cilium https://helm.cilium.io/
- 创建 etcd ssl 证书
[root@k8s-master-01 ~]# kubectl create secret generic -n kube-system cilium-etcd-secrets \
--from-file=etcd-client-ca.crt=/etc/etcd/ssl/ca.crt \
--from-file=etcd-client.key=/etc/etcd/ssl/etcd.key \
--from-file=etcd-client.crt=/etc/etcd/ssl/etcd.crt
- 安装 cilium
指定 cilium 版本为 v1.7.1, 开启 SSL 验证，开启 prometheus 监控，添加 etcd cluster 的 menber endpoints
[root@k8s-master-01 ~]# helm install cilium cilium/cilium \
--version 1.7.1\
--set global.etcd.enabled=true\
--set global.etcd.ssl=true\
--set global.prometheus.enabled=true\
--set global.etcd.endpoints[0]=https://172.19.50.7:2379\
--set global.etcd.endpoints[1]=https://172.19.60.32:2379\
--set global.etcd.endpoints[2]=https://172.19.100.16:2379\
--namespace kube-system
NAME: cilium
LAST DEPLOYED: Mon Mar 16 16:44:33 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
- 验证 cilium pod 都安装成功
[root@k8s-master-01 ~]# kubectl --namespace kube-system get ds cilium
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium 4 4 4 4 4 13h
[root@k8s-master-01 ~]# kubectl -n kube-system get deployments cilium-operator
NAME READY UP-TO-DATE AVAILABLE AGE
cilium-operator 1/1 1 1 13h
- 安装 cilium 连接测试用例
此用例将会部署一系列的 deployment，它们会使用多种路径来相互访问，连接路径包括带或者不带服务负载均衡和各种网络策略的组合
部署的 podName 表示连接方式，readiness/liveness 探针则可指示连接是否成功
[root@k8s-master-01 ~]# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/connectivity-check/connectivity-check.yaml -n app-service
[root@k8s-master-01 ~]# kubectl get pods -o wide -n app-service
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
echo-a-58dd59998d-n9g9p 1/1 Running 0 9m13s 10.244.1.50 k8s-master-02
echo-b-669ccc7765-lzqn7 1/1 Running 0 9m13s 10.244.2.50 k8s-master-03
host-to-b-multi-node-clusterip-6fb94d9df6-rbjwz 1/1 Running 3 9m13s 192.168.66.226 k8s-master-02
host-to-b-multi-node-headless-7c4ff79cd-hm6sr 1/1 Running 3 9m13s 192.168.66.226 k8s-master-02
pod-to-a-5c8dcf69f7-gldq9 1/1 Running 3 9m13s 10.244.2.30 k8s-master-03
pod-to-a-allowed-cnp-75684d58cc-tf9nn 1/1 Running 1 9m13s 10.244.2.239 k8s-master-03
pod-to-a-external-1111-669ccfb85f-7r4j8 1/1 Running 0 9m13s 10.244.2.251 k8s-master-03
pod-to-a-l3-denied-cnp-7b8bfcb66c-wd4nj 1/1 Running 0 9m13s 10.244.2.134 k8s-master-03
pod-to-b-intra-node-74997967f8-ml5ps 1/1 Running 3 9m13s 10.244.2.95 k8s-master-03
pod-to-b-multi-node-clusterip-587678cbc4-4qcb2 1/1 Running 3 9m13s 10.244.1.28 k8s-master-02
pod-to-b-multi-node-headless-574d9f5894-tmfwn 1/1 Running 3 9m13s 10.244.1.138 k8s-master-02
pod-to-external-fqdn-allow-google-cnp-6dd57bc859-l49z2 1/1 Running 0 9m12s 10.244.2.62 k8s-master-03
- 安装 hubble https://github.com/cilium/hubble
hubble 是一个用于云原生工作负载的完全分布式网络和安全可视化平台。它建立在 Cilium 和 eBPF 的基础上，以完全透明的方式深入了解服务以及网络基础结构的通信和行为。
[root@k8s-master-01 ~]# git clone https://github.com/cilium/hubble.git
[root@k8s-master-01 ~]# cd hubble/install/kubernetes
[root@k8s-master-01 ~]# helm install hubble ./hubble \
--namespace kube-system \
--set metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
--set ui.enabled=true
hubble 对前面安装的测试用例监控信息
image.png