参考资料
本次测试环境如下
10.100.1.0/24
,自定义联网后pod网段为10.120.1.0/24
自定义联网的需求
IPv4
地址数量有限注意事项
配置自定义联网比较简单,开启环境变量
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone
在两个可用区的创建crd
$ cat cn-north-1a.yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: cn-north-1a
spec:
securityGroups:
- sg-088b947d8817fdc2f
subnet: subnet-04cb3b7313ca37a80
$ cat cn-north-1b.yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: cn-north-1b
spec:
securityGroups:
- sg-088b947d8817fdc2f
subnet: subnet-0cd19152401a80eb5
kubectl apply -f cn-north-1a.yaml
kubectl apply -f cn-north-1b.yaml
aws-node启动日志
将net/ipv4/conf/eth0/rp_filter
设置为2,对应源码如下
关于nodeport支持的参数,https://github.com/aws/amazon-vpc-cni-k8s#aws_vpc_cni_node_port_support
if n.nodePortSupportEnabled {
// If node port support is enabled, configure the kernel's reverse path filter check on eth0 for "loose" filtering. This is required because NodePorts are exposed on eth0 The kernel's RPF check happens after incoming packets to NodePorts are DNATted to the pod IP. For pods assigned to secondary ENIs, the routing table includes source-based routing. When the kernel does the RPF check, it looks up the route using the pod IP as the source. Thus, it finds the source-based route that leaves via the secondary ENI. In "strict" mode, the RPF check fails because the return path uses a different interface to the incoming packet. In "loose" mode, the check passes because some route was found.
const eth0RPFilter = "/proc/sys/net/ipv4/conf/eth0/rp_filter"
const rpFilterLoose = "2"
err := n.setProcSys(eth0RPFilter, rpFilterLoose)
if err != nil {
return errors.Wrapf(err, "failed to configure eth0 RPF check")
}
}
将net/ipv4/tcp_early_demux
设置为1
aws-node Installed /host/opt/cni/bin/aws-cni
aws-node Installed /host/opt/cni/bin/egress-v4-cni
aws-node time="2023-03-22T07:59:16Z" level=info msg="Starting IPAM daemon... "
aws-node time="2023-03-22T07:59:16Z" level=info msg="Checking for IPAM connectivity... "
aws-node time="2023-03-22T07:59:17Z" level=info msg="Copying config file... "
aws-node time="2023-03-22T07:59:17Z" level=info msg="Successfully copied CNI plugin binary and config file."
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Copying CNI plugin binaries ..."
aws-vpc-cni-init Installed /host/opt/cni/bin/loopback
aws-vpc-cni-init Installed /host/opt/cni/bin/portmap
aws-vpc-cni-init Installed /host/opt/cni/bin/bandwidth
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Copied all CNI plugin binaries to /host/opt/cni/bin"
aws-vpc-cni-init Installed /host/opt/cni/bin/host-local
aws-vpc-cni-init Installed /host/opt/cni/bin/aws-cni-support.sh
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Found primaryMAC 02:67:d2:91:12:ae"
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Found primaryIF eth0"
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Updated net/ipv4/conf/eth0/rp_filter to 2\n"
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="Updated net/ipv4/tcp_early_demux to 1\n"
aws-vpc-cni-init time="2023-03-22T07:59:14Z" level=info msg="CNI init container done"
在几点的ipamd日志中可以看到aws-node环境便的相关配置,例如Custom networking enabled true
表示开启了自定义联网3
{"level":"info","ts":"2023-03-22T07:59:16.772Z","caller":"aws-k8s-agent/main.go:28","msg":"Starting L-IPAMD ..."}
{"level":"info","ts":"2023-03-22T07:59:16.772Z","caller":"aws-k8s-agent/main.go:39","msg":"Testing communication with server"}
{"level":"info","ts":"2023-03-22T07:59:16.783Z","caller":"wait/wait.go:222","msg":"Successful communication with the Cluster! Cluster Version is: v1.23+. git version: v1.23.16-eks-48e63af. git tree state: clean. commit: e6332a8a3feb9e0fe3db851878f88cb73d49dd7a. platform: linux/amd64"}
{"level":"warn","ts":"2023-03-22T07:59:16.799Z","caller":"awssession/session.go:64","msg":"HTTP_TIMEOUT env is not set or set to less than 10 seconds, defaulting to httpTimeout to 10sec"}
{"level":"debug","ts":"2023-03-22T07:59:16.801Z","caller":"ipamd/ipamd.go:392","msg":"Discovered region: cn-north-1"}
{"level":"info","ts":"2023-03-22T07:59:16.801Z","caller":"ipamd/ipamd.go:392","msg":"Custom networking enabled true"}
{"level":"debug","ts":"2023-03-22T07:59:16.802Z","caller":"awsutils/awsutils.go:431","msg":"Found availability zone: cn-north-1a "}
{"level":"debug","ts":"2023-03-22T07:59:16.803Z","caller":"awsutils/awsutils.go:431","msg":"Discovered the instance primary IPv4 address: 10.100.0.81"}
{"level":"debug","ts":"2023-03-22T07:59:16.803Z","caller":"awsutils/awsutils.go:431","msg":"Found instance-id: i-093668d8490163140 "}
{"level":"debug","ts":"2023-03-22T07:59:16.804Z","caller":"awsutils/awsutils.go:431","msg":"Found instance-type: m5.2xlarge "}
{"level":"debug","ts":"2023-03-22T07:59:16.804Z","caller":"awsutils/awsutils.go:431","msg":"Found primary interface's MAC address: 02:67:d2:91:12:ae"}
{"level":"debug","ts":"2023-03-22T07:59:16.805Z","caller":"awsutils/awsutils.go:431","msg":"eni-036525abd58c37374 is the primary ENI of this instance"}
{"level":"debug","ts":"2023-03-22T07:59:16.805Z","caller":"awsutils/awsutils.go:431","msg":"Found subnet-id: subnet-0fa7c3d3729a3009b "}
{"level":"debug","ts":"2023-03-22T07:59:16.805Z","caller":"ipamd/ipamd.go:401","msg":"Using WARM_ENI_TARGET 1"}
{"level":"debug","ts":"2023-03-22T07:59:16.805Z","caller":"ipamd/ipamd.go:404","msg":"Using WARM_PREFIX_TARGET 1"}
{"level":"info","ts":"2023-03-22T07:59:16.806Z","caller":"ipamd/ipamd.go:422","msg":"Prefix Delegation enabled false"}
{"level":"debug","ts":"2023-03-22T07:59:16.806Z","caller":"ipamd/ipamd.go:427","msg":"Start node init"}
{"level":"debug","ts":"2023-03-22T07:59:16.806Z","caller":"ipamd/ipamd.go:459","msg":"Max ip per ENI 14 and max prefixes per ENI 0"}
{"level":"info","ts":"2023-03-22T07:59:16.806Z","caller":"awsutils/awsutils.go:1684","msg":"Will attempt to clean up AWS CNI leaked ENIs after waiting 2m1s."}
创建完毕后查看pod的ip
$ k get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx-dep-6cb4756ddb-7sjgn 1/1 Running 0 119s 10.120.0.248 ip-10-100-0-215.cn-north-1.compute.internal
default nginx-dep-6cb4756ddb-8t8wz 1/1 Running 0 119s 10.120.0.196 ip-10-100-0-215.cn-north-1.compute.internal
default nginx-dep-6cb4756ddb-95rm5 1/1 Running 0 119s 10.120.0.60 ip-10-100-0-215.cn-north-1.compute.internal
cni插件没有获取托管权限AmazonEKS_CNI_Policy
查看aws-node日志,出现如下报错
Unauthorized operation: failed to call ec2:DescribeNetworkInterfaces due to missing permissions. Please refer https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/iam-policy.md to attach relevant policy to IAM role
查看节点ipamd日志,关键词为
Unauthorized operation
ec2:DescribeNetworkInterfaces
操作{"level":"debug","ts":"2023-03-22T07:50:48.809Z","caller":"awsutils/awsutils.go:1947","msg":"Sent pod event: eventType: Warning, reason: MissingIAMPermissions, message: Unauthorized operation: failed to call ec2:DescribeNetworkInterfaces due to missing permissions. Please refer https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/iam-policy.md to attach relevant policy to IAM role"}
{"level":"error","ts":"2023-03-22T07:50:48.809Z","caller":"ipamd/ipamd.go:475","msg":"Failed to call ec2:DescribeNetworkInterfaces for [eni-004de29dbc204f264 eni-0562c3edc5c3ebc75]: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 79aa8fa8-c7ed-4023-8f29-2b10d4abea7c"}
{"level":"error","ts":"2023-03-22T07:50:48.920Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd init: failed to retrieve attached ENIs info: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 79aa8fa8-c7ed-4023-8f29-2b10d4abea7c"}
配置错误的安全组sg-088b947d8817fdc2f
由于安全组为可选配置,因此不配置安全组反而不会受到影响。
Failed to find IPAMing CRI-migrated version
这里误导性很强,实际并不意味着需要升级cni插件,任何自定义联网的错误都可能会出现这个错误
{"level":"debug","ts":"2023-03-22T06:54:46.947Z","caller":"ipamd/rpc_handler.go:222","msg":"UnassignPodIPAddress: IP address pool staandbox aws-cni/c3631138652f4b26d6b52635e103880008f049a1a5e8dc016159d5659f9eb8bb/eth0"}
{"level":"debug","ts":"2023-03-22T06:54:46.947Z","caller":"ipamd/rpc_handler.go:222","msg":"UnassignPodIPAddress: Failed to find IPAMing CRI-migrated version"}
{"level":"warn","ts":"2023-03-22T06:54:46.947Z","caller":"ipamd/rpc_handler.go:222","msg":"UnassignPodIPAddress: Failed to find sandb1138652f4b26d6b52635e103880008f049a1a5e8dc016159d5659f9eb8bb/unknown"}
{"level":"info","ts":"2023-03-22T06:54:46.947Z","caller":"rpc/rpc.pb.go:731","msg":"Send DelNetworkReply: IPv4Addr , DeviceNumber: 0,od"}
如果在设置aws-node插件时,eniconfig的crd文件使用了可用区名称,但是没有指定如下环境变量,则会出现找不到eniconfig的错误
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone
此时pod处于ContainerCreating
状态
Warning FailedCreatePodSandBox 0s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "76d1xe56f8" network for pod "nginx-dep-6cb4756ddb-rr5hs": networkPlugin cni failed to set up pod "nginx-dep-6cb4756ddb-rr5hs_default" network: add cmd: failed to assign an IP address to container
查看节点的ipamd日志,出现如下报错
default
的eniconfig文件,但是始终没有找到{"level":"error","ts":"2023-03-22T09:19:07.812Z","caller":"ipamd/ipamd.go:871","msg":"error while retrieving eniconfig: ENIConfig.crd.k8s.amazonaws.com \"default\" not found"}
{"level":"error","ts":"2023-03-22T09:19:07.812Z","caller":"ipamd/ipamd.go:845","msg":"Failed to get pod ENI config"}
这个错误的逻辑在于
如果使用可用区作为eniconfig的名称,例如cn-north-1a
,则需要使用以上注解开启支持。ipamd会使用节点的可用区作为eniconfig的名称来查找配置文件
如果使用可用区之外的名称作为eniconfig文件的名称,则需要使用以下命令单独注解节点。
kubectl annotate node ip-192-168-0-126.us-west-2.compute.internal k8s.amazonaws.com/eniConfig=EniConfigName1
kubectl annotate node ip-192-168-0-92.us-west-2.compute.internal k8s.amazonaws.com/eniConfig=EniConfigName2
此时aws-node启动成功,但是pod处于ContainerCreating
状态
这个错误同样足够明显,即ipamd没有权限修改内核参数
https://aws.github.io/aws-eks-best-practices/networking/vpc-cni/
Amazon VPC CNI has two components CNI binary and ipamd (aws-node) Daemonset. The CNI runs as a binary on a node and has access to node root file system, also has privileged access as it deals with iptables at the node level.
但是实际上cni插件在特权模式启动,并且有权限访问根文件系统
节点的ipamd日志如下
{"level":"debug","ts":"2023-03-22T02:46:55.250Z","caller":"ipamd/ipamd.go:381","msg":"Setting RPF for primary interface: net/ipv4/conf/eth0/rp_filter"}
{"level":"error","ts":"2023-03-22T02:46:55.250Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd init: failed to set up host network: failed to configure eth0 RPF check: open /proc/sys/net/ipv4/conf/eth0/rp_filter: read-only file system"}
查看源码实际上是通过open函数修改配置文件,
func (n *linuxNetwork) setProcSys(key, value string) error {
f, err := n.openFile(key, os.O_WRONLY, 0644)
if err != nil {
return err
}
defer f.Close()
_, err = f.WriteString(value)
if err != nil {
return err
}
return nil
}
目前没有其他可鞥导致该问题的可能,解决方式为重启cni插件
官方文档中提到,如果将额外的cidr块添加到vpc中,并在此cidr中启动i新的节点组,则会出现节点无法加入集群的问题
https://aws.amazon.com/cn/premiumsupport/knowledge-center/eks-multiple-cidr-ranges/
在某些情况下,创建集群后,Amazon EKS 无法与通过添加到 VPC 的其他 CIDR 数据块在子网中启动的节点进行通信。向现有集群添加 CIDR 数据块所导致的更新范围可能需要长达 5 小时才能显示。
查看节点的kubelet日志会发现姐弟啊您无法访问到apiserver,但是实际并不存在联网问题。
这个新创建的cidr段被集群识别的时长可能会持续5小时。