How to configure and resource control Kubelet with configuration file for v1.15.0

Since Kubernetes v1.10, K8s considered to configure kubelet with configuration file, this has become a recommended approach because its simplifies node deployment and configuration management. in this article, I will describe how to use a configuration file to configure kubelet to do resource control.

how to pass a configuration file to kubelet

we need to use --config flag to set a path to a configure file to kubelet in kubelet service conf file, when system start kubelet, will pass this configuration to kubelet

when we initial Kubernetes cluster, this /var/lib/kubelet/config.yaml file will be generated by kubeadm tool with default value if we do not set any parameter for KubeletConfiguration in cluster configuration file. In below example, we give some parameters to enable resource management for K8s, all these parameters will write to config.yaml file

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
systemReserved:
  cpu: 1
  memory: 1Gi
  ephemeral-storage: 10Gi
systemReservedCgroup: /system.slice
kubeReserved:
  cpu: 1
  memory: 2Gi
  ephemeral-storage: 10Gi
kubeReservedCgroup: /system.slice/kubelet.service

enforceNodeAllocatable:
- pods
- kube-reserved
- system-reserved

evictionHard:
  imagefs.available: 15%
  memory.available: 500Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%

Since we use linux CGroup(control groups) in this example to manage resource for Kubernetes, so we need to create some new groups for Kubelet under cgroup before we initial K8s cluster, you can put all these steps into kubelet service configuration file

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
(redhat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf)
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
add following:
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/pids/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/devices/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/memory/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/blkio/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/system.slice/docker.service

after we done above steps, then we can use below command to initize cluster:

kubeadm init --config=/etc/kubernetes/k8s-cluster-bi.yaml --upload-certs

the config.yaml will be generated:

address: 0.0.0.0

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
cgroupsPerQOS: true
clusterDNS:

• 172.17.0.10
  clusterDomain: cluster.local
  configMapAndSecretChangeDetectionStrategy: Watch
  containerLogMaxFiles: 5
  containerLogMaxSize: 10Mi
  contentType: application/vnd.kubernetes.protobuf
  cpuCFSQuota: true
  cpuCFSQuotaPeriod: 100ms
  cpuManagerPolicy: none
  cpuManagerReconcilePeriod: 10s
  enableControllerAttachDetach: true
  enableDebuggingHandlers: true
  enforceNodeAllocatable:
• pods
• kube-reserved
• system-reserved
  eventBurst: 10
  eventRecordQPS: 5
  evictionHard:
  imagefs.available: 15%
  memory.available: 500Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
  evictionPressureTransitionPeriod: 5m0s
  failSwapOn: true
  fileCheckFrequency: 20s
  hairpinMode: promiscuous-bridge
  healthzBindAddress: 127.0.0.1
  healthzPort: 10248
  httpCheckFrequency: 20s
  imageGCHighThresholdPercent: 85
  imageGCLowThresholdPercent: 80
  imageMinimumGCAge: 2m0s
  iptablesDropBit: 15
  iptablesMasqueradeBit: 14
  kind: KubeletConfiguration
  kubeAPIBurst: 10
  kubeAPIQPS: 5
  kubeReserved:
  cpu: 1000m
  ephemeral-storage: 10Gi
  memory: 2Gi
  kubeReservedCgroup: /system.slice/kubelet.service
  makeIPTablesUtilChains: true
  maxOpenFiles: 1000000
  maxPods: 110
  nodeLeaseDurationSeconds: 40
  nodeStatusReportFrequency: 1m0s
  nodeStatusUpdateFrequency: 10s
  oomScoreAdj: -999
  podPidsLimit: -1
  port: 10250
  registryBurst: 10
  registryPullQPS: 5
  resolvConf: /etc/resolv.conf
  rotateCertificates: true
  runtimeRequestTimeout: 2m0s
  serializeImagePulls: true
  staticPodPath: /etc/kubernetes/manifests
  streamingConnectionIdleTimeout: 4h0m0s
  syncFrequency: 1m0s
  systemReserved:
  cpu: 1000m
  ephemeral-storage: 10Gi
  memory: 1Gi
  systemReservedCgroup: /system.slice
  volumeStatsAggPeriod: 1m0s

then you can use command to check other nodes resource allocatable number

kubectl describe node ppydalbik0101

Capacity:
 cpu:                16
 ephemeral-storage:  102821812Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             65943252Ki
 pods:               110
Allocatable:
 cpu:                14
 ephemeral-storage:  73285745303
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             62285524Ki
 pods:               110