k8s 容器资源可见性配置-lxcfs

问题

使用k8s容器化后，对内存 CPU 资源限制后，在容器内查看资源，显示的和容器所在宿主机信息一致，无法看到限制后的内存情况

目标

实现资源可见性
比如 resources.requests.memory: "1024Mi"
那么在容器内查看内存，执行free -m 后显示的内存也为1024

1.安装依赖

在k8s所有节点执行以下命令，安装依赖包
yum -y install fuse-devel fuse fuse-libs

yum -y install https://copr-be.cloud.fedoraproject.org/results/ganto/lxd/epel-7-x86_64/00486278-lxcfs/lxcfs-2.0.5-3.el7.centos.x86_64.rpm

2.安装lcxfs

安装lxcfs有如下两种方案：

1）.直接在所有k8s节点上安装lxcfs 包，并启动进程

yum -y install https://copr-be.cloud.fedoraproject.org/results/ganto/lxd/epel-7-x86_64/00486278-lxcfs/lxcfs-2.0.5-3.el7.centos.x86_64.rpm
systemctl start  lxcfs
systemctl enable  lxcfs

2）.使用k8s Daemonsets部署lxcfs，让其在每一台节点上运行lxcfs 进程

apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  name: lxcfs
  labels:
    app: lxcfs
spec:
  selector:
    matchLabels:
      app: lxcfs
  template:
    metadata:
      labels:
        app: lxcfs
    spec:
      hostPID: true
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: lxcfs
        image: registry.cn-hangzhou.aliyuncs.com/denverdino/lxcfs:2.0.8-1
        imagePullPolicy: Always
        securityContext:
          privileged: true
        volumeMounts:
        - name: cgroup
          mountPath: /sys/fs/cgroup
        - name: lxcfs
          mountPath: /var/lib/lxcfs
          mountPropagation: Bidirectional
        - name: usr-local
          mountPath: /usr/local
      volumes:
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      - name: usr-local
        hostPath:
          path: /usr/local
      - name: lxcfs
        hostPath:
          path: /var/lib/lxcfs
          type: DirectoryOrCreate

将以上内容保存为lxcfs-dadaemontset.yaml 文件
kubectl apply -f ``lxcfs-dadaemontset.yaml
以上两种方式二选一

3.使用lxcfs

使用lxcfs会有三种方式，分别是直接挂载，PodPreset， Initializer

方案1:直接挂载

在创建pod的时候，将lxcfs相应文件直接挂载到pod即可
相关文件：
/var/lib/lxcfs/proc/cpuinfo
/var/lib/lxcfs/proc/meminfo
/var/lib/lxcfs/proc/diskstats
/var/lib/lxcfs/proc/stat
/var/lib/lxcfs/proc/swaps
/var/lib/lxcfs/proc/uptime

案例：

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    "initializer.kubernetes.io/lxcfs": "true"
  labels:
    app: web
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: httpd:2.4.32
          volumeMounts:
            - name: cpuinfo
              mountPath: /proc/cpuinfo
            - name: meminfo
              mountPath: /proc/meminfo
            - name: diskstats
              mountPath: /proc/diskstats
            - name: stat
              mountPath: /proc/stat
            - name: swaps
              mountPath: /proc/swaps
            - name: uptime
              mountPath: /proc/uptime
          imagePullPolicy: Always
          resources:
            requests:
              memory: "1024Mi"
              cpu: "100m"
            limits:
              memory: "1024Mi"
              cpu: "100m"

      volumes:
      - name: cpuinfo
        hostPath:
          path: /var/lib/lxcfs/proc/cpuinfo
          type: File
      - name: meminfo
        hostPath:
          path: /var/lib/lxcfs/proc/meminfo
          type: File
      - name: diskstats
        hostPath:
          path: /var/lib/lxcfs/proc/diskstats
          type: File
      - name: stat
        hostPath:
          path: /var/lib/lxcfs/proc/stat
          type: File
      - name: swaps
        hostPath:
          path: /var/lib/lxcfs/proc/swaps
          type: File
      - name: uptime
        hostPath:
          path: /var/lib/lxcfs/proc/uptime
          type: File

方案2:使用PodPreset 注入

使用该方案，需要开启kube-apiserver 相关参数
--enable-admission-plugins=PodPreset,XXXX,XXXX
--runtime-config=settings.k8s.io/v1alpha1
PodPreset 会根据
然后创建PodPreset spec.selector 来选择pod并注入
以下案例就是PodPreset 会将该Namespace 下所有包含标签inject-lxcfs: "true" 的pod 进行注入volumeMounts，volumes

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: inject-lxcfs
spec:
  selector:
    matchLabels:
      inject-lxcfs: "true"
  volumeMounts:
    - name: cpuinfo
      mountPath: /proc/cpuinfo
    - name: meminfo
      mountPath: /proc/meminfo
    - name: diskstats
      mountPath: /proc/diskstats
    - name: stat
      mountPath: /proc/stat
    - name: swaps
      mountPath: /proc/swaps
    - name: uptime
      mountPath: /proc/uptime
  volumes:
  - name: cpuinfo
    hostPath:
      path: /var/lib/lxcfs/proc/cpuinfo
      type: File
  - name: meminfo
    hostPath:
      path: /var/lib/lxcfs/proc/meminfo
      type: File
  - name: diskstats
    hostPath:
      path: /var/lib/lxcfs/proc/diskstats
      type: File
  - name: stat
    hostPath:
      path: /var/lib/lxcfs/proc/stat
      type: File
  - name: swaps
    hostPath:
      path: /var/lib/lxcfs/proc/swaps
      type: File
  - name: uptime
    hostPath:
      path: /var/lib/lxcfs/proc/uptime
      type: File

创建Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    "kubernetes.io/inject-lxcfs": "true"
  labels:
    app: lxcfs-test
  name: lxcfs-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: lxcfs-test
  template:
    metadata:
      labels:
        app: lxcfs-test
        inject-lxcfs: "true"
    spec:
      containers:
        - name: lxcfs-test
          image: httpd:2.4.32
          imagePullPolicy: Always
          resources:
            requests:
              memory: "1024Mi"
              cpu: "100m"
            limits:
              memory: "1024Mi"
              cpu: "100m"

接下来查看pod 就会有相应的内容注入
kubectl get pod lxcfs-test-xxxxx-xxx -o yaml

4.验证

根据上面部署的httpd 将内存限制为1024m，cpu限制为0.1
因此效果如下

内存

image.png

shancangchen@localhost:~/hellobike/code/k8s$ kubectl exec -it lxcfs-test-b9d549ddf-vqhwt  bash
root@lxcfs-test-b9d549ddf-vqhwt:/usr/local/apache2# free  -m
             total       used       free     shared    buffers     cached
Mem:          1024         13       1010          3          0          0
-/+ buffers/cache:         13       1010
Swap:            0          0          0
root@lxcfs-test-b9d549ddf-vqhwt:/usr/local/apache2#

CPU