kata container与kubernetes集成的几种方式

概览

• kata container简介
• cri-containerd plugin + containerd方式
• cri-o方式

kata container简介

kata containers是由OpenStack基金会管理，但独立于OpenStack项目之外的容器项目。它是一个可 以使用容器镜像以超轻量级虚机的形式创建容器的运行时工具。 kata containers整合了Intel的 Clear Containers 和 Hyper.sh 的 runV，能够支持不同平台的硬件 （x86-64，arm等），并符合OCI(Open Container Initiative)规范，同时还可以兼容k8s的 CRI（Container Runtime Interface）接口规范。目前项目包含几个配套组件，即Runtime，Agent， Proxy，Shim，Kernel等。目前Kata Containers的运行时还没有整合，即Clear containers 和 runV 还在独立的组织内。

• runtime：符合OCI规范的容器运行时命令工具，主要用来创建轻量级虚机，并通过agent控制虚 拟机内容器的生命周期。目前kata containers还没有一个统一的运行时工具，用户可以选择 clear containers和runV中的其中之一。

• agent：运行在虚机中的一个运行时代理组件，主要用来执行runtime传给他的指令，在虚机内 管理容器的生命周期。

• shim：以对接docker为例，这里的shim相当于是containerd-shim的一个适配，用来处理容器 进程的stdio和signals。shim可以将containerd-shim发来的数据流（如stdin）传给proxy，然 后转交给agent，也可以将agent经由proxy发过来的数据流（stdout/stderr）传给containerdshim，同时也可以传输signal。

• kernel：kernel其实比较好理解，就是提供一个轻量化虚机的linux内核，根据不同的需要，提供 几个内核选择，最小的内核仅有4M多

kata和传统容器对比

kata and docker

kata in kubernetes

OCI部分即为kata,也可以是符合oci标准的运行时容器。

以下环境安装默认都是overlayfs ,centos 建议版本7.4，7.5版本默认f_type=0,不支持D_TYPE参数，会导致containerd和crio启动失败

kata + cri-containerd plugin + containerd

环境版本

kubeadm      v1.10.3
kubenetes    v1.10.3
crictl       v1.0.0-beta.0
containerd   v1.1.0
kata-runtime 1.0.0

kubeadm安装

https://www.jianshu.com/p/3ec8945a864f

此处只需安装一个主节点即可，kata容器所在虚拟机均为从节点

镜像版本

k8s.gcr.io/kube-proxy-amd64               v1.10.3 
k8s.gcr.io/kube-apiserver-amd64           v1.10.3 
k8s.gcr.io/kube-scheduler-amd64           v1.10.3 
k8s.gcr.io/kube-controller-manager-amd64  v1.10.3 
k8s.gcr.io/etcd-amd64                     3.1.12  
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.8  
k8s.gcr.io/k8s-dns-sidecar-amd64          1.14.8  
k8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.8  
k8s.gcr.io/pause-amd64                    3.1     
quay.io/coreos/flannel                    v0.10.0-amd64

kata 安装

sudo yum -y install yum-utils

source /etc/os-release

sudo -E VERSION_ID=$VERSION_ID yum-config-manager --add-repo \
"http://download.opensuse.org/repositories/home:/katacontainers:/release/CentOS_${VERSION_ID}/home:katacontainers:release.repo"

sudo -E yum -y install kata-runtime kata-proxy kata-shim

安装完成后测试

[root@localhost tmp]# kata-runtime -v
kata-runtime  : 1.0.0
   commit   : a3d11b73ac4f1d5c5ae6451d83fc4b1fca385098
   OCI specs: 1.0.1

cri-containerd 安装

wget https://github.com/containerd/containerd/releases/download/v1.1.0/containerd-1.1.0.linux-amd64.tar.gz

sudo tar -C / -xzf  containerd-1.1.0.linux-amd64.tar.gz

sudo systemctl start containerd

sudo systemctl status containerd

containerd config default #生成默认配置，位置/etc/containerd/config.toml

crictl 安装

确保有go环境

go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

export PATH=$PATH:$GOPATH/bin

配置修改

crictl配置

[root@localhost bin]# cat /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

containerd配置

[root@localhost bin]# cat /etc/containerd/config.toml 
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = ""
    stream_server_port = "10010"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause-amd64:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      ## default_runtime为默认runtime
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      ## untrusted_workload_runtime当pod参数带有
      ## io.kubernetes.cri.untrusted-workload: "true"时生效
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://9c56nwew.mirror.aliyuncs.com", "https://registry-1.docker.io"]
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

各参数配置含义：https://github.com/containerd/cri/blob/release/1.0/docs/config.md

kubelet配置

[root@localhost bin]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意该行配置，修改kubelet启动使用containerd
Environment="KUBELET_EXTRA_ARGS=--runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

关于镜像下载的问题

由于默认镜像都是 k8s.gcr.io上的无法访问，所以最好有docker环境给镜像打tag，然后使用

ctr cri load xxx.tar 

cri-o + kata

kata安装

同上

cri-o安装

yum install -y \
  btrfs-progs-devel \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  golang-github-cpuguy83-go-md2man \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  ostree-devel \
  pkgconfig \
  runc \
  skopeo-containers

git clone https://github.com/kubernetes-incubator/cri-o # or your fork
cd cri-o

make install.tools
make
sudo make install

##第一次安装时生成初始化配置文件
make install.config

make install.tools时可能有些package下载不下来，建议直接在git clone到go path下

配置

修改crio配置

vim /etc/crio/crio.conf 

主要修改crio.runtime的配置

[crio.runtime]
runtime = "/usr/bin/runc"
runtime_untrusted_workload = "/usr/bin/kata-runtime"
default_workload_trust = "trusted"

kubelet配置

[root@localhost crio]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意该行配置，修改kubelet启动使用crio
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

关于镜像

cri-o缺少像containerd那样的镜像管理工具，个人采用的方式时修改主节点kube-proxy和kube-flannel-ds这两个DaemonSet的镜像为本地个人上传的阿里云的镜像，才能启动成功。

[root@kata-node1 tmp]# kubectl get ds -n kube-system
NAME              DESIRED   CURRENT   READY UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-flannel-ds   3            3        3       3             3    beta.kubernetes.io/arch=amd64   4h
kube-proxy        3            3        3       3             3                           5h              

kubectl edit kube-flannel-ds -n kube-system

加入集群

环境安装完成后各节点使用

kubeadm join 172.20.95.208:6443 --token ss60t6.fb1hymd6qxf5tbx7 --discovery-token-ca-cert-hash sha256:ea3a8f578071f73e7a5aab2e783f9d1cce22d4cf70faad24b8eb9090a4ef6b3b --ignore-preflight-errors all

具体命令使用主节点初始化完成后的kubeadm join命令 ，只需加上--ignore-preflight-errors all即可

测试

[root@kata-node1 tmp]# cat nginx-untrusted-1.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted-1
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx-1
    image: nginx


kubecrl create -f  nginx-untrusted-1.yaml