Kubernetes-准入控制

一. 准入控制

Webhook 官方demo

• 默认准时控制器

  • NamespaceLifecycle
  • LimitRanger
  • ServiceAccount
  • TaintNodesByCondition
  • Priority
  • DefaultTolerationSeconds
  • DefaultStorageClass
  • StorageObjectInUseProtection
  • PersistentVolumeClaimResize
  • RuntimeClass
  • CertificateApproval
  • CertificateSigning
  • CertificateSubjectRestriction
  • DefaultIngressClass
  • MutatingAdmissionWebhook
  • ValidatingAdmissionWebhook
  • ResourceQuota

• Mutating admission webhooks 用于在资源存储之前通过 mutating webhooks 进行修改

• Validating admission webhooks 用于在资源存储之前通过 validating webhooks 自定义策略验证资源

• 当 API 请求进入时，mutating 和 validating 控制器使用配置中的外部 webhooks 列表并发调用，规则如下：

  • 如果所有的 webhooks 批准请求，准入控制链继续流转, 第一阶段，运行变更准入控制器, Mutating Admission, 它可以修改被它接受的对象，这就引出了它的另一个作用，将相关资源作为请求处理的一部分进行变更, 第二阶段，运行验证准入控制器 Validating Admission它只能进行验证，不能进行任何资源数据的修改操作
  • 如果有任意一个 webhooks 阻止请求，那么准入控制请求终止，并返回第一个 webhook 阻止的原因。其中，多个 webhooks 阻止也只会返回第一个 webhook 阻止的原因
  • 如果在调用 webhook 过程中发生错误，那么请求会被终止或者忽略 webhook

• Admission webhooks 可以使用如下几个场景：

  • 通过 mutating webhook 注入 side-car 到 Pod（istio 的 side-car 就是采用这种方式注入的）
  • 限制项目使用某个资源（如限制用户创建的 Pod 使用超过限制的资源等）
  • 自定义资源的字段复杂验证（如 CRD 资源相关字段的规则验证等）

• 准入控制可以通过Webhook是一个HTTP回调，通过一个条件触发HTTP POST请求发送到Webhook 服务端，服务端根据请求数据进行处理

1. Webhook准备

• webhook核心就是处理apiservers发送的AdmissionReview请求，并将其决定作为AdmissionReview对象发送回去, 也就是修改mutating和验证validating
• Webhook和controller类似，既能在kubernetes环境中运行，也能在kubernetes环境之外运行

2. 自定义webhook

• 要完成一个自定义admission webhook需要两个步骤
  • 将相关的webhook config注册给kubernetes，也就是让kubernetes知道你的webhook
  • 准备一个http server来处理 apiserver发过来验证的信息

1. 向kubernetes注册webhook对象

• kubernetes将这种形式抽象为两种资源, 这两种资源是没有namespace限制的,是全局资源
  • ValidatingWebhookConfiguration
  • MutatingWebhookConfiguration
• 创建两个钩子，/mutate 与 /validate
  • /mutate 将在创建deployment资源时，基于版本，给资源加上 webhook.example.com/allow: true的annotations(注释)
  • /validate 将对 /mutate 增加了 allow:true 的注释批准通行，否则拒绝

# 获得集群的ca证书base64
cat /etc/kubernetes/pki/ca.crt | base64 | tr -d '\n'

# 获得当前上下文中的CA base64 信息
kubectl config view --raw --flatten -o json | jq -r '.clusters[] | .cluster."certificate-authority-data"'

# webhook会对资源进行修改,需要一个sa
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admission-webhook-example-sa
  labels:
    app: admission-webhook-example

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: admission-webhook-example-cr
  labels:
    app: admission-webhook-example
rules:
- apiGroups:
  - qikqiak.com
  resources:
  - "*"
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - pods
  - events
  verbs:
  - "*"
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - "*"
- apiGroups:
  - autoscaling
  resources:
  - '*'
  verbs:
  - '*'

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: admission-webhook-example-crb 
  labels:
    app: admission-webhook-example
subjects:
- kind: ServiceAccount
  name: admission-webhook-example-sa
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admission-webhook-example-cr

apiVersion: v1
kind: Service
metadata:
  name: admission-webhook-example-svc
  labels:
    app: admission-webhook-example
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: admission-webhook-example

# rbac文件由个ServiceAccount, 会创建一个secret, 
[root@webhook ~/webhood]# kubectl get secrets 
NAME                                       TYPE                                  DATA   AGE
admission-webhook-example-sa-token-s4jbv   kubernetes.io/service-account-token   3      3m49s

• K8S集群默认是HTTPS通信的，所以APiserver调用webhook的过程也是HTTPS的
• k8s集群1.22版本以后此脚本需要修改,具体参考 官网

# 证书签发脚本, 因为K8S集群默认是HTTPS通信的，所以APiserver调用webhook的过程也是HTTPS的，所以需要进行证书认证, 通过 CertificateSigningRequest 签发, --secret一定要指定自己的(如上面创建的sa)
#!/bin/bash

set -e

usage() {
    cat <<EOF
Generate certificate suitable for use with an sidecar-injector webhook service.

This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with sidecar-injector webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.

The server key/cert k8s CA cert are stored in a k8s secret.

usage: ${0} [OPTIONS]

The following flags are required.

       --service          Service name of webhook.
       --namespace        Namespace where webhook service and secret reside.
       --secret           Secret name for CA certificate and server certificate/key pair.
EOF
    exit 1
}

while [[ $# -gt 0 ]]; do
    case ${1} in
        --service)
            service="$2"
            shift
            ;;
        --secret)
            secret="$2"
            shift
            ;;
        --namespace)
            namespace="$2"
            shift
            ;;
        *)
            usage
            ;;
    esac
    shift
done

# 如果不想传递参数,可以直接在这改
[ -z ${service} ] && service=admission-webhook-example-svc
[ -z ${secret} ] && secret=admission-webhook-example-sa-token-s4jbv
[ -z ${namespace} ] && namespace=default

if [ ! -x "$(command -v openssl)" ]; then
    echo "openssl not found"
    exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf

# 清理之前为我们的服务创建的任何 CSR。 如果不存在则忽略错误
kubectl delete csr ${csrName} 2>/dev/null || true

# 创建服务器证书/密钥 CSR 并发送到 k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# verify CSR has been created
while true; do
    kubectl get csr ${csrName}
    if [ "$?" -eq 0 ]; then
        break
    fi
done

# 批准并获取签名证书
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi

# 写入到证书中
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=${tmpdir}/server-key.pem \
        --from-file=cert.pem=${tmpdir}/server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

# 查看生成的证书
[root@webhook ~/webhood]# kubectl get csr
NAME                                    AGE   SIGNERNAME                     REQUESTOR          CONDITION
admission-webhook-example-svc.default   5s    kubernetes.io/legacy-unknown   kubernetes-admin   Approved,Issued


# 将证书放在特定位置,这也是我们启动 deployment必须的 用于和api-server通信
mkdir -p /etc/webhook/certs

kubectl get secret ${secret} -o json | jq -r '.data."key.pem"' | base64 -d > /etc/webhook/certs/key.pem
kubectl get secret ${secret} -o json | jq -r '.data."cert.pem"' | base64 -d > /etc/webhook/certs/cert.pem

• MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: mutating-example.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/mutate"			# 和代码中路径一致
      caBundle: ${CA_BUNDLE}	# ca base64信息
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
        # namespace具有该标签才作用于资源对象
        admission-webhook-example: enabled

• ValidatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: required-labels.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/validate"		# 和代码中路径一致
      caBundle: ${CA_BUNDLE}	# ca base64信息
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
         # namespace具有该标签才作用于资源对象
        admission-webhook-example: enabled

# 给要使用的 namespace 打标签
kubectl label namespace default admission-webhook-example=enabled

2. Webhook代码

code

go build -o admission-webhook-example

3. webhook-deployment

FROM alpine:latest

WORKDIR /
ADD ./cert.pem  /etc/webhook/certs/cert.pem
ADD ./key.pem /etc/webhook/certs/key.pem

ADD ./admission-webhook-example /admission-webhook-example
RUN chmod +x /admission-webhook-example
ENTRYPOINT ["./admission-webhook-example"]

# 编译好的go程序admission-webhook-example也放在这
cd /etc/webhook/certs
cp /etc/webhook/certs/* .


# build
docker build -t admission-webhook:v1 .

# deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook-example-deployment
  labels:
    app: admission-webhook-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-webhook-example
  template:
    metadata:
      labels:
        app: admission-webhook-example
    spec:
      serviceAccount: admission-webhook-example-sa
      containers:
        - name: admission-webhook-example
          image: admission-webhook:v1
          imagePullPolicy: IfNotPresent
          # 证书位置改为自己的
          args:
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=4
            - 2>&1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
          # 名字改为创建的sa的secret名字
            secretName: admission-webhook-example-sa-token-s4jbv

4. 测试

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reject
  annotations:
    admission-webhook-example.qikqiak.com/mutate: "false"
spec:
  selector:
    matchLabels:
      app: reject
  template:
    metadata:
      labels:
        app: reject
    spec:
      containers:
        - name: reject
          image: tutum/curl
          command: ["/bin/sleep","infinity"]
          imagePullPolicy: IfNotPresent

# yaml中的注释 admission-webhook-example.qikqiak.com/mutate: "false", value 为 false,准入控制就拒绝, 去掉 annotations 就可以创建
[root@webhook ~/webhood/debug]# kubectl apply -f jujue.yaml 
Error from server (required labels are not set): error when creating "jujue.yaml": admission webhook "required-labels.qikqiak.com" denied the request: required labels are not set