研发工程师玩转Kubernetes——使用Node特性定向调度Pod

在《研发工程师玩转Kubernetes——使用污点（taint）驱逐Pod》中我们提到亲和性（affinity）中的requiredDuringSchedulingIgnoredDuringExecution，它可以定向调度Pod。本节我们将使用相关特性完成定向调度的介绍。

nodeSelector定向调度

我们先模拟Pod被部署在Master Node上的场景。这个时候我们可以使用nodeSelector对Node的Labels做选择。
首先我们查看当前的Node的情况：

查看Node的Labels

get nodes --show-labels

NAME      STATUS   ROLES    AGE   VERSION   LABELS
ubuntuc   Ready    >   16m   v1.26.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntuc,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntue   Ready    >   15m   v1.26.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntue,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntud   Ready    >   15m   v1.26.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntud,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntub   Ready    >   16m   v1.26.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntub,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntua   Ready    >   20m   v1.27.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntua,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane

可以看到只有Master Node（UbuntuA）的Labels含有node.kubernetes.io/microk8s-controlplane=microk8s-controlplane，我们就用这个去做条件选择。

清单文件指定Label

# nginx_deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-container
        image: nginx
        ports:
        - containerPort: 80
      nodeSelector:
        node.kubernetes.io/microk8s-controlplane: microk8s-controlplane

注意最后两行，即使用nodeSelector让Pod在Labels含有node.kubernetes.io/microk8s-controlplane: microk8s-controlplane的Node上部署。

NodeSelector map[string]string `json:"nodeSelector,omitempty"`

部署

针对上述文件部署Pod

kubectl create -f nginx_deployment.yaml

deployment.apps/nginx-deployment created

kubectl get pods -o wide

NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
nginx-deployment-5d5bc8fb96-75c9d   1/1     Running   0          67s   10.1.94.67   ubuntua   >           >

可以看到Pod被部署到Master Node上了。

requiredDuringSchedulingIgnoredDuringExecution定向调度

requiredDuringSchedulingIgnoredDuringExecution是亲和性（affinity）下的一种类型，它表示：只有在规则被满足的时候才能执行调度。
我们修改下清单文件，让Pod只能被调度到Master Node(UbuntuA)上。

清单文件指定亲和性

# nginx_deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/microk8s-controlplane
                operator: In
                values:
                - microk8s-controlplane
      containers:
      - name: nginx-container
        image: nginx
        ports:
        - containerPort: 80

nodeSelectorTerms只能是一组matchExpressions。

type NodeAffinity struct {
	// If the affinity requirements specified by this field are not met at
	// scheduling time, the pod will not be scheduled onto the node.
	// If the affinity requirements specified by this field cease to be met
	// at some point during pod execution (e.g. due to a node label update),
	// the system will try to eventually evict the pod from its node.
	RequiredDuringSchedulingRequiredDuringExecution *NodeSelector  `json:"requiredDuringSchedulingRequiredDuringExecution,omitempty"`
	// If the affinity requirements specified by this field are not met at
	// scheduling time, the pod will not be scheduled onto the node.
	// If the affinity requirements specified by this field cease to be met
	// at some point during pod execution (e.g. due to a node label update),
	// the system may or may not try to eventually evict the pod from its node.
	RequiredDuringSchedulingIgnoredDuringExecution  *NodeSelector  `json:"requiredDuringSchedulingIgnoredDuringExecution,omitempty"`
	// The scheduler will prefer to schedule pods to nodes that satisfy
	// the affinity expressions specified by this field, but it may choose
	// a node that violates one or more of the expressions. The node that is
	// most preferred is the one with the greatest sum of weights, i.e.
	// for each node that meets all of the scheduling requirements (resource
	// request, RequiredDuringScheduling affinity expressions, etc.),
	// compute a sum by iterating through the elements of this field and adding
	// "weight" to the sum if the node matches the corresponding MatchExpressions; the
	// node(s) with the highest sum are the most preferred.
	PreferredDuringSchedulingIgnoredDuringExecution []PreferredSchedulingTerm  `json:"preferredDuringSchedulingIgnoredDuringExecution,omitempty"`
}

// A node selector represents the union of the results of one or more label queries
// over a set of nodes; that is, it represents the OR of the selectors represented
// by the nodeSelectorTerms.
type NodeSelector struct {
	// nodeSelectorTerms is a list of node selector terms. The terms are ORed.
	NodeSelectorTerms []NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`
}

// An empty node selector term matches all objects. A null node selector term
// matches no objects.
type NodeSelectorTerm struct {
	// matchExpressions is a list of node selector requirements. The requirements are ANDed.
	MatchExpressions []NodeSelectorRequirement `json:"matchExpressions,omitempty"`
}

// A node selector requirement is a selector that contains values, a key, and an operator
// that relates the key and values.
type NodeSelectorRequirement struct {
	// key is the label key that the selector applies to.
	Key string `json:"key" patchStrategy:"merge" patchMergeKey:"key"`
	// operator represents a key's relationship to a set of values.
	// Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
	Operator NodeSelectorOperator `json:"operator"`
	// values is an array of string values. If the operator is In or NotIn,
	// the values array must be non-empty. If the operator is Exists or DoesNotExist,
	// the values array must be empty. If the operator is Gt or Lt, the values
	// array must have a single element, which will be interpreted as an integer.
    // This array is replaced during a strategic merge patch.
	Values []string `json:"values,omitempty"`
}

// A node selector operator is the set of operators that can be used in
// a node selector requirement.
type NodeSelectorOperator string

const (
	NodeSelectorOpIn           NodeSelectorOperator = "In"
	NodeSelectorOpNotIn        NodeSelectorOperator = "NotIn"
	NodeSelectorOpExists       NodeSelectorOperator = "Exists"
	NodeSelectorOpDoesNotExist NodeSelectorOperator = "DoesNotExist"
	NodeSelectorOpGt           NodeSelectorOperator = "Gt"
	NodeSelectorOpLt           NodeSelectorOperator = "Lt"
)

观察

get pod --watch -o wide

NAME                                READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
nginx-deployment-549db84c68-gk66n   0/1     Pending             0          0s      >        >    >           >
nginx-deployment-549db84c68-gk66n   0/1     Pending             0          0s      >        ubuntua   >           >
nginx-deployment-549db84c68-gk66n   0/1     ContainerCreating   0          0s      >        ubuntua   >           >
nginx-deployment-549db84c68-gk66n   0/1     ContainerCreating   0          1s      >        ubuntua   >           >
nginx-deployment-549db84c68-gk66n   1/1     Running             0          4s      10.1.94.80    ubuntua   >           >

可以看到Pod是在Worker Node（UbuntuA）上部署着。

参考资料

• https://kubernetes.io/zh-cn/docs/concepts/scheduling-eviction/assign-pod-node/
• https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/podaffinity.md