要点01：软件配置项的来源主要有2块：命令行参数 和 配置文件
- k8s中的组件一般都是遵循上面的模式
- 我们以kube-scheduler为例
- 命令参数举例 ：可以看到--xxx=xxx的传参，当然所有参数都是有默认值的，如果你不传就走默认
/usr/local/bin/kube-scheduler --log-dir=/var/log/k8s --logtostderr=false --alsologtostderr=true --config=/etc/k8s/kube-scheduler.yaml --kube-api-qps=500 --kube-api-burst=500 --authentication-kubeconfig=/etc/k8s/scheduler.conf --authorization-kubeconfig=/etc/k8s/scheduler.conf --kubeconfig=/etc/k8s/scheduler.conf --leader-elect=true --v=2- 上面的命令行参数中可以看到--config=xxx.yaml指定了 配置文件的路径
- 比如我们可以查看 这个配置文件的内容 ：发现基本就是官方的默认配置
cat /etc/k8s/kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/etc/k8s/scheduler.conf"
leaderElection:
leaderElect: true
metricsBindAddress: 0.0.0.0:10251
要点02：k8s组件提供了一个应用运行时查看生效配置的接口
思考为何要提供？主要原因有下面几点
- 来自命令行和配置文件的配置2块可能有些覆盖的地方
- 配置项目太多了
在没有配置热更新的情况下：查看变更是否生效：
- 配置文件已经更改，但忘记是应用重启前还是重启后改的了
- 所以讲究的项目里面都会留有一个http接口
- 直接将当前应用内存中生效的配置项目打印出来，方便排查问题
- 所以我们后面写golang的项目也可以仿照这个接口
要点03 : 如何请求scheduler的配置接口
k8s组件对应查看配置的接口就是configz
访问组件接口需要鉴权，我们可以通过sa来实现
如何在1.24集群中创建rbac
apiVersion: rbac.authorization.k8s.io/v1 # api的version
kind: ClusterRole # 类型
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources: # 资源
- nodes
- nodes/metrics
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus # 自定义名字
namespace: kube-system # 命名空间
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef: # 选择需要绑定的Role
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects: # 对象
- kind: ServiceAccount
name: prometheus
namespace: kube-system
---
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: prometheus
annotations:
kubernetes.io/service-account.name: prometheus
type: kubernetes.io/service-account-token
获取token命令
root@k8s-master01:~# TOKEN=$(kubectl -n kube-system get secret prometheus -o jsonpath='{.data.token}'| base64 --decode )
root@k8s-master01:~# echo $TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6ImFVMS1mYlhobWIxcF92djBwbUIxZDhTVlFWd0VNa3VpNDlmOUhqcG9qSlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2OTg2NWUwYy0yOGE4LTQ3YTEtYWEzYy03NThmNDlkYjA1YWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06cHJvbWV0aGV1cyJ9.SlORtZtVvLptYWE3zvblaOMxHNMBrHVTTHra7fO1RrwxdK3Bzc42ETLvkzVfQmAQlWrq5yiB4HiKFLe4qY2KVwK3qLDS_sWADLI16Sv8-O1Dt0oOQ0UZD0VOSGY0XEq2EGUxgxnx_JWllgEuMd0rjxtAtyjFh9wjCo_07lFCj44BffGGFp6Kovd8Dl_CJKpORakaJW-haIvTmTlbFPbRKojRTyKvtNCVn0zIXsz8Esp7z9XZmtUvZmHqNlY7bAFtGM9qLWUY_PkM1C0lQ2ZDKASdhZpx6LJr1Wo4WSNILCVfECT0sd6TnFHbgd1NwBc0kcTct5VbST76AJwUpG5esA
在http请求中把上面的TOKEN 作为beare-token传过去就可以了
请求的curl命令如下(在kube-scheduler部署的节点上执行，通常是master)
curl -k -s https://localhost:10259/configz --header "Authorization: Bearer $TOKEN" |python -m json.tool要点04 根据返回的json配置段来分析插件开启的情况
- json数据如下
{
"componentconfig": {
"AlgorithmSource": {
"Policy": null,
"Provider": "DefaultProvider"
},
"ClientConnection": {
"AcceptContentTypes": "",
"Burst": 100,
"ContentType": "application/vnd.kubernetes.protobuf",
"Kubeconfig": "/etc/k8s/scheduler.conf",
"QPS": 50
},
"EnableContentionProfiling": true,
"EnableProfiling": true,
"Extenders": null,
"HealthzBindAddress": "0.0.0.0:10251",
"LeaderElection": {
"LeaderElect": true,
"LeaseDuration": "15s",
"RenewDeadline": "10s",
"ResourceLock": "leases",
"ResourceName": "kube-scheduler",
"ResourceNamespace": "kube-system",
"RetryPeriod": "2s"
},
"MetricsBindAddress": "0.0.0.0:10251",
"Parallelism": 16,
"PercentageOfNodesToScore": 0,
"PodInitialBackoffSeconds": 1,
"PodMaxBackoffSeconds": 10,
"Profiles": [
{
"PluginConfig": [
{
"Args": {
"MinCandidateNodesAbsolute": 100,
"MinCandidateNodesPercentage": 10
},
"Name": "DefaultPreemption"
},
{
"Args": {
"HardPodAffinityWeight": 1
},
"Name": "InterPodAffinity"
},
{
"Args": {
"AddedAffinity": null
},
"Name": "NodeAffinity"
},
{
"Args": {
"IgnoredResourceGroups": null,
"IgnoredResources": null
},
"Name": "NodeResourcesFit"
},
{
"Args": {
"Resources": [
{
"Name": "cpu",
"Weight": 1
},
{
"Name": "memory",
"Weight": 1
}
]
},
"Name": "NodeResourcesLeastAllocated"
},
{
"Args": {
"DefaultConstraints": null,
"DefaultingType": "System"
},
"Name": "PodTopologySpread"
},
{
"Args": {
"BindTimeoutSeconds": 600
},
"Name": "VolumeBinding"
}
],
"Plugins": {
"Bind": {
"Disabled": null,
"Enabled": [
{
"Name": "DefaultBinder",
"Weight": 0
}
]
},
"Filter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeUnschedulable",
"Weight": 0
},
{
"Name": "NodeName",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
},
{
"Name": "NodeAffinity",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "NodeResourcesFit",
"Weight": 0
},
{
"Name": "VolumeRestrictions",
"Weight": 0
},
{
"Name": "EBSLimits",
"Weight": 0
},
{
"Name": "GCEPDLimits",
"Weight": 0
},
{
"Name": "NodeVolumeLimits",
"Weight": 0
},
{
"Name": "AzureDiskLimits",
"Weight": 0
},
{
"Name": "VolumeBinding",
"Weight": 0
},
{
"Name": "VolumeZone",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "InterPodAffinity",
"Weight": 0
}
]
},
"Permit": {
"Disabled": null,
"Enabled": null
},
"PostBind": {
"Disabled": null,
"Enabled": null
},
"PostFilter": {
"Disabled": null,
"Enabled": [
{
"Name": "DefaultPreemption",
"Weight": 0
}
]
},
"PreBind": {
"Disabled": null,
"Enabled": [
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"PreFilter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeResourcesFit",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "InterPodAffinity",
"Weight": 0
},
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"PreScore": {
"Disabled": null,
"Enabled": [
{
"Name": "InterPodAffinity",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
}
]
},
"QueueSort": {
"Disabled": null,
"Enabled": [
{
"Name": "PrioritySort",
"Weight": 0
}
]
},
"Reserve": {
"Disabled": null,
"Enabled": [
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"Score": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeResourcesBalancedAllocation",
"Weight": 1
},
{
"Name": "ImageLocality",
"Weight": 1
},
{
"Name": "InterPodAffinity",
"Weight": 1
},
{
"Name": "NodeResourcesLeastAllocated",
"Weight": 1
},
{
"Name": "NodeAffinity",
"Weight": 1
},
{
"Name": "NodePreferAvoidPods",
"Weight": 10000
},
{
"Name": "PodTopologySpread",
"Weight": 2
},
{
"Name": "TaintToleration",
"Weight": 1
}
]
}
},
"SchedulerName": "default-scheduler"
}
]
}
}
前置知识 k8s的调度框架 scheduler framework
- 文档位置
- 这张架构图一定要烂熟于心
我们发现在 componentconfig.Profiles其实主要分两块：
Plugins段代表调度框架每个扩展点的插件开启和禁用的情况：
- 可以理解为在各个阶段：需要依次执行开启列表中的插件，不执行禁用列表中的插件
- 比如在Filter阶段(我节选了一部分) 都是咱们经常能看到的插件：去掉NodeUnschedulable 、过滤NodeName、过滤TaintToleration 还有根据request资源情况的NodeResourcesFit
"Filter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeUnschedulable",
"Weight": 0
},
{
"Name": "NodeName",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
},
{
"Name": "NodeAffinity",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "NodeResourcesFit",
"Weight": 0
},
]
},- PluginConfig段代表各个插件的配置字段：这里是个map 的k-v形式：很好理解就是每个插件的配置字段不一样，直接用map最方便
- 比如 NodeResourcesLeastAllocated插件
{
"Args": {
"Resources": [
{
"Name": "cpu",
"Weight": 1
},
{
"Name": "memory",
"Weight": 1
}
]
},
"Name": "NodeResourcesLeastAllocated"
},要点6：关于插件权重的讨论
- 在上面的配置中可以看到很多 阶段如Filter和Score 的插件中都有权重的配置
{
"Name": "NodePreferAvoidPods",
"Weight": 10000
},- 但是只有在Score中的插件的Weight值才大于0
- 其余阶段的插件的Weight值都是0
结合代码看看 ：找到 Plugin.Weight (1.22分支)
- 位置 D:\go_path\src\github.com\kubernetes\kubernetes\pkg\scheduler\apis\config\types.go
// Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.
type Plugin struct {
// Name defines the name of plugin
Name string
// Weight defines the weight of plugin, only used for Score plugins.
Weight int32
}
- 我们发现Plugin结构体就是一个名字和权重
- 并且从注释中明确的看到了权重只有在 Score阶段的插件中才有作用