Kubelet v1.25.x源码——PluginManager
1. 环境说明
Kubernetes源码版本：remotes/origin/release-1.25
Kubernetes编译出来的Kubelet版本：Kubernetes v1.24.0-beta.0.2463+ee7799bab469d7
Kubernetes集群实验环境：使用Kubernetes v1.25.4二进制的方式搭建了一个单节点集群
K8S 单节点单节点搭建可以参考：Kubernetes v1.25 搭建单节点集群用于Debug K8S源码
Golang版本：go1.19.3 linux/amd64
IDEA版本：2022.2.3
Delve版本：1.9.1
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]# dlv version
Delve Debugger
Version: 1.9.1
Build: $Id: d81b9fd12bfa603f3cf7a4bc842398bd61c42940 $
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]# go version
go version go1.19.3 linux/amd64
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]# kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master1 Ready <none> 31h v1.25.4 192.168.11.71 <none> CentOS Linux 7 (Core) 3.10.0-1160.80.1.el7.x86_64 containerd://1.6.10
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]#
[root@k8s-master1 kubernetes]# kubectl get componentstatus
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
etcd-0 Healthy {"health":"true","reason":""}
controller-manager Healthy ok
scheduler Healthy ok
[root@k8s-master1 kubernetes]#
Kubelet启动参数配置如下：
[root@k8s-master1 kubernetes]# ps -ef|grep "/usr/local/bin/kubelet"
root 7972 1 6 07:06 ? 00:00:06 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-conf.yml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-labels=node.kubernetes.io/node= --v=8
root 9549 6424 0 07:07 pts/0 00:00:00 grep --color=auto /usr/local/bin/kubelet
[root@k8s-master1 kubernetes]#
Kubelet参数配置如下：
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
2. 组件概览
| 组件概览 |
在还未看到代码之前，从名字上来看，PluginManager用于管理插件，我心中不免泛起以下几个疑问？
- 1、
Plugin是如何注册的？ - 2、
Plugin可以用于完成什么功能？ - 3、
Plugin的作用发生在什么时间点？ - 4、
Plugin的生命周期？ - 5、
Plugin是扩展Kubernetes功能的一种方式么？ - 6、我们知道
DevicePlugin适用于管理gpu这样的物理设备，那么这里的Plugin和DevicePlugin是一个东西么？
下面，让我们带着这些疑问，进入到PluginManager的源码当中。
3. 源码分析
3.1. PluginManager
| PluginManager |
可以看到，PluginManager的定义非常简单，定了一个Run方法，用于启动PluginManager，可以预见，Run方法中一定完成了对于Plugin的Manager。而AddHandler则用用于添加PluginHandler.
Run方法中出现了Source，这个Source应该就是之前分析过的Pod来源，一共就三个，分别是http, file, apiserver
type PluginManager interface {
Run(sourcesReady config.SourcesReady, stopCh <-chan struct{})
AddHandler(pluginType string, pluginHandler cache.PluginHandler)
}
3.2. PluginHandler
| PluginHandler |
PluginHandler中则是定义了插件的校验，注册、注销方法。
type PluginHandler interface {
ValidatePlugin(pluginName string, endpoint string, versions []string) error
RegisterPlugin(pluginName, endpoint string, versions []string) error
DeRegisterPlugin(pluginName string)
}
PluginManager是通过组合cache, operationexecutor, pluginwatcher, reconciler这四个模块来完成的，我们先分别看看这四个模块完成了什么功能
3.3. cache
| cache |
3.3.1. ActualStateOfWorld
| ActualStateOfWorld |
从名字上看，ActualStateOfWorld表示的是某个事物的真实状态，它的实现本质上就是一个Map
type ActualStateOfWorld interface {
GetRegisteredPlugins() []PluginInfo
AddPlugin(pluginInfo PluginInfo) error
RemovePlugin(socketPath string)
PluginExistsWithCorrectTimestamp(pluginInfo PluginInfo) bool
}
// NewActualStateOfWorld returns a new instance of ActualStateOfWorld
func NewActualStateOfWorld() ActualStateOfWorld {
return &actualStateOfWorld{
socketFileToInfo: make(map[string]PluginInfo),
}
}
type actualStateOfWorld struct {
socketFileToInfo map[string]PluginInfo
sync.RWMutex
}
var _ ActualStateOfWorld = &actualStateOfWorld{}
// PluginInfo holds information of a plugin
type PluginInfo struct {
SocketPath string
Timestamp time.Time
Handler PluginHandler
Name string
}
func (asw *actualStateOfWorld) AddPlugin(pluginInfo PluginInfo) error {
asw.Lock()
defer asw.Unlock()
if pluginInfo.SocketPath == "" {
return fmt.Errorf("socket path is empty")
}
if _, ok := asw.socketFileToInfo[pluginInfo.SocketPath]; ok {
klog.V(2).InfoS("Plugin exists in actual state cache", "path", pluginInfo.SocketPath)
}
asw.socketFileToInfo[pluginInfo.SocketPath] = pluginInfo
return nil
}
func (asw *actualStateOfWorld) RemovePlugin(socketPath string) {
asw.Lock()
defer asw.Unlock()
delete(asw.socketFileToInfo, socketPath)
}
func (asw *actualStateOfWorld) GetRegisteredPlugins() []PluginInfo {
asw.RLock()
defer asw.RUnlock()
currentPlugins := []PluginInfo{}
for _, pluginInfo := range asw.socketFileToInfo {
currentPlugins = append(currentPlugins, pluginInfo)
}
return currentPlugins
}
func (asw *actualStateOfWorld) PluginExistsWithCorrectTimestamp(pluginInfo PluginInfo) bool {
asw.RLock()
defer asw.RUnlock()
actualStatePlugin, exists := asw.socketFileToInfo[pluginInfo.SocketPath]
return exists && (actualStatePlugin.Timestamp == pluginInfo.Timestamp)
}
3.3.2. DesiredStateOfWorld
| DesiredStateOfWorld |
DesiredStateOfWorld表示的是期望状态，显然，reconcile的目标就是吧AutualStateOfWorld调整为DesiredStateOfWorld状态。DesiredStateOfWorld的实现本质上也是一个map，逻辑也并不复杂，这里不做过多解释。
type DesiredStateOfWorld interface {
AddOrUpdatePlugin(socketPath string) error
RemovePlugin(socketPath string)
GetPluginsToRegister() []PluginInfo
PluginExists(socketPath string) bool
}
func NewDesiredStateOfWorld() DesiredStateOfWorld {
return &desiredStateOfWorld{
socketFileToInfo: make(map[string]PluginInfo),
}
}
type desiredStateOfWorld struct {
socketFileToInfo map[string]PluginInfo
sync.RWMutex
}
var _ DesiredStateOfWorld = &desiredStateOfWorld{}
// Generate a detailed error msg for logs
func generatePluginMsgDetailed(prefixMsg, suffixMsg, socketPath, details string) (detailedMsg string) {
return fmt.Sprintf("%v for plugin at %q %v %v", prefixMsg, socketPath, details, suffixMsg)
}
func generatePluginMsg(prefixMsg, suffixMsg, socketPath, details string) (simpleMsg, detailedMsg string) {
simpleMsg = fmt.Sprintf("%v for plugin at %q %v", prefixMsg, socketPath, suffixMsg)
return simpleMsg, generatePluginMsgDetailed(prefixMsg, suffixMsg, socketPath, details)
}
func (plugin *PluginInfo) GenerateMsgDetailed(prefixMsg, suffixMsg string) (detailedMsg string) {
detailedStr := fmt.Sprintf("(plugin details: %v)", plugin)
return generatePluginMsgDetailed(prefixMsg, suffixMsg, plugin.SocketPath, detailedStr)
}
func (plugin *PluginInfo) GenerateMsg(prefixMsg, suffixMsg string) (simpleMsg, detailedMsg string) {
detailedStr := fmt.Sprintf("(plugin details: %v)", plugin)
return generatePluginMsg(prefixMsg, suffixMsg, plugin.SocketPath, detailedStr)
}
func (plugin *PluginInfo) GenerateErrorDetailed(prefixMsg string, err error) (detailedErr error) {
return fmt.Errorf(plugin.GenerateMsgDetailed(prefixMsg, errSuffix(err)))
}
func (plugin *PluginInfo) GenerateError(prefixMsg string, err error) (simpleErr, detailedErr error) {
simpleMsg, detailedMsg := plugin.GenerateMsg(prefixMsg, errSuffix(err))
return fmt.Errorf(simpleMsg), fmt.Errorf(detailedMsg)
}
func errSuffix(err error) string {
errStr := ""
if err != nil {
errStr = fmt.Sprintf(": %v", err)
}
return errStr
}
func (dsw *desiredStateOfWorld) AddOrUpdatePlugin(socketPath string) error {
dsw.Lock()
defer dsw.Unlock()
if socketPath == "" {
return fmt.Errorf("socket path is empty")
}
if _, ok := dsw.socketFileToInfo[socketPath]; ok {
klog.V(2).InfoS("Plugin exists in actual state cache, timestamp will be updated", "path", socketPath)
}
dsw.socketFileToInfo[socketPath] = PluginInfo{
SocketPath: socketPath,
Timestamp: time.Now(),
}
return nil
}
3.4. operationexecutor
3.4.1. OperationGenerator
| OperationGenerator |
OperationGenerator用于生成插件的注销和注册方法
type OperationGenerator interface {
GenerateRegisterPluginFunc(
socketPath string,
timestamp time.Time,
pluginHandlers map[string]cache.PluginHandler,
actualStateOfWorldUpdater ActualStateOfWorldUpdater) func() error
GenerateUnregisterPluginFunc(
pluginInfo cache.PluginInfo,
actualStateOfWorldUpdater ActualStateOfWorldUpdater) func() error
}
3.4.1.1. GenerateRegisterPluginFunc
| GenerateRegisterPluginFunc |
GenerateRegisterPluginFunc我们来看看，它是如何生成插件的注册方法的。具体逻辑如下：
- 1、和要注册的插件建立
grpc连接 - 2、通过
grpc接口GetInfo获取插件信息 - 3、从获取到的插件信息中获取插件类型，然后根据插件的类型获取插件处理器
- 4、如果获取插件处理器失败，那么通过
grpc接口NotifyRegistrationStatus通知插件 - 5、利用获取的插件处理器校验插件，如果校验出错，那么通过
NotifyRegistrationStatus grpc接口通知插件 - 6、向
ActualStateOfWorld中添加插件 - 7、利用插件处理器注册插件
- 8、通过
NotifyRegistrationStatus grpc接口通知插件已经注册成功
func dial(unixSocketPath string, timeout time.Duration) (registerapi.RegistrationClient, *grpc.ClientConn, error) {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
c, err := grpc.DialContext(ctx, unixSocketPath,
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithBlock(),
grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", addr)
}),
)
if err != nil {
return nil, nil, fmt.Errorf("failed to dial socket %s, err: %v", unixSocketPath, err)
}
return registerapi.NewRegistrationClient(c), c, nil
}
func (og *operationGenerator) GenerateRegisterPluginFunc(
socketPath string,
timestamp time.Time,
pluginHandlers map[string]cache.PluginHandler,
actualStateOfWorldUpdater ActualStateOfWorldUpdater) func() error {
registerPluginFunc := func() error {
client, conn, err := dial(socketPath, dialTimeoutDuration)
if err != nil {
return fmt.Errorf("RegisterPlugin error -- dial failed at socket %s, err: %v", socketPath, err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
infoResp, err := client.GetInfo(ctx, ®isterapi.InfoRequest{})
if err != nil {
return fmt.Errorf("RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket %s, err: %v", socketPath, err)
}
handler, ok := pluginHandlers[infoResp.Type]
if !ok {
if err := og.notifyPlugin(client, false, fmt.Sprintf("RegisterPlugin error -- no handler registered for plugin type: %s at socket %s", infoResp.Type, socketPath)); err != nil {
return fmt.Errorf("RegisterPlugin error -- failed to send error at socket %s, err: %v", socketPath, err)
}
return fmt.Errorf("RegisterPlugin error -- no handler registered for plugin type: %s at socket %s", infoResp.Type, socketPath)
}
if infoResp.Endpoint == "" {
infoResp.Endpoint = socketPath
}
if err := handler.ValidatePlugin(infoResp.Name, infoResp.Endpoint, infoResp.SupportedVersions); err != nil {
if err = og.notifyPlugin(client, false, fmt.Sprintf("RegisterPlugin error -- plugin validation failed with err: %v", err)); err != nil {
return fmt.Errorf("RegisterPlugin error -- failed to send error at socket %s, err: %v", socketPath, err)
}
return fmt.Errorf("RegisterPlugin error -- pluginHandler.ValidatePluginFunc failed")
}
err = actualStateOfWorldUpdater.AddPlugin(cache.PluginInfo{
SocketPath: socketPath,
Timestamp: timestamp,
Handler: handler,
Name: infoResp.Name,
})
if err != nil {
klog.ErrorS(err, "RegisterPlugin error -- failed to add plugin", "path", socketPath)
}
if err := handler.RegisterPlugin(infoResp.Name, infoResp.Endpoint, infoResp.SupportedVersions); err != nil {
return og.notifyPlugin(client, false, fmt.Sprintf("RegisterPlugin error -- plugin registration failed with err: %v", err))
}
if err := og.notifyPlugin(client, true, ""); err != nil {
return fmt.Errorf("RegisterPlugin error -- failed to send registration status at socket %s, err: %v", socketPath, err)
}
return nil
}
return registerPluginFunc
}
3.4.1.2. GenerateUnregisterPluginFunc
| GenerateUnregisterPluginFunc |
注销插件功能非常简单，首先从ActualStateOfWorld缓存中移除插件，然后利用插件的处理器注销插件即可
func (og *operationGenerator) GenerateUnregisterPluginFunc(
pluginInfo cache.PluginInfo,
actualStateOfWorldUpdater ActualStateOfWorldUpdater) func() error {
unregisterPluginFunc := func() error {
if pluginInfo.Handler == nil {
return fmt.Errorf("UnregisterPlugin error -- failed to get plugin handler for %s", pluginInfo.SocketPath)
}
actualStateOfWorldUpdater.RemovePlugin(pluginInfo.SocketPath)
pluginInfo.Handler.DeRegisterPlugin(pluginInfo.Name)
klog.V(4).InfoS("DeRegisterPlugin called", "pluginName", pluginInfo.Name, "pluginHandler", pluginInfo.Handler)
return nil
}
return unregisterPluginFunc
}
3.4.2. OperationExecutor
| OperationExecutor |
OperationExecutor的逻辑到不复杂，主要还是利用了OperationGenerator的能力生成注册和注销方法，然后调用
type OperationExecutor interface {
RegisterPlugin(socketPath string, timestamp time.Time, pluginHandlers map[string]cache.PluginHandler, actualStateOfWorld ActualStateOfWorldUpdater) error
UnregisterPlugin(pluginInfo cache.PluginInfo, actualStateOfWorld ActualStateOfWorldUpdater) error
}
func NewOperationExecutor(
operationGenerator OperationGenerator) OperationExecutor {
return &operationExecutor{
pendingOperations: goroutinemap.NewGoRoutineMap(true /* exponentialBackOffOnError */),
operationGenerator: operationGenerator,
}
}
type ActualStateOfWorldUpdater interface {
AddPlugin(pluginInfo cache.PluginInfo) error
RemovePlugin(socketPath string)
}
type operationExecutor struct {
pendingOperations goroutinemap.GoRoutineMap
operationGenerator OperationGenerator
}
var _ OperationExecutor = &operationExecutor{}
func (oe *operationExecutor) IsOperationPending(socketPath string) bool {
return oe.pendingOperations.IsOperationPending(socketPath)
}
func (oe *operationExecutor) RegisterPlugin(
socketPath string,
timestamp time.Time,
pluginHandlers map[string]cache.PluginHandler,
actualStateOfWorld ActualStateOfWorldUpdater) error {
generatedOperation :=
oe.operationGenerator.GenerateRegisterPluginFunc(socketPath, timestamp, pluginHandlers, actualStateOfWorld)
return oe.pendingOperations.Run(
socketPath, generatedOperation)
}
func (oe *operationExecutor) UnregisterPlugin(
pluginInfo cache.PluginInfo,
actualStateOfWorld ActualStateOfWorldUpdater) error {
generatedOperation :=
oe.operationGenerator.GenerateUnregisterPluginFunc(pluginInfo, actualStateOfWorld)
return oe.pendingOperations.Run(
pluginInfo.SocketPath, generatedOperation)
}
3.5. PluginWatcher
| PluginWatcher |
PluginWatcher主要用于Socket注册。
type Watcher struct {
path string
fs utilfs.Filesystem
fsWatcher *fsnotify.Watcher
desiredStateOfWorld cache.DesiredStateOfWorld
}
// NewWatcher provides a new watcher for socket registration
func NewWatcher(sockDir string, desiredStateOfWorld cache.DesiredStateOfWorld) *Watcher {
return &Watcher{
path: sockDir,
fs: &utilfs.DefaultFs{},
desiredStateOfWorld: desiredStateOfWorld,
}
}
func (w *Watcher) Start(stopCh <-chan struct{}) error {
klog.V(2).InfoS("Plugin Watcher Start", "path", w.path)
if err := w.init(); err != nil {
return err
}
fsWatcher, err := fsnotify.NewWatcher()
if err != nil {
return fmt.Errorf("failed to start plugin fsWatcher, err: %v", err)
}
w.fsWatcher = fsWatcher
// Traverse plugin dir and add filesystem watchers before starting the plugin processing goroutine.
if err := w.traversePluginDir(w.path); err != nil {
klog.ErrorS(err, "Failed to traverse plugin socket path", "path", w.path)
}
go func(fsWatcher *fsnotify.Watcher) {
for {
select {
case event := <-fsWatcher.Events:
//TODO: Handle errors by taking corrective measures
if event.Op&fsnotify.Create == fsnotify.Create {
err := w.handleCreateEvent(event)
if err != nil {
klog.ErrorS(err, "Error when handling create event", "event", event)
}
} else if event.Op&fsnotify.Remove == fsnotify.Remove {
w.handleDeleteEvent(event)
}
continue
case err := <-fsWatcher.Errors:
if err != nil {
klog.ErrorS(err, "FsWatcher received error")
}
continue
case <-stopCh:
w.fsWatcher.Close()
return
}
}
}(fsWatcher)
return nil
}
func (w *Watcher) init() error {
klog.V(4).InfoS("Ensuring Plugin directory", "path", w.path)
if err := w.fs.MkdirAll(w.path, 0755); err != nil {
return fmt.Errorf("error (re-)creating root %s: %v", w.path, err)
}
return nil
}
func (w *Watcher) traversePluginDir(dir string) error {
// watch the new dir
err := w.fsWatcher.Add(dir)
if err != nil {
return fmt.Errorf("failed to watch %s, err: %v", w.path, err)
}
// traverse existing children in the dir
return w.fs.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
if path == dir {
return fmt.Errorf("error accessing path: %s error: %v", path, err)
}
klog.ErrorS(err, "Error accessing path", "path", path)
return nil
}
// do not call fsWatcher.Add twice on the root dir to avoid potential problems.
if path == dir {
return nil
}
mode := info.Mode()
if mode.IsDir() {
if err := w.fsWatcher.Add(path); err != nil {
return fmt.Errorf("failed to watch %s, err: %v", path, err)
}
} else if isSocket, _ := util.IsUnixDomainSocket(path); isSocket {
event := fsnotify.Event{
Name: path,
Op: fsnotify.Create,
}
//TODO: Handle errors by taking corrective measures
if err := w.handleCreateEvent(event); err != nil {
klog.ErrorS(err, "Error when handling create", "event", event)
}
} else {
klog.V(5).InfoS("Ignoring file", "path", path, "mode", mode)
}
return nil
})
}
func (w *Watcher) handleCreateEvent(event fsnotify.Event) error {
klog.V(6).InfoS("Handling create event", "event", event)
fi, err := os.Stat(event.Name)
if err != nil && runtime.GOOS == "windows" {
fi, err = os.Lstat(event.Name)
}
if err != nil {
return fmt.Errorf("stat file %s failed: %v", event.Name, err)
}
if strings.HasPrefix(fi.Name(), ".") {
klog.V(5).InfoS("Ignoring file (starts with '.')", "path", fi.Name())
return nil
}
if !fi.IsDir() {
isSocket, err := util.IsUnixDomainSocket(util.NormalizePath(event.Name))
if err != nil {
return fmt.Errorf("failed to determine if file: %s is a unix domain socket: %v", event.Name, err)
}
if !isSocket {
klog.V(5).InfoS("Ignoring non socket file", "path", fi.Name())
return nil
}
return w.handlePluginRegistration(event.Name)
}
return w.traversePluginDir(event.Name)
}
func (w *Watcher) handlePluginRegistration(socketPath string) error {
if runtime.GOOS == "windows" {
socketPath = util.NormalizePath(socketPath)
}
klog.V(2).InfoS("Adding socket path or updating timestamp to desired state cache", "path", socketPath)
err := w.desiredStateOfWorld.AddOrUpdatePlugin(socketPath)
if err != nil {
return fmt.Errorf("error adding socket path %s or updating timestamp to desired state cache: %v", socketPath, err)
}
return nil
}
func (w *Watcher) handleDeleteEvent(event fsnotify.Event) {
klog.V(6).InfoS("Handling delete event", "event", event)
socketPath := event.Name
klog.V(2).InfoS("Removing socket path from desired state cache", "path", socketPath)
w.desiredStateOfWorld.RemovePlugin(socketPath)
}
3.6. Reconciler
| Reconciler |
在进入PluginManager之前，我们先来看看Reconciler是用来干嘛的、
写过Operator的小伙伴应该会特别熟悉，Reconciler就是就是kubebuilder留给用户需要填充的业务逻辑。Reconciler主要用于完成把实际状态调整为期望状态。
那么，这里的Reconciler是用来干嘛的呢？
从注释上来看，Reconciler定期执行register/unregister逻辑以达到期望的逻辑。
type Reconciler interface {
Run(stopCh <-chan struct{})
AddHandler(pluginType string, pluginHandler cache.PluginHandler)
}
| reconciler |
从Reconciler接口的实现者reconciler定义来看，Reconcile通过operationExecutor操作把actualStateOfWorld状态变为desiredStateOfWorld
type reconciler struct {
operationExecutor operationexecutor.OperationExecutor
loopSleepDuration time.Duration
desiredStateOfWorld cache.DesiredStateOfWorld
actualStateOfWorld cache.ActualStateOfWorld
handlers map[string]cache.PluginHandler
sync.RWMutex
}
3.6.1. Run
| Run |
Run 实现的非常简单，就是一个死循环，不停的执行reconcile逻辑
func (rc *reconciler) Run(stopCh <-chan struct{}) {
wait.Until(func() {
rc.reconcile()
},
rc.loopSleepDuration,
stopCh)
}
3.6.1.1. reconcile
| reconcile |
func (rc *reconciler) reconcile() {
for _, registeredPlugin := range rc.actualStateOfWorld.GetRegisteredPlugins() {
unregisterPlugin := false
if !rc.desiredStateOfWorld.PluginExists(registeredPlugin.SocketPath) {
unregisterPlugin = true
} else {
for _, dswPlugin := range rc.desiredStateOfWorld.GetPluginsToRegister() {
if dswPlugin.SocketPath == registeredPlugin.SocketPath && dswPlugin.Timestamp != registeredPlugin.Timestamp {
klog.V(5).InfoS("An updated version of plugin has been found, unregistering the plugin first before reregistering", "plugin", registeredPlugin)
unregisterPlugin = true
break
}
}
}
if unregisterPlugin {
klog.V(5).InfoS("Starting operationExecutor.UnregisterPlugin", "plugin", registeredPlugin)
err := rc.operationExecutor.UnregisterPlugin(registeredPlugin, rc.actualStateOfWorld)
if err != nil &&
!goroutinemap.IsAlreadyExists(err) &&
!exponentialbackoff.IsExponentialBackoff(err) {
// Ignore goroutinemap.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
// Log all other errors.
klog.ErrorS(err, "OperationExecutor.UnregisterPlugin failed", "plugin", registeredPlugin)
}
if err == nil {
klog.V(1).InfoS("OperationExecutor.UnregisterPlugin started", "plugin", registeredPlugin)
}
}
}
// Ensure plugins that should be registered are registered
for _, pluginToRegister := range rc.desiredStateOfWorld.GetPluginsToRegister() {
if !rc.actualStateOfWorld.PluginExistsWithCorrectTimestamp(pluginToRegister) {
klog.V(5).InfoS("Starting operationExecutor.RegisterPlugin", "plugin", pluginToRegister)
err := rc.operationExecutor.RegisterPlugin(pluginToRegister.SocketPath, pluginToRegister.Timestamp, rc.getHandlers(), rc.actualStateOfWorld)
if err != nil &&
!goroutinemap.IsAlreadyExists(err) &&
!exponentialbackoff.IsExponentialBackoff(err) {
// Ignore goroutinemap.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
klog.ErrorS(err, "OperationExecutor.RegisterPlugin failed", "plugin", pluginToRegister)
}
if err == nil {
klog.V(1).InfoS("OperationExecutor.RegisterPlugin started", "plugin", pluginToRegister)
}
}
}
}
3.6.2. AddHandler
| AddHandler |
func (rc *reconciler) AddHandler(pluginType string, pluginHandler cache.PluginHandler) {
rc.Lock()
defer rc.Unlock()
rc.handlers[pluginType] = pluginHandler
}
3.7. Run
| Run |
最后，我们来看看PluginManager的Run方法都干了啥
func (pm *pluginManager) Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) {
defer runtime.HandleCrash()
if err := pm.desiredStateOfWorldPopulator.Start(stopCh); err != nil {
klog.ErrorS(err, "The desired_state_of_world populator (plugin watcher) starts failed!")
return
}
klog.V(2).InfoS("The desired_state_of_world populator (plugin watcher) starts")
klog.InfoS("Starting Kubelet Plugin Manager")
go pm.reconciler.Run(stopCh)
metrics.Register(pm.actualStateOfWorld, pm.desiredStateOfWorld)
<-stopCh
klog.InfoS("Shutting down Kubelet Plugin Manager")
}