Kubernetes API Server源码学习（一）：API Server架构设计、API Server启动过程、APIObject的装载、Scheme详解、GenericAPIServer

本文基于Kubernetes v1.22.4版本进行源码学习

1、API Server架构设计

API Server由3个HTTP Server组成：

• AggregatorServer：暴露的功能类似于一个七层负载均衡，将来自用户的请求拦截转发给其他服务器，并且负责整个APIServer的Discovery功能
• KubeAPIServer：负责处理Kubernetes内建资源的REST请求
• APIExtensionsServer：负责处理CustomResourceDefinition（CRD）和CustomResource（CR）的REST请求

3个HTTP Server的处理顺序如上图所示，当用户请求进来，先判断AggregatorServer能否处理，否则给KubeAPIServer，如果KubeAPIServer不能处理给APIExtensionsServer处理，APIExtensionsServer是Delegation的最后一环，如果对应请求不能被处理的话则会返回404

2、API Server启动过程

API Server的启动入口在cmd/kube-apiserver/apiserver.go文件中，该文件就包含一个main入口函数：

// cmd/kube-apiserver/apiserver.go
func main() {
	rand.Seed(time.Now().UnixNano())

	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)

	// 初始化Cobra.Command对象
	command := app.NewAPIServerCommand()

	logs.InitLogs()
	defer logs.FlushLogs()

	// 执行命令
	if err := command.Execute(); err != nil {
		os.Exit(1)
	}
}

main函数中通过app.NewAPIServerCommand()方法获得一个Cobra的Command对象，然后调用command.Execute()方法执行这个命令，NewAPIServerCommand()方法代码如下：

// cmd/kube-apiserver/app/server.go
func NewAPIServerCommand() *cobra.Command {
	// 获取默认的配置参数
	s := options.NewServerRunOptions()
	cmd := &cobra.Command{
		Use: "kube-apiserver",
		Long: `The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
cluster's shared state through which all other components interact.`,

		// stop printing usage when the command errors
		SilenceUsage: true,
		PersistentPreRunE: func(*cobra.Command, []string) error {
			// silence client-go warnings.
			// kube-apiserver loopback clients should not log self-issued warnings.
			rest.SetDefaultWarningHandler(rest.NoWarnings{})
			return nil
		},
		RunE: func(cmd *cobra.Command, args []string) error {
			verflag.PrintAndExitIfRequested()
			fs := cmd.Flags()
			cliflag.PrintFlags(fs)

			err := checkNonZeroInsecurePort(fs)
			if err != nil {
				return err
			}
			// set default options
			completedOptions, err := Complete(s)
			if err != nil {
				return err
			}

			// validate options
			if errs := completedOptions.Validate(); len(errs) != 0 {
				return utilerrors.NewAggregate(errs)
			}

			// 启动API Server主逻辑
			return Run(completedOptions, genericapiserver.SetupSignalHandler())
		},
		Args: func(cmd *cobra.Command, args []string) error {
			for _, arg := range args {
				if len(arg) > 0 {
					return fmt.Errorf("%q does not take any arguments, got %q", cmd.CommandPath(), args)
				}
			}
			return nil
		},
	}

	fs := cmd.Flags()
	namedFlagSets := s.Flags()
	verflag.AddFlags(namedFlagSets.FlagSet("global"))
	globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name())
	options.AddCustomGlobalFlags(namedFlagSets.FlagSet("generic"))
	for _, f := range namedFlagSets.FlagSets {
		fs.AddFlagSet(f)
	}

	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
	cliflag.SetUsageAndHelpFunc(cmd, namedFlagSets, cols)

	return cmd
}

Run()方法包含启动API Server主逻辑，代码如下：

// cmd/kube-apiserver/app/server.go
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
	// To help debugging, immediately log version
	klog.Infof("Version: %+v", version.Get())

	// 1)构建http server链,其中包含ApiServer要启动的三个server,以及为每个server注册对应资源的路由
	server, err := CreateServerChain(completeOptions, stopCh)
	if err != nil {
		return err
	}

	// 2)进行http server运行前的准备,如设置健康检查、存活检查和OpenAPI路由的注册工作等操作
	prepared, err := server.PrepareRun()
	if err != nil {
		return err
	}

	// 3)启动https server
	return prepared.Run(stopCh)
}

Run()方法主要逻辑如下：

1. 构建HTTP Server链，其中包含API Server要启动的三个Server，以及为每个Server注册对应资源的路由
2. 进行HTTP Server运行前的准备，如设置健康检查、存活检查和OpenAPI路由的注册工作等操作
3. 启动HTTPS Server

3、APIObject的装载

API Server的内容是APIObject，通过Restful服务对外提供操作APIObject的能力，那么API Server是如何建立起针对各个APIObject的服务的呢？

来看下KubeAPIServer中装载APIObject过程函数之间调用逻辑如下：

pkg/controlplane/instance.go中的New()方法包含两部分：

1. Legacy APIObject（Pod之类core APIGroup下的）的装载：调用InstallLegacyAPI()方法，InstallLegacyAPI()方法中再调用InstallLegacyAPIGroup()方法把core APIGroupInfo对象装载到API Server中
2. 内建的APIGroup（非core）中APIObject的装载：定义所有内建的APIGroup的RESTStorageProvider，调用InstallAPIs()方法传入RESTStorageProvider数组。InstallAPIs()方法中会遍历RESTStorageProvider数组来制作APIGroupInfo的对象，然后把所有内建APIGroupInfo对象装载到API Server中

4、Scheme详解

1）、Scheme作用和应用场景

Scheme是一个结构体，内含处理内外部Version之间转换，GVK和Go Type之间转换之用的数据和方法

• GVK与Go Type之间转换：Scheme内部有两个map，分别对应GVK到Type和Type到GVK；使得两者可以互相找到
• APIObject的默认值：APIObject有诸多属性，使用者在操作一个Object时，不太可能给出所有属性值；另外在Object从一个Version转换到另一个Version时也可能需要为不存在对应关系的字段填值
• 内外部Version之间Convert：所有外部Version都会别转换为内部Version，转换函数是记录在Scheme之内的

2）、Scheme内部结构

// staging/src/k8s.io/apimachinery/pkg/runtime/scheme.go
type Scheme struct {
	// versionMap allows one to figure out the go type of an object with
	// the given version and name.
	// gvk转换为go type
	gvkToType map[schema.GroupVersionKind]reflect.Type

	// typeToGroupVersion allows one to find metadata for a given go object.
	// The reflect.Type we index by should *not* be a pointer.
	// go type转换为gvk
	typeToGVK map[reflect.Type][]schema.GroupVersionKind

	// unversionedTypes are transformed without conversion in ConvertToVersion.
	// 不区分version的type
	unversionedTypes map[reflect.Type]schema.GroupVersionKind

	// unversionedKinds are the names of kinds that can be created in the context of any group
	// or version
	// TODO: resolve the status of unversioned types.
	// 不区分version的kind
	unversionedKinds map[string]reflect.Type

	// Map from version and resource to the corresponding func to convert
	// resource field labels in that version to internal version.
	fieldLabelConversionFuncs map[schema.GroupVersionKind]FieldLabelConversionFunc

	// defaulterFuncs is an array of interfaces to be called with an object to provide defaulting
	// the provided object must be a pointer.
	// 设置默认值的函数
	defaulterFuncs map[reflect.Type]func(interface{})

	// converter stores all registered conversion functions. It also has
	// default converting behavior.
	// 用于做转换
	converter *conversion.Converter

	// versionPriority is a map of groups to ordered lists of versions for those groups indicating the
	// default priorities of these versions as registered in the scheme
	// 记录version的优先级
	versionPriority map[string][]string

	// observedVersions keeps track of the order we've seen versions during type registration
	observedVersions []schema.GroupVersion

	// schemeName is the name of this scheme.  If you don't specify a name, the stack of the NewScheme caller will be used.
	// This is useful for error reporting to indicate the origin of the scheme.
	schemeName string
}

Scheme的重要方法：

• AddKnownTypes(gv schema.GroupVersion, types ...Object)：将APIObject的GVK存储到Scheme中
• AddConversionFunc(a, b interface{}, fn conversion.ConversionFunc)：将转换函数注册到Scheme中
• AddTypeDefaultingFunc(srcType Object, fn func(interface{}))：将设置默认值的函数注册到Scheme中

3）、Scheme的注册过程

Scheme的注册过程是通过Go语言的import和init机制触发的

// cmd/kube-apiserver/app/server.go
import (
  // ...
	"k8s.io/kubernetes/pkg/controlplane"
	// ...
)

cmd/kube-apiserver/app/server.go文件中import了k8s.io/kubernetes/pkg/controlplane这个包，这个包下有个pkg/controlplane/import_known_versions.go文件，代码如下：

// pkg/controlplane/import_known_versions.go
package controlplane

import (
	// These imports are the API groups the API server will support.
	_ "k8s.io/kubernetes/pkg/apis/admission/install"
	_ "k8s.io/kubernetes/pkg/apis/admissionregistration/install"
	_ "k8s.io/kubernetes/pkg/apis/apiserverinternal/install"
	_ "k8s.io/kubernetes/pkg/apis/apps/install"
	_ "k8s.io/kubernetes/pkg/apis/authentication/install"
	_ "k8s.io/kubernetes/pkg/apis/authorization/install"
	_ "k8s.io/kubernetes/pkg/apis/autoscaling/install"
	_ "k8s.io/kubernetes/pkg/apis/batch/install"
	_ "k8s.io/kubernetes/pkg/apis/certificates/install"
	_ "k8s.io/kubernetes/pkg/apis/coordination/install"
	_ "k8s.io/kubernetes/pkg/apis/core/install"
	_ "k8s.io/kubernetes/pkg/apis/discovery/install"
	_ "k8s.io/kubernetes/pkg/apis/events/install"
	_ "k8s.io/kubernetes/pkg/apis/extensions/install"
	_ "k8s.io/kubernetes/pkg/apis/flowcontrol/install"
	_ "k8s.io/kubernetes/pkg/apis/imagepolicy/install"
	_ "k8s.io/kubernetes/pkg/apis/networking/install"
	_ "k8s.io/kubernetes/pkg/apis/node/install"
	_ "k8s.io/kubernetes/pkg/apis/policy/install"
	_ "k8s.io/kubernetes/pkg/apis/rbac/install"
	_ "k8s.io/kubernetes/pkg/apis/scheduling/install"
	_ "k8s.io/kubernetes/pkg/apis/storage/install"
)

pkg/controlplane/import_known_versions.go文件中import了各个内置API Group的install包，这里会触发各个内置API Group的注册工作。以apps这个API Group为例，代码如下：

// pkg/apis/apps/install/install.go
package install

import (
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"k8s.io/kubernetes/pkg/api/legacyscheme"
	"k8s.io/kubernetes/pkg/apis/apps"
	"k8s.io/kubernetes/pkg/apis/apps/v1"
	"k8s.io/kubernetes/pkg/apis/apps/v1beta1"
	"k8s.io/kubernetes/pkg/apis/apps/v1beta2"
)

func init() {
	Install(legacyscheme.Scheme)
}

// Install registers the API group and adds types to a scheme
func Install(scheme *runtime.Scheme) {
	// 内部版本注册
	utilruntime.Must(apps.AddToScheme(scheme))
	// 外部版本注册
	utilruntime.Must(v1beta1.AddToScheme(scheme))
	utilruntime.Must(v1beta2.AddToScheme(scheme))
	utilruntime.Must(v1.AddToScheme(scheme))
	utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion, v1beta2.SchemeGroupVersion, v1beta1.SchemeGroupVersion))
}

init()方法中调用了apps这个API group内外部各个版本的AddToScheme()方法，把对应版本的类型等信息放入同一个Scheme，对于KubeAPIServer这个Scheme是legacyscheme.Scheme变量

4）、内部版本注册

pkg/apis/apps/install/install.go文件的Install()方法中先调用了apps.AddToScheme(scheme)进行内部版本注册，代码如下：

// pkg/apis/apps/register.go
package apps

import (
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/kubernetes/pkg/apis/autoscaling"
)

var (
	// SchemeBuilder stores functions to add things to a scheme.
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme applies all stored functions t oa scheme.
	AddToScheme = SchemeBuilder.AddToScheme
)

// GroupName is the group name use in this package
const GroupName = "apps"

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: runtime.APIVersionInternal}

// Kind takes an unqualified kind and returns a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

// Adds the list of known types to the given scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	// TODO this will get cleaned up with the scheme types are fixed
	scheme.AddKnownTypes(SchemeGroupVersion,
		&DaemonSet{},
		&DaemonSetList{},
		&Deployment{},
		&DeploymentList{},
		&DeploymentRollback{},
		&autoscaling.Scale{},
		&StatefulSet{},
		&StatefulSetList{},
		&ControllerRevision{},
		&ControllerRevisionList{},
		&ReplicaSet{},
		&ReplicaSetList{},
	)
	return nil
}

该文件中先调用runtime.NewSchemeBuilder(addKnownTypes)初始化了SchemeBuilder，传入了addKnownTypes()这个函数，pkg/apis/apps/install/install.go文件的init()方法中调用的apps.AddToScheme()实际是SchemeBuilder.AddToScheme的函数引用

来看下SchemeBuilder，代码如下：

// vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go
// SchemeBuilder collects functions that add things to a scheme. It's to allow
// code to compile without explicitly referencing generated types. You should
// declare one in each package that will have generated deep copy or conversion
// functions.
type SchemeBuilder []func(*Scheme) error

// AddToScheme applies all the stored functions to the scheme. A non-nil error
// indicates that one function failed and the attempt was abandoned.
func (sb *SchemeBuilder) AddToScheme(s *Scheme) error {
	for _, f := range *sb {
		if err := f(s); err != nil {
			return err
		}
	}
	return nil
}

// Register adds a scheme setup function to the list.
func (sb *SchemeBuilder) Register(funcs ...func(*Scheme) error) {
	for _, f := range funcs {
		*sb = append(*sb, f)
	}
}

// NewSchemeBuilder calls Register for you.
func NewSchemeBuilder(funcs ...func(*Scheme) error) SchemeBuilder {
	var sb SchemeBuilder
	sb.Register(funcs...)
	return sb
}

SchemeBuilder本质上是一个函数数组集合，其中的函数入参为Scheme类型，有两个重要的方法：

• Register()：向函数数组集合中再添加一个入参为Scheme类型的函数
• AddToScheme()：把该Scheme对象传入函数数组集合中的每一个函数，然后依次运行

使用SchemeBuilder注册Scheme过程如下图：

以apps这个API Group内部版本注册为例，看下这个流程：

1. install.go中的init()时的Install()方法中的apps.AddToScheme(scheme)调用了register.go中的AddToScheme函数
2. register.go中的AddToScheme函数指向了SchemeBuilder.AddToScheme
3. register.go中定义SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)，这里是向SchemeBuilder这个函数数组集合中添加了addKnownTypes()函数
4. register.go中定义addKnownType方法是调用scheme.AddKnownTypes()将apps这个API Group中的APIObject的GVK存储到Scheme中
5. 当 install.go调用register.go中的AddToScheme函数，实际调用的是SchemeBuilder.AddToScheme，该函数会依次调用SchemeBuilder这个函数数组集合中的所有函数。也会调用到register.go中添加的addKnownTypes()函数，最终是调用scheme.AddKnownTypes()将APIObject的GVK存储到Scheme中

内部版本注册过程如下图：

5）、外部版本注册

外部版本注册以apps这个API Group的v1beta1版本为例，pkg/apis/apps/install/install.go文件的Install()方法中先调用了v1beta1.AddToScheme(scheme)，代码如下：

// pkg/apis/apps/v1beta1/register.go
var (
	localSchemeBuilder = &appsv1beta1.SchemeBuilder
	AddToScheme        = localSchemeBuilder.AddToScheme
)

func init() {
	// We only register manually written functions here. The registration of the
	// generated functions takes place in the generated files. The separation
	// makes the code compile even when the generated files are missing.
	localSchemeBuilder.Register(addDefaultingFuncs, addConversionFuncs)
}

和内部版本不同的是这里的localSchemeBuilder是指向appsv1beta1.SchemeBuilder，同时在init()方法中传入了addDefaultingFuncs()和addConversionFuncs()两个函数，appsv1beta1.SchemeBuilder实际是定义在vendor/k8s.io/api/apps/v1beta1/register.go文件中代码如下：

// vendor/k8s.io/api/apps/v1beta1/register.go
var (
	// TODO: move SchemeBuilder with zz_generated.deepcopy.go to k8s.io/api.
	// localSchemeBuilder and AddToScheme will stay in k8s.io/kubernetes.
	SchemeBuilder      = runtime.NewSchemeBuilder(addKnownTypes)
	localSchemeBuilder = &SchemeBuilder
	AddToScheme        = localSchemeBuilder.AddToScheme
)

// Adds the list of known types to the given scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Deployment{},
		&DeploymentList{},
		&DeploymentRollback{},
		&Scale{},
		&StatefulSet{},
		&StatefulSetList{},
		&ControllerRevision{},
		&ControllerRevisionList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

外部版本注册过程如下图：

5、GenericAPIServer

1）、GenericAPIServer的定位

1. 提供暴露HTTP服务所需的基础设施

   各个内部Server做的事情是一致的：对外提供Restful服务来操作APIObject。所以大框架上大家是一致的：需要去实现HTTP Restful服务，大家都需要HTTP Server，那么这可以集中提供

   各个内部Server会相互连接，形成处理链，这同样需要有实体来负责

2. 统一各种处理机制

   对于同一个事项，不同的内部Server应该采取同样的方式，这在开源项目中是比较那一保证的。例如API Resource的对外暴露形式、登录鉴权机制

3. 避免重复

   大量的实现都是可以被复用的

每个内部Server都是构建在GenericAPIServer之上，把自己的内容填入GenericAPIServer

每个GenericAPIServer最重要的输出是一个叫Director的东西，它本质上是一个mux和一个go container的组合，所有的HTTP Request最终都是被这些Director处理的

2）、go http和go restful

用go http包做HTTP Server：

• http.Server
• Mux：请求的路由，把url映射成handler
• HandleFunc（函数签名有要求）

用go-restful库做HTTP服务：

• http.Server
• Container：相当于WebService聚合地，内含一个mux，所以实现了http.Handler。http.Server生成时作为mux传入
• Router：url、http method和handler的三元组
• WebService：一组route构成一个WebService，一个WebService内到的route具有相同的rootPath

3）、GenericAPIServer的装配

vendor/k8s.io/apiserver/pkg/server/config.go文件中的New()方法中构建了GenericAPIServer，代码如下：

// vendor/k8s.io/apiserver/pkg/server/config.go
func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
	if c.Serializer == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.Serializer == nil")
	}
	if c.LoopbackClientConfig == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.LoopbackClientConfig == nil")
	}
	if c.EquivalentResourceRegistry == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.EquivalentResourceRegistry == nil")
	}

	// 构建handlerChain
	// config.BuildHandlerChainFunc的实现为DefaultBuildHandlerChain方法
	handlerChainBuilder := func(handler http.Handler) http.Handler {
		return c.BuildHandlerChainFunc(handler, c.Config)
	}
	// 构建NewAPIServerHandler
	apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

	// 实例化GenericAPIServer
	s := &GenericAPIServer{
		discoveryAddresses:         c.DiscoveryAddresses,
		LoopbackClientConfig:       c.LoopbackClientConfig,
		legacyAPIGroupPrefixes:     c.LegacyAPIGroupPrefixes,
		admissionControl:           c.AdmissionControl,
		Serializer:                 c.Serializer,
		AuditBackend:               c.AuditBackend,
		Authorizer:                 c.Authorization.Authorizer,
		delegationTarget:           delegationTarget,
		EquivalentResourceRegistry: c.EquivalentResourceRegistry,
		HandlerChainWaitGroup:      c.HandlerChainWaitGroup,

		minRequestTimeout:     time.Duration(c.MinRequestTimeout) * time.Second,
		ShutdownTimeout:       c.RequestTimeout,
		ShutdownDelayDuration: c.ShutdownDelayDuration,
		SecureServingInfo:     c.SecureServing,
		ExternalAddress:       c.ExternalAddress,

		// 构建了http request的路由,连接了url和响应函数;同时包含了一个request需要经过的预处理函数
		Handler: apiServerHandler,

		listedPathProvider: apiServerHandler,

		openAPIConfig:           c.OpenAPIConfig,
		skipOpenAPIInstallation: c.SkipOpenAPIInstallation,

		// 这些hook集合在New方法接下来的代码中填充,包含自己定义的和delegationTarget上的
		postStartHooks:         map[string]postStartHookEntry{},
		preShutdownHooks:       map[string]preShutdownHookEntry{},
		disabledPostStartHooks: c.DisabledPostStartHooks,

		healthzChecks:    c.HealthzChecks,
		livezChecks:      c.LivezChecks,
		readyzChecks:     c.ReadyzChecks,
		livezGracePeriod: c.LivezGracePeriod,

		DiscoveryGroupManager: discovery.NewRootAPIsHandler(c.DiscoveryAddresses, c.Serializer),

		maxRequestBodyBytes: c.MaxRequestBodyBytes,
		livezClock:          clock.RealClock{},

		lifecycleSignals: c.lifecycleSignals,

		APIServerID:           c.APIServerID,
		StorageVersionManager: c.StorageVersionManager,

		Version: c.Version,
	}

	for {
		if c.JSONPatchMaxCopyBytes <= 0 {
			break
		}
		existing := atomic.LoadInt64(&jsonpatch.AccumulatedCopySizeLimit)
		if existing > 0 && existing < c.JSONPatchMaxCopyBytes {
			break
		}
		if atomic.CompareAndSwapInt64(&jsonpatch.AccumulatedCopySizeLimit, existing, c.JSONPatchMaxCopyBytes) {
			break
		}
	}

	// 处理钩子hook操作
	// first add poststarthooks from delegated targets
	for k, v := range delegationTarget.PostStartHooks() {
		s.postStartHooks[k] = v
	}

	for k, v := range delegationTarget.PreShutdownHooks() {
		s.preShutdownHooks[k] = v
	}

	// add poststarthooks that were preconfigured.  Using the add method will give us an error if the same name has already been registered.
	for name, preconfiguredPostStartHook := range c.PostStartHooks {
		if err := s.AddPostStartHook(name, preconfiguredPostStartHook.hook); err != nil {
			return nil, err
		}
	}

	genericApiServerHookName := "generic-apiserver-start-informers"
	if c.SharedInformerFactory != nil {
		if !s.isPostStartHookRegistered(genericApiServerHookName) {
			err := s.AddPostStartHook(genericApiServerHookName, func(context PostStartHookContext) error {
				c.SharedInformerFactory.Start(context.StopCh)
				return nil
			})
			if err != nil {
				return nil, err
			}
		}
		// TODO: Once we get rid of /healthz consider changing this to post-start-hook.
		err := s.AddReadyzChecks(healthz.NewInformerSyncHealthz(c.SharedInformerFactory))
		if err != nil {
			return nil, err
		}
	}

	const priorityAndFairnessConfigConsumerHookName = "priority-and-fairness-config-consumer"
	if s.isPostStartHookRegistered(priorityAndFairnessConfigConsumerHookName) {
	} else if c.FlowControl != nil {
		err := s.AddPostStartHook(priorityAndFairnessConfigConsumerHookName, func(context PostStartHookContext) error {
			go c.FlowControl.MaintainObservations(context.StopCh)
			go c.FlowControl.Run(context.StopCh)
			return nil
		})
		if err != nil {
			return nil, err
		}
		// TODO(yue9944882): plumb pre-shutdown-hook for request-management system?
	} else {
		klog.V(3).Infof("Not requested to run hook %s", priorityAndFairnessConfigConsumerHookName)
	}

	// Add PostStartHooks for maintaining the watermarks for the Priority-and-Fairness and the Max-in-Flight filters.
	if c.FlowControl != nil {
		const priorityAndFairnessFilterHookName = "priority-and-fairness-filter"
		if !s.isPostStartHookRegistered(priorityAndFairnessFilterHookName) {
			err := s.AddPostStartHook(priorityAndFairnessFilterHookName, func(context PostStartHookContext) error {
				genericfilters.StartPriorityAndFairnessWatermarkMaintenance(context.StopCh)
				return nil
			})
			if err != nil {
				return nil, err
			}
		}
	} else {
		const maxInFlightFilterHookName = "max-in-flight-filter"
		if !s.isPostStartHookRegistered(maxInFlightFilterHookName) {
			err := s.AddPostStartHook(maxInFlightFilterHookName, func(context PostStartHookContext) error {
				genericfilters.StartMaxInFlightWatermarkMaintenance(context.StopCh)
				return nil
			})
			if err != nil {
				return nil, err
			}
		}
	}

	for _, delegateCheck := range delegationTarget.HealthzChecks() {
		skip := false
		for _, existingCheck := range c.HealthzChecks {
			if existingCheck.Name() == delegateCheck.Name() {
				skip = true
				break
			}
		}
		if skip {
			continue
		}
		s.AddHealthChecks(delegateCheck)
	}

	s.listedPathProvider = routes.ListedPathProviders{s.listedPathProvider, delegationTarget}

	// 安装API相关参数
	installAPI(s, c.Config)

	// use the UnprotectedHandler from the delegation target to ensure that we don't attempt to double authenticator, authorize,
	// or some other part of the filter chain in delegation cases.
	if delegationTarget.UnprotectedHandler() == nil && c.EnableIndex {
		s.Handler.NonGoRestfulMux.NotFoundHandler(routes.IndexLister{
			StatusCode:   http.StatusNotFound,
			PathProvider: s.listedPathProvider,
		})
	}

	return s, nil
}

GenericAPIServer中重要的属性如下：

• Handler：构建了HTTP Request的路由，连接了url和响应函数；同时包含了一个Request需要经过的预处理函数
• postStartHooks和preShutdownHooks：这些Hook集合在New()方法接下来的代码中填充，包含自己定义的和delegationTarget上的

4）、Request Handler的构建

New()方法中调用NewAPIServerHandler()方法来构建ApiServerHandler，代码如下：

// vendor/k8s.io/apiserver/pkg/server/handler.go
func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
	nonGoRestfulMux := mux.NewPathRecorderMux(name)
	// 把下个server的director设为NotFoundHandler
	if notFoundHandler != nil {
		nonGoRestfulMux.NotFoundHandler(notFoundHandler)
	}

	gorestfulContainer := restful.NewContainer()
	gorestfulContainer.ServeMux = http.NewServeMux()
	gorestfulContainer.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*}
	gorestfulContainer.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
		logStackOnRecover(s, panicReason, httpWriter)
	})
	gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
		serviceErrorHandler(s, serviceErr, request, response)
	})

	director := director{
		name:               name,
		goRestfulContainer: gorestfulContainer,
		nonGoRestfulMux:    nonGoRestfulMux,
	}

	// 主要由director来处理http请求,用handlerChainBuilder加了一层
	return &APIServerHandler{
		FullHandlerChain:   handlerChainBuilder(director),
		GoRestfulContainer: gorestfulContainer,
		NonGoRestfulMux:    nonGoRestfulMux,
		Director:           director,
	}
}

APIServerHandler实现了http.Handler接口，http.Server将能够把request交给APIServerHandler处理。APIServerHandler的ServeHTTP方法直接把request交给FullHandlerChain，代码如下：

// vendor/k8s.io/apiserver/pkg/server/handler.go
func (a *APIServerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	a.FullHandlerChain.ServeHTTP(w, r)
}

handlerChainBuilder是对completedConfig的BuildHandlerChainFunc函数的封装，默认实现如下：

// vendor/k8s.io/apiserver/pkg/server/config.go
// 通过装饰器模式包装apiHandler,包含认证、鉴权等一系列http filter chain,要先过这些http filter chain才访问到apiHandler
func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
	handler := filterlatency.TrackCompleted(apiHandler)
	handler = genericapifilters.WithAuthorization(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "authorization")

	if c.FlowControl != nil {
		handler = filterlatency.TrackCompleted(handler)
		handler = genericfilters.WithPriorityAndFairness(handler, c.LongRunningFunc, c.FlowControl, c.RequestWidthEstimator)
		handler = filterlatency.TrackStarted(handler, "priorityandfairness")
	} else {
		handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
	}

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "impersonation")

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
	handler = filterlatency.TrackStarted(handler, "audit")

	failedHandler := genericapifilters.Unauthorized(c.Serializer)
	failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)

	failedHandler = filterlatency.TrackCompleted(failedHandler)
	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
	handler = filterlatency.TrackStarted(handler, "authentication")

	handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")

	// WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the
	// context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client.
	handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

	handler = genericapifilters.WithRequestDeadline(handler, c.AuditBackend, c.AuditPolicyChecker,
		c.LongRunningFunc, c.Serializer, c.RequestTimeout)
	handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
	if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 {
		handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance)
	}
	handler = genericapifilters.WithAuditAnnotations(handler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithWarningRecorder(handler)
	handler = genericapifilters.WithCacheControl(handler)
	handler = genericfilters.WithHSTS(handler, c.HSTSDirectives)
	handler = genericfilters.WithHTTPLogging(handler)
	if utilfeature.DefaultFeatureGate.Enabled(genericfeatures.APIServerTracing) {
		handler = genericapifilters.WithTracing(handler, c.TracerProvider)
	}
	handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithRequestReceivedTimestamp(handler)
	handler = genericfilters.WithPanicRecovery(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithAuditID(handler)
	return handler
}

DefaultBuildHandlerChain()方法通过装饰器模式包装apiHandler，包含认证、鉴权等一系列http filter chain，要先过这些http filter chain才访问到apiHandler

5）、Server Chain的形成

// vendor/k8s.io/apiserver/pkg/server/config.go
func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
  // ...
  // 构建NewAPIServerHandler
	apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

New()方法中调用NewAPIServerHandler()方法传入的notFoundHandler为delegationTarget.UnprotectedHandler()

// vendor/k8s.io/apiserver/pkg/server/handler.go
func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
	nonGoRestfulMux := mux.NewPathRecorderMux(name)
	// 把下个server的director设为NotFoundHandler
	if notFoundHandler != nil {
		nonGoRestfulMux.NotFoundHandler(notFoundHandler)
	}

	gorestfulContainer := restful.NewContainer()
	gorestfulContainer.ServeMux = http.NewServeMux()
	gorestfulContainer.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*}
	gorestfulContainer.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
		logStackOnRecover(s, panicReason, httpWriter)
	})
	gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
		serviceErrorHandler(s, serviceErr, request, response)
	})

	director := director{
		name:               name,
		goRestfulContainer: gorestfulContainer,
		nonGoRestfulMux:    nonGoRestfulMux,
	}

	// 主要由director来处理http请求,用handlerChainBuilder加了一层
	return &APIServerHandler{
		FullHandlerChain:   handlerChainBuilder(director),
		GoRestfulContainer: gorestfulContainer,
		NonGoRestfulMux:    nonGoRestfulMux,
		Director:           director,
	}
}

NewAPIServerHandler()方法中把下个Server的director设为NotFoundHandler

// vendor/k8s.io/apiserver/pkg/server/handler.go
func (a *APIServerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  // 请求被转发给FullHandlerChain处理
	a.FullHandlerChain.ServeHTTP(w, r)
}

请求被转发给FullHandlerChain处理，最终被转给director处理

// vendor/k8s.io/apiserver/pkg/server/handler.gos
func (d director) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	path := req.URL.Path

	// check to see if our webservices want to claim this path
	for _, ws := range d.goRestfulContainer.RegisteredWebServices() {
		switch {
		case ws.RootPath() == "/apis":
			// if we are exactly /apis or /apis/, then we need special handling in loop.
			// normally these are passed to the nonGoRestfulMux, but if discovery is enabled, it will go directly.
			// We can't rely on a prefix match since /apis matches everything (see the big comment on Director above)
			if path == "/apis" || path == "/apis/" {
				klog.V(5).Infof("%v: %v %q satisfied by gorestful with webservice %v", d.name, req.Method, path, ws.RootPath())
				// don't use servemux here because gorestful servemuxes get messed up when removing webservices
				// TODO fix gorestful, remove TPRs, or stop using gorestful
				d.goRestfulContainer.Dispatch(w, req)
				return
			}

		case strings.HasPrefix(path, ws.RootPath()):
			// ensure an exact match or a path boundary match
			if len(path) == len(ws.RootPath()) || path[len(ws.RootPath())] == '/' {
				klog.V(5).Infof("%v: %v %q satisfied by gorestful with webservice %v", d.name, req.Method, path, ws.RootPath())
				// don't use servemux here because gorestful servemuxes get messed up when removing webservices
				// TODO fix gorestful, remove TPRs, or stop using gorestful
				d.goRestfulContainer.Dispatch(w, req)
				return
			}
		}
	}

	// if we didn't find a match, then we just skip gorestful altogether
	klog.V(5).Infof("%v: %v %q satisfied by nonGoRestful", d.name, req.Method, path)
	// 当director发现当前server没有处理该请求时,转给NotFoundHandler
	d.nonGoRestfulMux.ServeHTTP(w, req)
}

当director发现当前server没有处理该请求时，转给NotFoundHandler。也就是当前Server找不到对应Request的处理方法时，交给delegationTarget，AggregatorServer的delegationTarget为KubeAPIServer，KubeAPIServer的delegationTarget为APIExtensionsServer，这样就形成了HTTP Server链

参考：

Kubernetes源码开发之旅三：API Server源码剖析

设计模式之Builder及其在Kubernetes API Server 中的应用