Kubernetes 使用自定义资源（CRD）扩展API

K8s CRD 即 Kubernetes CustomResourceDefinition，是 Kubernetes 提供的一种扩展机制，允许用户在 Kubernetes 集群中定义和使用自定义的资源类型。通过定义 CRD，用户可以在 Kubernetes 集群中创建、读取、更新和删除自定义资源对象，就像使用原生的 Pod、Service 等资源一样。

本文主要介绍如何使用kubebuilder快速创建自定义资源类型。完成guestbook资源类型自定义，并通过CRD Controller创建deloyment对象，通过deployment运行管理pod

前提条件

• 安装go https://go.dev/doc/install
• 安装k8s（可查看上一篇文章）

安装kubebuilder

参考文档步骤：https://book.kubebuilder.io/quick-start

# download kubebuilder and install locally.
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

创建CRD

创建Project & CRD API

mkdir -p ~/projects/guestbook
cd ~/projects/guestbook
kubebuilder init --domain my.domain --repo my.domain/guestbook
kubebuilder create api --group webapp --version v1 --kind Guestbook

执行完上面两个命令之后，会初始化一个k8s控制器项目，并且创建自定义资源及相关代码。

安装CRD

执行make manifests生成资源配置文件

[root@master guestbook]# make manifests 
/root/crd/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

根据需要编辑api/v1/guestbook_types.go文件

// GuestbookSpec defines the desired state of Guestbook.
type GuestbookSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Guestbook. Edit guestbook_types.go to remove/update
        Foo      string `json:"foo,omitempty"`
        Replicas int32  `json:"replicas,omitempty"`
        Image    string `json:"image,omitempty"`
}

// GuestbookStatus defines the observed state of Guestbook.
type GuestbookStatus struct {
        // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
        // Important: Run "make" to regenerate code after modifying this file
        ServiceStatus string `json:"serviceStatus"`
        Replicas      int32  `json:"replicas,omitempty"`
}

编写CRD Controller代码internal/controller/guestbook_controller.go
前面只是定义了guestbook的资源类型，当创建guestbook类型时需要做什么操作是通过CRD Controller来完成的， 本示例用controller创建一个deployment类型的资源运行两个Nginx Pod。

kubebuilder已经帮我们生成了代码框架guestbook_controller.go，只需要在Reconcile方法添加相关的逻辑即可。

/*
Copyright 2025.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
    "context"
    "fmt"

    "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"
    corev1 "k8s.io/api/core/v1"
    appsv1 "k8s.io/api/apps/v1"
    webappv1 "my.domain/guestbook/api/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// GuestbookReconciler reconciles a Guestbook object
type GuestbookReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=webapp.my.domain,resources=guestbooks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=webapp.my.domain,resources=guestbooks/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=webapp.my.domain,resources=guestbooks/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Guestbook object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/reconcile

//资源对象创建/更新/删除/变化时触发调用
func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)
    fmt.Printf("Guestbook Reconcile called....%v...%v \n",req.Name,req.Namespace)

    // 获取当前已存在的Deployment对象（如果有的话）
    existingDeployment := &appsv1.Deployment{}
    err1 := r.Get(ctx, client.ObjectKey{
       Namespace: req.Namespace,
       Name:      fmt.Sprintf("%s-deployment", req.Name),
    }, existingDeployment)


    var guestbook webappv1.Guestbook
    if err := r.Get(ctx, req.NamespacedName, &guestbook); err != nil {
       //如果guestbook资源不存在，删除已有的deployment
       if errors.IsNotFound(err) {
          return r.handleGuestbookDeleted(existingDeployment,ctx,req)
       }
    }

    desiredDeployment := r.buildDeploymentForGuestbook(&guestbook,req)
    if err1!=nil && errors.IsNotFound(err1) {
       // 如果不存在，则创建Deployment
       if err := r.Create(ctx, desiredDeployment); err != nil {
          return ctrl.Result{}, fmt.Errorf("failed to create Deployment: %v", err)
       }
       logger.Info("Deployment created successfully")
    } else {
       // 对比期望的Deployment和已存在的Deployment，若有差异则更新
       if !r.deploymentsEqual(desiredDeployment, existingDeployment) {
          existingDeployment.Spec = desiredDeployment.Spec
          if err := r.Update(ctx, existingDeployment); err != nil {
             return ctrl.Result{}, fmt.Errorf("failed to update Deployment: %v", err)
          }
          logger.Info("Deployment updated successfully")
       }
    }

    // 更新Guestbook资源的状态（这里简单示例，可根据实际情况完善状态更新逻辑）
    fmt.Printf("guestbook spec replicas: %v \n",guestbook.Spec.Replicas)

    guestbook.Status.Replicas = guestbook.Spec.Replicas
    guestbook.Status.ServiceStatus = "Running"

    if err := r.Status().Update(ctx, &guestbook); err != nil {
       return ctrl.Result{}, fmt.Errorf("failed to update Guestbook status: %v", err)
    }

    return ctrl.Result{}, nil
}


// handleGuestbookDeleted处理Guestbook资源被删除的情况，删除对应的Deployment
func (r *GuestbookReconciler) handleGuestbookDeleted(existedDeployment *appsv1.Deployment,ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

    if (existedDeployment != nil && existedDeployment.Name != "") {
       err := r.Delete(ctx, existedDeployment)
       if err != nil {
          log.FromContext(ctx).Error(err, "failed to delete Deployment")
          return ctrl.Result{}, err
       }
       return ctrl.Result{}, nil
    }

    //logger := log.FromContext(ctx)
     获取当前已存在的Deployment对象（如果有的话）
    //existingDeployment := &appsv1.Deployment{}
    //
    //err := r.Get(ctx, client.ObjectKey{
    // Namespace: req.Namespace,
    // Name:      fmt.Sprintf("%s-deployment", req.Name),
    // Name:      "guestbook-sample-deployment",
    //}, existingDeployment)
    //
    logger.Error(err, "handleGuestbookDeleted err is not null")
    //
     If GuestBook does not exist, remove the Deployment if it exists
    //if err == nil {
    //
    // logger.Info("GuestBook resource not found, removing the associated Deployment")
    // err = r.Delete(ctx, existingDeployment)
    // if err != nil {
    //    logger.Error(err, "failed to delete Deployment")
    //    return ctrl.Result{}, err
    // }
    // return ctrl.Result{}, nil
    //} else {
    // logger.Error(err, "handleGuestbookDeleted err is not null")
    // //fmt.Println("handleGuestbookDeleted err is not null")
    //}

    return ctrl.Result{}, nil
}

// buildDeploymentForGuestbook根据Guestbook资源构建期望的Deployment对象
func (r *GuestbookReconciler) buildDeploymentForGuestbook(guestbook *webappv1.Guestbook,req ctrl.Request) *appsv1.Deployment {
    labels := map[string]string{
       "app": guestbook.Name,
    }
    replicas := guestbook.Spec.Replicas
    return &appsv1.Deployment{
       ObjectMeta: metav1.ObjectMeta{
          Name:      fmt.Sprintf("%s-deployment", req.Name),
          Namespace: req.Namespace,
          Labels:    labels,
       },
       Spec: appsv1.DeploymentSpec{
          Replicas: &replicas,
          Selector: &metav1.LabelSelector{
             MatchLabels: labels,
          },
          Template: corev1.PodTemplateSpec{
             ObjectMeta: metav1.ObjectMeta{
                Labels: labels,
             },
             Spec: corev1.PodSpec{
                Containers: []corev1.Container{
                   {
                      Name:  "guestbook-container",
                      Image: guestbook.Spec.Image,
                      Ports: []corev1.ContainerPort{
                         {
                            Name:          "http",
                            ContainerPort: 80,
                         },
                      },
                   },
                },
             },
          },
       },
    }
}

// deploymentsEqual比较两个Deployment对象是否相等（简单比较主要属性）
func (r *GuestbookReconciler) deploymentsEqual(d1, d2 *appsv1.Deployment) bool {
    return d1.Spec.Replicas == d2.Spec.Replicas && d1.Spec.Template.Spec.Containers[0].Image == d2.Spec.Template.Spec.Containers[0].Image
}

// SetupWithManager sets up the controller with the Manager.
func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
       For(&webappv1.Guestbook{}).
       Named("guestbook").
       Complete(r)
}

执行make install安装命令

[root@master guestbook]# make install
/root/crd/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/crd/guestbook/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.my.domain created

验证资源是否存在
执行 kubectl api-resources，可以看到guestbooks资源已经被创建。

[root@master guestbook]# kubectl api-resources

---前面省略---
priorityclasses                   pc           scheduling.k8s.io/v1                   false        PriorityClass
csidrivers                                     storage.k8s.io/v1                      false        CSIDriver
csinodes                                       storage.k8s.io/v1                      false        CSINode
csistoragecapacities                           storage.k8s.io/v1                      true         CSIStorageCapacity
storageclasses                    sc           storage.k8s.io/v1                      false        StorageClass
volumeattachments                              storage.k8s.io/v1                      false        VolumeAttachment
guestbooks                                     webapp.my.domain/v1                    true         Guestbook

创建guestbook资源对象

先执行make run命令，将前面创建的Controller运行起来

[root@master guestbook]# make run
/root/crd/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/crd/guestbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./cmd/main.go
2025-01-09T07:36:11-05:00	INFO	setup	starting manager
2025-01-09T07:36:11-05:00	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2025-01-09T07:36:11-05:00	INFO	Starting EventSource	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook", "source": "kind source: *v1.Guestbook"}
2025-01-09T07:36:11-05:00	INFO	Starting Controller	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook"}
2025-01-09T07:36:11-05:00	INFO	Starting workers	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook", "worker count": 1}

创建刚定义的CRD资源
kubebuilder已经生成了一个yaml文件样例
/config/samples/webapp_v1_guestbook.yaml

修改下相关参数

apiVersion: webapp.my.domain/v1
kind: Guestbook
metadata:
  labels:
    app.kubernetes.io/name: guestbook
    app.kubernetes.io/managed-by: kustomize
  name: guestbook-sample
spec:
  replicas: 2
  image: nginx

执行创建命令

[root@master guestbook]# kubectl apply -k config/samples
guestbook.webapp.my.domain/guestbook-sample created
# 验证是否创建成功
[root@master guestbook]# kubectl get guestbook
NAME               AGE
guestbook-sample   17s
[root@master guestbook]# kubectl get deployment
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
guestbook-sample-deployment   2/2     2            2           22s
tomcat-deployment             2/2     2            2           17d
wordpress                     0/1     1            0           12d
# 两个Nginx Pod已经被创建
[root@master guestbook]# kubectl get pod
NAME                                           READY   STATUS    RESTARTS     AGE
guestbook-sample-deployment-654b67bc8f-kkpdr   1/1     Running   0            25s
guestbook-sample-deployment-654b67bc8f-sf6t5   1/1     Running   0            25s

# 如果要查看guestbook更详细信息，执行
kubectl get guestbook -o yaml

如果需要删除 guestbook资源类型，可以执行

kubectl delete -k config/samples