- 0. 下载kubebuilder
- 1. 创建目录
- 2. 初始化项目
- 3. 创建api和controller
- 4. 实现自己的crd和控制器逻辑
- 5. make manifests, 创建crd的相关yaml
- 6. 在集群中部署crd
- 7. 部署controller
简介
从0到1，手把手教会如何使用kubebuilder创建crd, 并且定制自己的控制器。
代码：https://github.com/zoux86/operator-example
0. 下载kubebuilder
# download kubebuilder and install locally.
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
1. 创建目录
~/go/src 是我的go src目录
github.com/zoux86/operator-example是想自定义的crd项目
// 可以看出来go mod init 指定的字符串就是mod文件里面的module目录
~/go/src/github.com/zoux86/operator-example# go mod init github.com/zoux86/operator-example
go: creating new go.mod: module github.com/zoux86/operator-example
~/go/src/github.com/zoux86/operator-example # ls
go.mod
~/go/src/github.com/zoux86/operator-example # cat go.mod
module github.com/zoux86/operator-example
go 1.18
2. 初始化项目
执行kubebuilder init这一条命令就行了
~/go/src/github.com/zoux86/operator-example # ~/kubebuilder init --domain github.com --license apache2 --owner "zoux86"
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/[email protected]
go: downloading sigs.k8s.io/controller-runtime v0.11.2
go: downloading k8s.io/apimachinery v0.23.5
go: downloading k8s.io/client-go v0.23.5
go: downloading k8s.io/utils v0.0.0-20211116205334-6203023598ed
go: downloading k8s.io/component-base v0.23.5
go: downloading k8s.io/api v0.23.5
go: downloading k8s.io/apiextensions-apiserver v0.23.5
go: downloading sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go: downloading golang.org/x/net v0.0.0-20211209124913-491a49abca63
go: downloading golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
Update dependencies:
$ go mod tidy
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
Next: define a resource with:
$ kubebuilder create api
查看文件目录
k8s apis通常有三个组件Resource, Controller, Manager，它们分别定义/实现在以下的三个package当中：
- cmd/...：主流程程序
Manager入口，负责初始化依赖包、启停Controller。用户通常不需要编辑此包，可以依赖脚手架。通过kubebuilder init自动创建生成。 - pkg/apis/...：包含API资源的定义。编辑
*_types.go文件来修改资源定义。每个资源的定义文件存在于pkg/apis/中。通过/ / _types.go kubebuilder create api自动创建生成。 - pkg/controller/...：包含Controller的实现。编辑
*_controller.go实现Controller。通过kubebuilder create api自动创建生成。
~/go/src/github.com/zoux86/operator-example tree
.
├── Dockerfile
├── Makefile
├── PROJECT
├── README.md
├── config
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ └── manager_config_patch.yaml
│ ├── manager
│ │ ├── controller_manager_config.yaml
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ └── rbac
│ ├── auth_proxy_client_clusterrole.yaml
│ ├── auth_proxy_role.yaml
│ ├── auth_proxy_role_binding.yaml
│ ├── auth_proxy_service.yaml
│ ├── kustomization.yaml
│ ├── leader_election_role.yaml
│ ├── leader_election_role_binding.yaml
│ ├── role_binding.yaml
│ └── service_account.yaml
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go
3. 创建api和controller
其实从create api 后的输出我们可以看出来：我们修改逻辑后就可以部署了
~/go/src/github.com/zoux86/operator-example # ~/kubebuilder create api --group zouxapp --kind PodCount --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit... // 先修改这2个文件
Writing scaffold for you to edit...
api/v1/podcount_types.go
controllers/podcount_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
mkdir -p /Users/game-netease/go/src/github.com/zoux86/operator-example/bin
GOBIN=/Users/game-netease/go/src/github.com/zoux86/operator-example/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
go: downloading sigs.k8s.io/controller-tools v0.8.0
go: downloading github.com/spf13/cobra v1.2.1
go: downloading golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go: downloading github.com/fatih/color v1.12.0
go: downloading k8s.io/api v0.23.0
go: downloading k8s.io/apimachinery v0.23.0
go: downloading github.com/gobuffalo/flect v0.2.3
go: downloading k8s.io/apiextensions-apiserver v0.23.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go: downloading golang.org/x/mod v0.4.2
/Users/game-netease/go/src/github.com/zoux86/operator-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
执行create api后，生成以下文件：
api/v1/groupversion_info.go
api/v1/podcount_types.go // 需要修改这个文件中crd的定义
api/v1/zz_generated.deepcopy.go
config/crd/kustomization.yaml
config/crd/kustomizeconfig.yaml
config/crd/patches/cainjection_in_podcounts.yaml
config/crd/patches/webhook_in_podcounts.yaml
config/rbac/podcount_editor_role.yaml
config/rbac/podcount_viewer_role.yaml
config/samples/zouxapp_v1_podcount.yaml
controllers/podcount_controller.go // 需要修改这个文件的controller运行逻辑
controllers/suite_test.go
go.mod
main.go
4. 实现自己的crd和控制器逻辑
根据实际情况而定，这里的控制器逻辑很简单，就是创建同步podCount的spec.count到status里面。
func (r *PodCountReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
rlog := log.FromContext(ctx)
rlog.Info("start to reconciling podCount %s", req.Name)
podCount := &zouxappv1.PodCount{}
err := r.Client.Get(ctx, req.NamespacedName, podCount)
if err != nil {
rlog.Error(err, fmt.Sprintf("get podcount %s/%s err during reconcile.", req.Namespace, req.Name))
return ctrl.Result{}, nil
}
podCountCopy := podCount.DeepCopy()
if podCount.Spec.Count <= 0 {
podCountCopy.Status.Count = 0
} else {
podCountCopy.Status.Count = podCount.Spec.Count
}
err = r.Client.Status().Update(ctx, podCountCopy)
if err != nil {
rlog.Error(err, fmt.Sprintf("update crd podcount status error %s/%s during reconcile.", req.Namespace, req.Name))
}
//r.Status().Update(ctx, podCountCopy, metav1.UpdateOptions{})
// TODO(user): your logic here
return ctrl.Result{}, err
}
5. make manifests, 创建crd的相关yaml
~/go/src/github.com/zoux86/operator-example # make manifests
/Users/game-netease/go/src/github.com/zoux86/operator-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
执行make manifests之后，我们会得到2个文件。
config/crd/bases/
config/rbac/role.yaml
# cat config/rbac/role.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: manager-role
rules:
- apiGroups:
- zouxapp.github.com
resources:
- podcounts
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- zouxapp.github.com
resources:
- podcounts/finalizers
verbs:
- update
- apiGroups:
- zouxapp.github.com
resources:
- podcounts/status
verbs:
- get
- patch
- update
# cat config/crd/bases/zouxapp.github.com_podcounts.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.8.0
creationTimestamp: null
name: podcounts.zouxapp.github.com
spec:
group: zouxapp.github.com
names:
kind: PodCount
listKind: PodCountList
plural: podcounts
singular: podcount
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: PodCount is the Schema for the podcounts API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: PodCountSpec defines the desired state of PodCount
properties:
count:
description: Foo is an example field of PodCount. Edit podcount_types.go
to remove/update
type: integer
type: object
status:
description: PodCountStatus defines the observed state of PodCount
properties:
count:
description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
of cluster Important: Run "make" to regenerate code after modifying
this file'
type: integer
type: object
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
6. 在集群中部署crd
~/go/src/github.com/zoux86/operator-example #kubectl --kubeconfig=kubeconfig get node create -f config/crd/bases
customresourcedefinition.apiextensions.k8s.io/podcounts.zouxapp.github.com created
~/go/src/github.com/zoux86/operator-example # kubectl --kubeconfig=kubeconfig create -f config/samples/zouxapp_v1_podcount.yaml
podcount.zouxapp.github.com/podcount-sample created
上集群验证，可以看到创建成功了，但是可以看出来没有status.count，这个因为集群还没部署控制器
root# kubectl get crd | grep podc
podcounts.zouxapp.github.com 2022-08-25T06:57:09Z
root # kubectl get podcounts.zouxapp.github.com
NAME AGE
podcount-sample 11s
root # kubectl get podcounts.zouxapp.github.com -oyaml
apiVersion: v1
items:
- apiVersion: zouxapp.github.com/v1
kind: PodCount
metadata:
creationTimestamp: "2022-08-25T07:01:16Z"
generation: 1
name: podcount-sample
namespace: default
resourceVersion: "467368378"
selfLink: /apis/zouxapp.github.com/v1/namespaces/default/podcounts/podcount-sample
uid: a8b42a4c-1ebd-430a-890f-b0238f4ad125
spec:
count: 3
kind: List
metadata:
resourceVersion: ""
selfLink: ""
7. 部署controller
之前 CRD 并不会完成任何工作，只是在 ETCD 中创建了一条记录。所以我们需要部署写的controller。
运行CRD controller
~/go/src/github.com/zoux86/operator-example ## go run ./main.go
I0825 15:35:12.827589 63628 request.go:665] Waited for 1.000074041s due to client-side throttling, not priority and fairness, request: GET:https://xxx/apis/apiextensions.k8s.io/v1?timeout=32s
1.6614129137223601e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.6614129137230568e+09 INFO setup starting manager
1.661412913723448e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.661412913723448e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.661412913723506e+09 INFO controller.podcount Starting EventSource {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount", "source": "kind source: *v1.PodCount"}
1.661412913723542e+09 INFO controller.podcount Starting Controller {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount"}
1.661412913825421e+09 INFO controller.podcount Starting workers {"reconciler group": "zouxapp.github.com", "reconciler kind": "PodCount", "worker count": 1}
I0825 15:35:13.825542 63628 podcount_controller.go:50] start to reconciling podCount podcount-sample
I0825 15:35:13.868618 63628 podcount_controller.go:50] start to reconciling podCount podcount-sample
查看发现生效了
root# kubectl get podcounts.zouxapp.github.com -oyaml
apiVersion: v1
items:
- apiVersion: zouxapp.github.com/v1
kind: PodCount
metadata:
creationTimestamp: "2022-08-25T07:01:16Z"
generation: 1
name: podcount-sample
namespace: default
resourceVersion: "467385745"
selfLink: /apis/zouxapp.github.com/v1/namespaces/default/podcounts/podcount-sample
uid: a8b42a4c-1ebd-430a-890f-b0238f4ad125
spec:
count: 3
status:
count: 3
kind: List
metadata:
resourceVersion: ""
selfLink: ""