JuiceFS-K8s部署
目录
-
- 1、部署JuiceFS-CSI驱动
- 2、创建OBS认证信息Secret
- 3、创建存储类
- 4、创建PVC--PVC创建时会自动创建PV
- 5、创建测试Pod--测试Pod创建容器内是否挂载成功
官网文档地址：https://juicefs.com/docs/zh/csi/introduction/
1、部署JuiceFS-CSI驱动
部署yaml如下，镜像已进行更改为镜像仓库内地址
# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-controller-sa
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-node-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-external-node-service-role
rules:
- apiGroups:
- ""
resources:
- pods
- pods/log
- secrets
verbs:
- get
- list
- create
- update
- delete
- patch
- watch
- apiGroups:
- batch
resources:
- jobs
verbs:
- get
- list
- create
- update
- delete
- patch
- watch
- apiGroups:
- ""
resources:
- nodes/proxy
verbs:
- '*'
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-external-provisioner-role
rules:
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- watch
- create
- delete
- apiGroups:
- ""
resources:
- persistentvolumeclaims
verbs:
- get
- list
- watch
- update
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- get
- list
- watch
- create
- update
- patch
- apiGroups:
- storage.k8s.io
resources:
- csinodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- pods
- pods/log
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- batch
resources:
- jobs
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-node-service-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccount
name: juicefs-csi-node-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-provisioner-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccount
name: juicefs-csi-controller-sa
namespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: juicefs-csi-controller
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
serviceName: juicefs-csi-controller
template:
metadata:
labels:
app: juicefs-csi-controller
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
spec:
containers:
- args:
- --endpoint=$(CSI_ENDPOINT)
- --logtostderr
- --nodeid=$(NODE_NAME)
- --v=5
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: JUICEFS_MOUNT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBELET_PORT
value: "10250"
- name: JUICEFS_MOUNT_PATH
value: /var/lib/juicefs/volume
- name: JUICEFS_CONFIG_PATH
value: /var/lib/juicefs/config
image: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
name: juicefs-plugin
ports:
- containerPort: 9909
name: healthz
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
securityContext:
capabilities:
add:
- SYS_ADMIN
privileged: true
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- mountPath: /jfs
mountPropagation: Bidirectional
name: jfs-dir
- mountPath: /root/.juicefs
mountPropagation: Bidirectional
name: jfs-root-dir
- args:
- --csi-address=$(ADDRESS)
- --timeout=60s
- --v=5
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: harbor.t3caic.com/juicefs/csi-provisioner:v1.6.0
name: csi-provisioner
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=$(ADDRESS)
- --health-port=$(HEALTH_PORT)
env:
- name: ADDRESS
value: /csi/csi.sock
- name: HEALTH_PORT
value: "9909"
image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0
name: liveness-probe
volumeMounts:
- mountPath: /csi
name: socket-dir
priorityClassName: system-cluster-critical
serviceAccount: juicefs-csi-controller-sa
tolerations:
- key: CriticalAddonsOnly
operator: Exists
volumes:
- emptyDir: {}
name: socket-dir
- hostPath:
path: /var/lib/juicefs/volume
type: DirectoryOrCreate
name: jfs-dir
- hostPath:
path: /var/lib/juicefs/config
type: DirectoryOrCreate
name: jfs-root-dir
volumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: node
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: juicefs-csi-node
namespace: kube-system
spec:
selector:
matchLabels:
app: juicefs-csi-node
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
template:
metadata:
labels:
app: juicefs-csi-node
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
spec:
containers:
- args:
- --endpoint=$(CSI_ENDPOINT)
- --logtostderr
- --nodeid=$(NODE_NAME)
- --v=5
- --enable-manager=true
env:
- name: CSI_ENDPOINT
value: unix:/csi/csi.sock
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: JUICEFS_MOUNT_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBELET_PORT
value: "10250"
- name: JUICEFS_MOUNT_PATH
value: /var/lib/juicefs/volume
- name: JUICEFS_CONFIG_PATH
value: /var/lib/juicefs/config
image: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- rm /csi/csi.sock
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
name: juicefs-plugin
ports:
- containerPort: 9909
name: healthz
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/kubelet
mountPropagation: Bidirectional
name: kubelet-dir
- mountPath: /csi
name: plugin-dir
- mountPath: /dev
name: device-dir
- mountPath: /jfs
mountPropagation: Bidirectional
name: jfs-dir
- mountPath: /root/.juicefs
mountPropagation: Bidirectional
name: jfs-root-dir
- args:
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
- --v=5
env:
- name: ADDRESS
value: /csi/csi.sock
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock
image: harbor.t3caic.com/juicefs/csi-node-driver-registrar:v2.1.0
name: node-driver-registrar
volumeMounts:
- mountPath: /csi
name: plugin-dir
- mountPath: /registration
name: registration-dir
- args:
- --csi-address=$(ADDRESS)
- --health-port=$(HEALTH_PORT)
env:
- name: ADDRESS
value: /csi/csi.sock
- name: HEALTH_PORT
value: "9909"
image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0
name: liveness-probe
volumeMounts:
- mountPath: /csi
name: plugin-dir
dnsPolicy: ClusterFirstWithHostNet
priorityClassName: system-node-critical
serviceAccount: juicefs-csi-node-sa
tolerations:
- key: CriticalAddonsOnly
operator: Exists
volumes:
- hostPath:
path: /var/lib/kubelet
type: Directory
name: kubelet-dir
- hostPath:
path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/
type: DirectoryOrCreate
name: plugin-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry/
type: Directory
name: registration-dir
- hostPath:
path: /dev
type: Directory
name: device-dir
- hostPath:
path: /var/lib/juicefs/volume
type: DirectoryOrCreate
name: jfs-dir
- hostPath:
path: /var/lib/juicefs/config
type: DirectoryOrCreate
name: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
labels:
app.kubernetes.io/instance: juicefs-csi-driver
app.kubernetes.io/name: juicefs-csi-driver
app.kubernetes.io/version: master
name: csi.juicefs.com
spec:
attachRequired: false
podInfoOnMount: false
2、创建OBS认证信息Secret
需要修改的信息
metaurl: “用户名:密码@IP:端口/库名”
postgres://用户名:密码@IP:端口/库名 数据库需要提前进入进行创建库
storage: “obs” 存储类型标识
bucket: “” 桶的域名
access-key: “”
secret-key: “”
apiVersion: v1
kind: Secret
metadata:
name: juicefs-sc-secret
namespace: kube-system
type: Opaque
stringData:
name: "test"
metaurl: ""
storage: ""
bucket: ""
access-key: ""
secret-key: ""
3、创建存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
4、创建PVC–PVC创建时会自动创建PV
目前而言，JuiceFS CSI 驱动不支持设置存储容量。在 PersistentVolume 和 PersistentVolumeClaim 中指定的容量会被忽略，填写任意有效值即可，例如 10Gi
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "juicefs-test-pvc"
namespace: test-juicefs
spec:
storageClassName: sc-topolvm
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: juicefs-sc
5、创建测试Pod–测试Pod创建容器内是否挂载成功
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-run
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: harbor.t3caic.com/nginxdemos/hello:plain-text
ports:
- containerPort: 80
volumeMounts:
- mountPath: /config
name: web-data
volumes:
- name: web-data
persistentVolumeClaim:
claimName: juicefs-test-pvc
ERR注意：如果测试Pod创建卡死，查看kube-system命名空间的juicefs-XXX-pvc容器起来了吗
如因为拉取不下镜像，请在此容器所在节点手动解决Pod拉取 docker pull
如果镜像拉取下来Pod一直在重启可以考虑一下是否是OBS连接不上，如果OBS需要做域名解析可以在CoreDNS里配置
kubectl edit configmap coredns -n kube-system
添加
hosts {
100.125.32.189 juicefs.obs.cn-east-263.t3caic.com
fallthrough
}