关于k8s的存储知识,大家可以参考博文应用存储和持久化数据卷:核心知识和Kubernetes 存储架构及插件使用,我这就不科普了。阿里云官网上的文档都是基于阿里云自己的k8s的,找了半天资料也没有文章好好介绍一下自己通过kubeadm搭建的k8s集群该怎么用阿里云的oss作为pv/pvc使用,今天这篇文章就是我踩坑之后写下来作为记录的。
说明:
1、k8s集群,我是通过kubeadm搭建的三节点集群。搭建步骤参见我的另外一篇博文:搭建k8s多节点集群
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready master 51d v1.16.2 10.53.5.94 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://19.3.4
node2 Ready 51d v1.16.2 10.53.6.185 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://19.3.4
node3 Ready 51d v1.16.2 10.53.7.37 Ubuntu 16.04.1 LTS 4.4.0-127-generic docker://19.3.4
2、阿里云oss账号,并创建bucket。这个没什么好说的,参见阿里云OSS官方文档
3、在每台机器上安装阿里云ossfs软件。这个软件必须要安装,因为pv/pvc如果想要用阿里云的oss的话,这是必须的软件。安装步骤参见阿里云oss安装文档,我的机器是三节点ubuntu 16.04,下边我就写一下ubuntu 16.04该怎么安装ossfs。
# 下载安装包
$ wget http://gosspublic.alicdn.com/ossfs/ossfs_1.80.6_ubuntu16.04_amd64.deb
#安装
$ sudo apt-get update
$ sudo apt-get install gdebi-core
#sudo gdebi
$ sudo gdebi ossfs_1.80.6_ubuntu16.04_amd64.deb
1.rbac.yaml
# This YAML file contains all RBAC objects that are necessary to run external
# CSI provisioner.
#
# In production, each CSI driver deployment has to be customized:
# - to avoid conflicts, use non-default namespace and different names
# for non-namespaced entities like the ClusterRole
# - decide whether the deployment replicates the external CSI
# provisioner, in which case leadership election must be enabled;
# this influences the RBAC setup, see below
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
# replace with the same namespace name with plugin
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["create", "list", "watch", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
subjects:
- kind: ServiceAccount
name: admin
namespace: kube-system
roleRef:
kind: ClusterRole
name: alicloud-csi-plugin
apiGroup: rbac.authorization.k8s.io
2.oss-plugin.yaml
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: ossplugin.csi.alibabacloud.com
spec:
attachRequired: false
---
# This YAML defines all API objects to create RBAC roles for csi node plugin.
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-ossplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-ossplugin
template:
metadata:
labels:
app: csi-ossplugin
spec:
tolerations:
- operator: Exists
priorityClassName: system-node-critical
serviceAccount: admin
hostNetwork: true
hostPID: true
containers:
- name: driver-registrar
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.1.0
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/ossplugin.csi.alibabacloud.com /registration/ossplugin.csi.alibabacloud.com-reg.sock"]
args:
- "--v=5"
- "--csi-address=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
- name: registration-dir
mountPath: /registration
- name: csi-ossplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8.32-c77e277b-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--driver=ossplugin.csi.alibabacloud.com"
- "--nodeid=$(KUBE_NODE_NAME)"
env:
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins/ossplugin.csi.alibabacloud.com/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
mountPropagation: "Bidirectional"
- name: etc
mountPath: /host/etc
- mountPath: /var/log/
name: host-log
- mountPath: /host/usr/
name: flexvolumedir
volumes:
- name: kubelet-dir
hostPath:
path: /var/lib/kubelet/
type: Directory
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
- name: etc
hostPath:
path: /etc
- name: flexvolumedir
hostPath:
path: /usr/
- name: host-log
hostPath:
path: /var/log/
updateStrategy:
type: RollingUpdate
3.pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: oss-csi-pv
labels:
alicloud-pvname: oss-csi-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: ossplugin.csi.alibabacloud.com
# set volumeHandle same value pv name
volumeHandle: oss-csi-pv
volumeAttributes:
bucket: "*****" #重要
url: "******" #重要
otherOpts: "-o max_stat_cache_size=0 -o allow_other"
akId: "****" #重要
akSecret: "*******" #重要
path: "/"
说明:
-o *** -o ***
。
4.pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: oss-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
alicloud-pvname: oss-csi-pv
5. deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-oss
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: oss-pvc
mountPath: "/data"
volumes:
- name: oss-pvc
persistentVolumeClaim:
claimName: oss-pvc
至此,部署阶段就比较简单了。按照上边准备的文件顺序知行yaml文件:
#创建rbac权限
$ kubectl create -f ./rbac.yaml
serviceaccount/admin created
clusterrole.rbac.authorization.k8s.io/alicloud-csi-plugin created
clusterrolebinding.rbac.authorization.k8s.io/alicloud-csi-plugin created
#创建oss-plugin
$ kubectl create -f ./oss-plugin.yaml
#检查创建情况
$ kubectl get pod -n kube-system | grep csi-oss
kube-system csi-ossplugin-9jdhw 2/2 Running 0 55m
kube-system csi-ossplugin-f7n5f 2/2 Running 0 55m
kube-system csi-ossplugin-vgkcp 2/2 Running 0 55m
#查验CSIDriver安装情况
$ kubectl get CSIDriver
NAME CREATED AT
ossplugin.csi.alibabacloud.com 2020-06-23T14:48:18Z
#创建pv
$ kubectl create -f ./pv.yaml
#创建pvc
$ kubectl create -f ./pvc.yaml
#检验一下阿里云oss是否可以成功挂载到k8s集群中做pv使用
$ kubectl create -f ./deploy.yaml
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-oss-795894886d-lhpsx 1/1 Running 0 11h
#pod成功后通过kubectl exec 进入到pod中,你能看到你账号下bucket里边的所有文件。样例如下:
$ kubectl exec -it deployment-oss-795894886d-lhpsx -- sh
$ ls
bin boot data dev etc home lib lib64 media mnt opt proc root run sbin selinux srv sys tmp usr var
$ cd data
$ ls
osstest
1、K8S有状态服务-OSS存储使用最佳实践;
2、阿里云oss CSI安装步骤
3、阿里云oss官方文档