Rook下快速部署ceph分布式文件系统

由于CRD资源版本不匹配的问题，参考下面文章需要切换到0.9分支
https://github.com/rook/rook/issues/2338
$ git clone https://github.com/rook/rook.git   #克隆仓库
$ cd rook/cluster/examples/kubernetes/ceph   
$ git checkout remotes/origin/release-0.9   #切换到0.9的git分支
$ git branch -a   #显示所有分支
$ kubectl create -f operator.yaml   #创建rook-ceph-operator，然后由rook-ceph-operator创建rook-ceph-agent和rook-discover
$ kubectl get pods --all-namespaces #查看所有namespace
$ kubectl create -f cluster.yaml  #部署ceph cluster
$ kubectl -n rook-ceph get pod  #查看部署完成后的pod,这里可以看到有一个mgr,三个monitor,三个osd
NAME                                        READY   STATUS      RESTARTS   AGE
rook-ceph-mgr-a-68cb58b456-kv5j8            1/1     Running     0          7h51m
rook-ceph-mon-a-548b988957-2jc7r            1/1     Running     0          7h56m
rook-ceph-mon-d-bf54cdbf6-dw8j8             1/1     Running     0          7h55m
rook-ceph-mon-f-7fd45445fc-xbx57            1/1     Running     0          7h53m
rook-ceph-osd-0-7547b99fd8-swm8v            1/1     Running     0          7h51m
rook-ceph-osd-1-7b484f8469-glh6t            1/1     Running     0          7h51m
rook-ceph-osd-2-66d7b56965-5jsfr            1/1     Running     0          7h49m
rook-ceph-osd-prepare-192-168-60-13-w6k8l   0/2     Completed   0          7h51m
rook-ceph-osd-prepare-192-168-60-14-tpckx   0/2     Completed   0          7h51m
rook-ceph-osd-prepare-192-168-60-15-g9p6r   0/2     Completed   0          7h51m
rook-ceph-tools-bffbf4d8f-j95t4             1/1     Running     0          6h27m

$ kubectl -n rook-ceph-system get pod  #查看agent和discover全部已经running
NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-agent-8lpd6                1/1     Running   0          23h
rook-ceph-agent-blsgj                1/1     Running   0          23h
rook-ceph-agent-jjzsx                1/1     Running   0          23h
rook-ceph-operator-d97564799-rw7nl   1/1     Running   0          23h
rook-discover-k4dhn                  1/1     Running   0          23h
rook-discover-nhpvl                  1/1     Running   0          23h
rook-discover-x5z94                  1/1     Running   0          23h


$ kubectl get cephcluster -n rook-ceph
NAME        DATADIRHOSTPATH   MONCOUNT   AGE     STATE
rook-ceph   /var/lib/rook     3          7h59m   Created

CRD参数见如下
https://rook.github.io/docs/rook/v0.9/ceph-cluster-crd.html

$ kubectl get svc -n rook-ceph |grep mgr-dashboard #配置dashboard,可见是集群内部IP，只能集群内部访问
rook-ceph-mgr-dashboard   ClusterIP   10.111.188.167           8443/TCP   61m
$ kubectl create -f dashboard-external-https.yaml #使用NodePort把端口暴露出来
service/rook-ceph-mgr-dashboard-external-https created
$ kubectl get svc -n rook-ceph |grep mgr-dashboard   #可见可以通过31768可以访问了
rook-ceph-mgr-dashboard                  ClusterIP   10.111.188.167           8443/TCP         63m
rook-ceph-mgr-dashboard-external-https   NodePort    10.107.79.213            8443:31768/TCP   13s

$ kubectl cluster-info |grep master #查看master运行在那一台服务器
Kubernetes master is running at https://192-168-60-13:6443

部署toolbox
$ kubectl create -f toolbox.yaml
deployment.apps/rook-ceph-tools created
$ kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rook-ceph-tools-bffbf4d8f-j95t4 1/1 Running 0 7s 192.168.60.14 192-168-60-14  


获取密码的两种方式
$ kubectl -n rook-ceph logs rook-ceph-mgr-a-68cb58b456-kv5j8 | grep password  #获取admin的密码
2019-05-06 01:42:36.295 7f4770700700  0 log_channel(audit) log [DBG] : from='client.4133 10.254.184.3:0/1942469392' entity='client.admin' cmd=[{"username": "admin", "prefix": "dashboard set-login-credentials", "password": "KumZw7KvNA", "target": ["mgr", ""], "format": "json"}]: dispatch

$kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}'  |  base64 --decode  #同样可以获取密码