概述
我们知道ceph是统一的分布式存储系统，其可以提供给应用三种访问数据的接口：对象(RGW)、块(RBD)和文件(CEPHFS)接口。我们在使用对象接口的时候通常都是基于HTTP协议来存取数据。
下面介绍另外一种方式来使用ceph的对象(RGW)接口 -- nfs-ganesha。这种方式可以通过基于文件的访问协议（如NFSv3和NFSv4）来访问Ceph对象网关命名空间。详细的介绍可以去ceph官网看看：http://docs.ceph.com/docs/master/radosgw/nfs/。
1、环境准备
1.1、准备虚拟机
单纯为了测试可用性，所以我这里就是使用一个虚拟机搭建ceph集群，然后配置nfs-ganesha将rgw导出为nfs文件接口
[root@ceph01 tmp]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@ceph01 tmp]# uname -a
Linux ceph01 4.17.3-1.el7.elrepo.x86_64 #1 SMP Tue Jun 26 10:14:25 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
1.2、配置yum源
[root@ceph01 tmp]# ll /etc/yum.repos.d/
total 12
-rw-r--r--. 1 root root 2523 Jun 16 06:22 CentOS-Base.repo
-rw-r--r--. 1 root root 180 Jul 3 11:38 ceph.repo
-rw-r--r--. 1 root root 664 May 11 11:35 epel.repo
1.2.1、base源
[root@ceph01 tmp]# cat /etc/yum.repos.d/CentOS-Base.repo
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
1.2.2、epel源
[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
1.2.3、ceph源
我这里配置的Jewel的ceph源
[root@ceph01 tmp]# cat /etc/yum.repos.d/ceph.repo
[ceph]
name=ceph
baseurl=http://download.ceph.com/rpm-jewel/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch/
gpgcheck=0
2、相关软件包准备
2.1、安装ceph相关包
注意这里的 librgw2-devel 包后面编译nfs-ganesha时需要，所以一定要安装
[root@ceph01 tmp]# yum install ceph librgw2-devel -y
[root@ceph01 tmp]# ceph -v
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
部署ceph集群，下面是我部署好的单机ceph集群
[root@ceph01 tmp]# ceph -s
cluster 29978a95-87f1-4a0c-96fa-2369e985bfbe
health HEALTH_OK
monmap e1: 1 mons at {ceph01=192.168.10.20:6789/0}
election epoch 4, quorum 0 ceph01
osdmap e26: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v61: 128 pgs, 9 pools, 2390 bytes data, 177 objects
217 MB used, 30480 MB / 30697 MB avail
128 active+clean
2.2、编译安装nfs-ganesha
下载nfs-ganesha源代码：https://github.com/nfs-ganesha/nfs-ganesha/tree/3cf14c95d0a6efb5d366f97333d07a08d6f3b2a5
下载nfs-ganesha依赖的模块：https://github.com/nfs-ganesha/ntirpc/tree/5575ddb022b996c318d263e504273ee6a34499a3
因为我使用的ceph是Jewel 10.2.11，所以这里下载的nfs-ganesha是V2.5.0，高版本或低版本都会有点问题。
将上面下载好的两个包放到/tmp目录下
[root@ceph01 tmp]# ll
···
-rw-r--r--. 1 root root 3936969 Jul 3 15:08 nfs-ganesha.zip
-rw-r--r--. 1 root root 632189 Jul 3 15:08 ntirpc.zip
解压
[root@ceph01 tmp]# unzip nfs-ganesha.zip
[root@ceph01 tmp]# unzip ntirpc.zip
因为ntirpc是nfs-ganesha所依赖的子模块，所以这里将ntirpc源代码放到nfs-ganasha对应的目录中去
[root@ceph01 tmp]# cp -r ntirpc/* nfs-ganesha/src/libntirpc/
编译安装nfs-nanesha
[root@ceph01 tmp]# mkdir mybuild && cd mybuild
[root@ceph01 mybuild]# cmake -DUSE_FSAL_RGW=ON /tmp/nfs-ganesha/src/
[root@ceph01 mybuild]# make && make install
如果有报错，根据报错信息去处理下，每个环境配置不一样，报错可能不一样。
3、开始配置
3.1、准备rgw用户
[root@ceph01 mybuild]# radosgw-admin user create --uid=testuser --display-name="testuser " --access-key=testuser --secret=testuser
3.2、准备nfs-ganesha配置文件
配置起来还是比较简单的（不过是最简配置··· 哈哈）
[root@ceph01 mybuild]# cat /etc/ganesha/ganesha.conf
EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
NFS_Protocols = 4;
Transport_Protocols = TCP;
FSAL {
# 下面的值就是开始创建的rgw用户信息
Name = RGW;
User_Id = "testuser";
Access_Key_Id ="testuser";
Secret_Access_Key = "testuser";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
}
3.3、启动nfs-ganesha服务
[root@ceph01 mybuild]# ganesha.nfsd -L /var/log/ganesha.log
3.4、查看nfs-ganesha服务是否正常启动
[root@ceph01 tmp]# ps aux|grep ganesha
root 16055 0.2 1.0 5222380 42292 ? Ssl 15:22 0:06 ganesha.nfsd -L /var/log/ganesha.log
可以看到nfs-ganesha服务已经正常启动了
4、使用
现在来到另外一台服务器上面
4.1、安装nfs-utils
[root@ceph07 ~]# yum install -y nfs-utils
4.2、挂载nfs
这里的192.168.10.20就是我们上面的服务器ip
[root@ceph07 ~]# mount -t nfs4 192.168.10.20:/ /mnt/
[root@ceph07 ~]# mount |grep mnt
192.168.10.20:/ on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.32,local_lock=none,addr=192.168.10.20)
4.3、使用nfs客户端挂载目录
在创建之前查看ceph rgw池里面的对象数
[root@ceph01 tmp]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
30697M 30480M 217M 0.71
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 14472M 0
.rgw.root 1 1588 0 14472M 4
default.rgw.control 2 0 0 14472M 8
default.rgw.data.root 3 0 0 14472M 0
default.rgw.gc 4 0 0 14472M 32
default.rgw.log 5 0 0 14472M 127
default.rgw.users.uid 6 202 0 14472M 1
default.rgw.users.keys 7 9 0 14472M 1
可以看到现在rgw data池里面的对象数是0个，现在在nfs挂载目录下创建一个目录试试
[root@ceph07 ~]# mkdir /mnt/bk001
再次查看rgw data池里面的对象数
[root@ceph01 ceph01 ]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
30697M 30480M 217M 0.71
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 14472M 0
.rgw.root 1 1588 0 14472M 4
default.rgw.control 2 0 0 14472M 8
default.rgw.data.root 3 591 0 14472M 2
default.rgw.gc 4 0 0 14472M 32
default.rgw.log 5 0 0 14472M 127
default.rgw.users.uid 6 202 0 14472M 2
default.rgw.users.keys 7 9 0 14472M 1
default.rgw.buckets.index 8 0 0 14472M 1
可以看到rgw data池里面的对象数已经由0增加到2了，其实这里的目录bk001就是对应到rgw里面的bucket
总结
我们上面主要做了这些步骤：
- 安装ceph并搭建ceph集群
- 使用radosgw-admin创建rgw用户（在创建rgw用户的时候，会自动创建rgw所需的存储池）
- 获取nfs-ganesha及其依赖的ntirpc模块
- 编译、安装和配置nfs-ganesha
- 最后启动nfs-ganesha服务并在其客户端挂载、测试使用