Ceph对象存储系统部署
存储系统：ceph-14.2.22
操作系统：ubuntu-server-16.04.07
对象存储集群部署
创建rgw服务
[ ceph-deploy --overwrite-conf rgw create 主机名 ]
ceph-deploy --overwrite-conf rgw create node
配置rgw
默认情况下rgw采用Civetweb作为网络服务，端口为7480。一般情况下修改为80端口，也可以不修改；
[client.rgw.node]
rgw_frontends = "civetweb port=80"
client.rgw.node为rados用户名字，用户类型为rgw，node为rgw的名字；可以通过 ceph auth ls 命令获取；
为了使修改生效，需要重启rgw服务；
systemctl restart ceph-radosgw.target
关闭防火墙
systemctl stop firewalld.service
测试访问
[ curl http://客户端主机名:80 或者curl http://客户端ip:80 ]
curl http:/ /node: 80
-----------------------------------------------------------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<0wner>
<ID>anonymous</ID>
<DisplayName></DisplayName>
</0wner>
<Buckets></Buckets>
</ListALLMyBucketsResult>
使用对象存储
创建s3用户
[ radosgw-admin user create --uid=用户名 --display-name=对外显示的用户名 ]
radosgw-admin user create --uid=s3user --display-name=s3user
需要记住上面的s3user密钥：权限密钥和安全密钥，后面配置s3cmd时需要用到；
安装s3cmd命令行工具
apt install s3cmd
配置s3cmd
s3cmd --configure
-----------------------------------------------------------------------------------------------------------------------
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key:
Secret Key:
Default Region [US] :
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all Communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes] : no
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name :
New settings :
Access Key:
Secret Key:
Default Region: US
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
修改s3配置文件
s3的配置文件放在/root/.s3cfg中，主要修改host_base和host_bucket，需要配置为rgw服务端的ip地址；
gpg_passphrase =
guess_mime_type = True
host_base = 192.168.3.11
host_bucket = 192.168.3.11
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
创建bucket
[ s3cmd mb s3://桶的名字 ]
s3cmd mb s3://bucket1
上传文件
[ s3cmd put 文件名 s3://桶的名字 ]
s3cmd put abc s3://bucket1
查看集群I/O读写状态
ceph -s
-----------------------------------------------------------------------------------------------------------------------
cluster :
id: e7406bb4-35c7-420b-a04a-991510c7f143
health: HEALTH 0K
services :
mon: 1 daemons, quorum node (age 51m)
mgr: node (active, since 60m)
mds: cephfs:1 {0=node=up:active}
osd: 3 osds: 3 up (since 58m)， 3 in (since 58m)
data:
pools: 2 pools, 2 pgs
objects: 23 objects, 7.4 KiB
usage: 3.0 GiB used, 27 GiB / 30 GiB avail
pgs : 2 active+clean
io:
client: 426 B/s wr, 0 op/s rd, 0 op/s wr
查看数据池存储状态
ceph df
-----------------------------------------------------------------------------------------------------------------------
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 30 GiB 27 GiB 122 MiB 3.1 MiB 10.40
TOTAL 30 GiB 27 GiB 122 MiB 3.1 MiB 10.40
POOLS:
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.rgw.root 1 32 1.2 KiB 4 768 KiB 0 8.5 GiB
default.rgw.control 2 32 0 B 8 0 B 0 8.5 GiB
default.rgw.meta 3 32 805 B 5 768 KiB 0 8.5 GiB
default.rgw.log 4 32 0 B 175 0 B 0 8.5 GiB
default.rgw.buckets.index 5 32 0 B 1 0 B 0 8.5 GiB
default.rgw.buckets.data 6 32 4 B 1 192 KiB 0 8.5 GiB
查看数据池存储中的对象
rados -p default.rgw.buckets.data ls
-----------------------------------------------------------------------------------------------------------------------
e763329b-e838-4633-8640-6fb8e35161f4.34178.2 abc