Ceph radosgw 安装配置

http://blog.csdn.net/changtao381/article/details/48015623

Ceph radosgw对象存储的接口，研究配置了许久，现分享如下。首先配置radosgw的前提，是你已经成功的配置好了Ceph集群，通过ceph –s 查看ceph集群，处于health状态。在这里，ceph集群的auth的配置为none，所以有关auth的部分，也就是访问权限相关的keying部分的配置在这里省略。

1)创建rgw相关的pool 
ceph osd pool create .rgw 128 128 
ceph osd pool create .rgw.root 128 128 
ceph osd pool create .rgw.control 128 128 
ceph osd pool create .rgw.gc 128 128 
ceph osd pool create .rgw.buckets 128 128 
ceph osd pool create .rgw.buckets.index 128 128 
ceph osd pool create .log 128 128 
ceph osd pool create .intent-log 128 128 
ceph osd pool create .usage 128 128 
ceph osd pool create .users 128 128 
ceph osd pool create .users.email 128 128 
ceph osd pool create .users.Swift 128 128 
ceph osd pool create .users.uid 128 128

2）配置 ceph.conf 
这里配置在ceph.conf里添加有个radowgw的配置。radosgw有两种方式运行，一种是直接用civetweb的方式，其内置了一个比较小巧的http服务器mongoose,这种方式配置比较简单，不需要配置Apache httpd服务器，其配置如下：

[client.radosgw.gateway] 
host = node1 
log file = /var/log/ceph/client.radosgw.gateway.log 
rgw_frontends =civetweb port=80 
rgw print continue = false 
特别说明的是civetweb默认的端口是7480, 这里因为用s3cmd, 所以改为80，这里的node1是需要改成你自己的host名字

另一种方式就是 apache httpd + factcgi的方式，其配置如下： 
[client.radosgw.gateway] 
host = node1 
log file = /var/log/ceph/client.radosgw.gateway.log 
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0 
rgw print continue = false

注意，这里的node1，要替换成你自己的服务器 
3)启动 radosgw 
radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway 
主要，这里的-n后跟的名字，是ceph.conf 里配置的。如果出错，可以查看日志/var/log/ceph/client.radosgw.gateway.log的提示，做相应的处理。

4) 配置 httpd 
如果用的是civetweb方式，这一步可以省略。

在目录/etc/httpd/conf.d 里添加文件rgw.conf

<code class="hljs apache has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><VirtualHost *:80></span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">ServerName</span> 182.92.171.199
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">DocumentRoot</span> /var/www/html

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">ErrorLog</span> /var/log/httpd/rgw_error.log
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CustomLog</span> /var/log/httpd/rgw_access.log combined

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">RewriteEngine</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">On</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">RewriteRule</span> .* -<span class="hljs-sqbracket" style="box-sizing: border-box; color: rgb(155, 133, 157);"> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]</span>

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">SetEnv</span> proxy-nokeepalive 1

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">ProxyPass</span> / fcgi://182.92.171.199:9000/
<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></VirtualHost></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

这样这里的ServerName是http server name，如果有多个ip的情况，最好直接写ip，这里182.92.171.199的IP就是node1的ip，这里的ProxyPass 的fcgi 需要改成在ceph.conf里配置的ip和port

5) 启动httpd服务器 
如果用的是civetweb方式，这一步可以省略 
service httpd restart

6）创建一个用户 
radosgw-admin user create –uid=cephtest –display-name=”ceph test” –[email protected] 
显示： 
{ “user_id”: “cephtest”, 
“display_name”: “ceph test”, 
“email”: “[email protected]”, 
“suspended”: 0, 
“max_buckets”: 1000, 
“auid”: 0, 
“subusers”: [], 
“keys”: [ 
{ “user”: “cephtest”, 
“access_key”: “8JGTP1714JRRAPRDLBI4“, 
“secret_key”: “i4O+yJMNii87ruLMwcIIYbjLmKIUaoSO1svQOoB9“}], 
“swift_keys”: [], 
“caps”: [], 
“op_mask”: “read, write, delete”, 
“default_placement”: “”, 
“placement_tags”: [], 
“bucket_quota”: { “enabled”: false, 
“max_size_kb”: -1, 
“max_objects”: -1}, 
“user_quota”: { “enabled”: false, 
“max_size_kb”: -1, 
“max_objects”: -1}, 
“temp_url_keys”: []} 
注意：如果生成的key中有”\”,最后把这个user删掉，再重新生成直到不含”\”：

7）配置s3cmd 
$vim ~/.s3cfg 
[default] 
access_key = PPA789F4W5ANH3COC51O 
bucket_location = US 
cloudfront_host = cloudfront.amazonaws.com 
cloudfront_resource = /2010-07-15/distribution 
default_mime_type = binary/octet-stream 
delete_removed = False 
dry_run = False 
encoding = UTF-8 
encrypt = False 
follow_symlinks = False 
force = False 
get_continue = False 
gpg_command = /usr/bin/gpg 
gpg_decrypt = %(gpg_command)s -d –verbose –no-use-agent –batch –yes –passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s 
gpg_encrypt = %(gpg_command)s -c –verbose –no-use-agent –batch –yes –passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s 
gpg_passphrase = 
guess_mime_type = True 
host_base = node1 
host_bucket = %(bucket)s.node1 
human_readable_sizes = False 
list_md5 = False 
log_target_prefix = 
preserve_attrs = True 
progress_meter = True 
proxy_host = 
proxy_port = 0 
recursive = False 
recv_chunk = 4096 
reduced_redundancy = False 
secret_key = kqHLxnI14WqSl0Eh5akr73evrqmFZjfxYxUmG04h 
send_chunk = 4096 
simpledb_host = sdb.amazonaws.com 
skip_existing = False 
socket_timeout = 10 
urlencoding_mode = normal 
use_https = False 
verbosity = WARNING

在这里，access_key 和 secret_key 就是 radosgw-admin创建用户获得的，也可以用以下命令获取。hostbase 和 hostbucket 里的 node1都需要修改成你自己的server name 
radosgw-admin user info –uid=cephtest

8）测试 
首先用命令检查以下http 服务器是否正常工作： 
curl node1 
如果正确，应该返回：

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-pi" style="color: rgb(0, 102, 102); box-sizing: border-box;"><?xml version="1.0" encoding="UTF-8"?></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">ListAllMyBucketsResult</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">xmlns</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"http://s3.amazonaws.com/doc/2006-03-01/"</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">Owner</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">ID</span>></span>anonymous<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">ID</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">DisplayName</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">DisplayName</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">Owner</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">Buckets</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">Buckets</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">ListAllMyBucketsResult</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

如果不正确，那么http的服务器有问题。如果是apache http的服务器，那么查看服务器的配置的日志，查找相应的问题。 
/etc/httpd/logs/rgw_error.log

一般的常遇见问题都是: 
proxy: FCGI: attempt to connect to 182.92.171.199:9000 (182.92.171.199) failed 
显示连不上FCGI，那么看以下ceph.conf配置: 
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0 
host和 port是否一致。

如果是civetweb方式启动，可以查看你在ceph.conf配置的日志 /var/log/ceph/client.radosgw.gateway.log来解决问题。

如果上述没有问题，可以用命令 
s3cmd la 
这么命令是list 所有的bucket下的对象

s3cmd mb s3://BUCKET 创建一个bucket，目前遇到的一个问题是，只能创建“BUCKET”为前缀的bucket，这个不知道为啥，还需要进一步研究和配置。