ceph 对象存储网关radosgw和S3接口详细安装配置测试文档

前言

环境 ubuntu 14.04，ceph 0.80.5，安装主要流程当然是根据ceph官网得来，如果本文写的不明白请留言或者参考官网。

安装apache2，libapache2-mod-fastcgi，CEPH OBJECT GATEWAY

1. 安装apache2 和libapache2-mod-fastcgi

需要先添加这两个包的源，主要是第二个的源是默认没有的，apt-get不到

wget -q -O- https://raw.github.com/ceph/ceph/master/keys/autobuild.asc | sudo apt-key add -

echo deb http://gitbuilder.ceph.com/apache2-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-apache.list

echo deb http://gitbuilder.ceph.com/libapache-mod-fastcgi-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-fastcgi.list

sudo apt-get update && sudo apt-get install apache2 libapache2-mod-fastcgi

2. 配置apache/fastcgi

sudo vim /etc/apache2/apache2.conf
增加一行ServerName {fqdn}，我本机hostname是node29

ServerName node29

Enable the URL rewrite modules for Apache and FastCGI：

sudo a2enmod rewrite sudo a2enmod fastcgi

sudo service apache2 restart

3. ssl配置
Some REST clients use HTTPS by default. So you should consider enabling SSL for Apache.
这个配置没搞会，后续导致连不上，删了才弄好的，所以如果不需要https就不要搞了

4.ADD WILDCARD TO DNS
这个地方也不太明白，没有搞，后面倒也不是必须

5.INSTALL CEPH OBJECT GATEWAY
To install the Ceph Object Gateway daemon：

sudo apt-get install radosgw

To install the Ceph Object Gateway synchronization agent：

sudo apt-get install radosgw-agent

配置 CEPH OBJECT GATEWAY

The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:

• A name for the gateway instance. We use gateway in this guide.
• A storage cluster user name with appropriate permissions in a keyring.
• Pools to store its data.
• A data directory for the gateway instance.
• An instance entry in the Ceph Configuration file.
• A configuration file for the web server to interact with FastCGI.

1. CREATE A USER AND KEYRING

Create a keyring for the gateway：
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring


2. Generate a Ceph Object Gateway user name and key for each instance
For exemplary purposes, we will use the name gateway after client.radosgw:
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
Add capabilities to the key. See Configuration Reference - Pools for details on the effect of write permissions for the monitor and creating pools.
sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

3. 将key添加到ceph集群中 
Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
注意：实际在我的环境上路径是这个：
sudo ceph -k /etc/ceph/keyring.client.admin auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring

4. 如果想在多个节点都装客户端
Distribute the keyring to the node with the gateway instance：
sudo scp /etc/ceph/ceph.client.radosgw.keyring  ceph@{hostname}:/home/ceph
ssh {hostname}
sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring

5. 创建pool
Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. 在有些情况下有些pool是可以在执行过程中默认创建的，但是有些可能创建不成功，
所以建议手工创建吧：
ceph osd pool create .rgw 128 128
ceph osd pool create .rgw.root 128 128
ceph osd pool create .rgw.control 128 128
ceph osd pool create .rgw.gc 128 128
ceph osd pool create .rgw.buckets 128 128
ceph osd pool create .rgw.buckets.index 128 128
ceph osd pool create .log 128 128
ceph osd pool create .intent-log 128 128
ceph osd pool create .usage 128 128
ceph osd pool create .users 128 128
ceph osd pool create .users.email 128 128
ceph osd pool create .users.swift 128 128
ceph osd pool create .users.uid 128 128

6. 添加一个gateway配置到ceph.conf

[client.radosgw.{instance-name}]
host = {host-name}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.{instance-name}.fastcgi.sock
log file = /var/log/ceph/client.radosgw.{instance-name}.log

这里的{instance-name}是gateway：

[client.radosgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log

以下配置为非必须：

CONFIGURING PRINT CONTINUE
On CentOS/RHEL distributions, turn off print continue. If you have it set to true, you may encounter problems with PUT operations.

rgw print continue = false

CONFIGURING OPERATIONS LOGGING
In early releases of Ceph (v0.66 and earlier), the Ceph Object Gateway will log every successful operation in the Ceph Object Gateway backend by default. This means that every request, whether it is a read request or a write request will generate a gateway operation that writes data. This does not come without cost, and may affect overall performance. Turning off logging completely can be done by adding the following config option to the Ceph configuration file:

rgw enable ops log = false

Another way to reduce the logging load is to send operations logging data to a UNIX domain socket, instead of writing it to the Ceph Object Gateway backend:
rgw ops log rados = false
rgw enable ops log = true
rgw ops log socket path = <path to socket>

When specifying a UNIX domain socket, it is also possible to specify the maximum amount of memory that will be used to keep the data backlog:
rgw ops log data backlog = <size in bytes>

Any backlogged data in excess to the specified size will be lost, so the socket needs to be read constantly.


7. 使能S3的子域名调用，这一步不确定是不是必须
ENABLING SUBDOMAIN S3 CALLS
To use a Ceph Object Gateway with subdomain S3 calls (e.g., http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.radosgw.gateway] section of your Ceph configuration file:
还需要在/etc/ceph.conf的gateway配置中添加：
[client.radosgw.gateway]
 ...
 rgw dns name = {hostname}

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:

安装dnsmasq：
$ sudo apt-get install dnsmasq

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:
address=/{hostname}/{host-ip-address}
listen-address={client-loopback-ip}

我这里的配置：
$ vim /etc/dnsmasq.conf中添加：
address=/node29/10.2.0.29
listen-address=127.0.0.1

Then, add the {client-loopback-ip} IP address as the first DNS nameserver on client the machine(s).
See Add Wildcard to DNS for details.

sudo /etc/init.d/dnsmasq restart

8. 增加一个ceph对象网关脚本
ADD A CEPH OBJECT GATEWAY SCRIPT
Add a s3gw.fcgi file (use the same name referenced in the first line of rgw.conf). For Debian/Ubuntu distributions, save the file to the /var/www directory. For CentOS/RHEL distributions, save the file to the /var/www/html directory. Assuming a cluster named ceph (default), and the user created in previous steps, the contents of the file should include:
增加一个/var/www/s3gw.fcgi配置文件（use the same name referenced in the first line of rgw.conf）
$vim /var/www/s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway

对s3gw.fcgi增加可执行权限：
sudo chmod +x s3gw.fcgi

On some distributions, you must also change the ownership to apache.
sudo chown apache:apache s3gw.fcgi

注意：ubuntu 默认的apache用户是www-data
sudo chown www-data:www-data s3gw.fcgi

创建数据目录：
CREATE DATA DIRECTORY
Deployment scripts may not create the default Ceph Object Gateway data directory. Create data directories for each instance of a radosgw daemon (if you haven’t done so already). The host variables in the Ceph configuration file determine which host runs each instance of a radosgw daemon. The typical form specifies the radosgw daemon, the cluster name and the daemon ID.

sudo mkdir -p /var/lib/ceph/radosgw/{$cluster}-{$id}

Using the exemplary ceph.conf settings above, you would execute the following:
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway

创建一个网关配置文件：
CREATE A GATEWAY CONFIGURATION

On the host where you installed the Ceph Object Gateway, create an rgw.conf file. For Debian/Ubuntu systems, place the file in the /etc/apache2/sites-available directory. For CentOS/RHEL systems, place the file in the /etc/httpd/conf.d directory.
We recommend deploying FastCGI as an external server, because allowing Apache to manage FastCGI sometimes introduces high latency. To manage FastCGI as an external server, use the FastCgiExternalServer directive. See FastCgiExternalServer for details on this directive. See Module mod_fastcgi for general details. See Apache Virtual Host documentation for details on <VirtualHost> format and settings. See <IfModule> Directive for additional details.
Ceph Object Gateway requires a rewrite rule for the Amazon S3-compatible interface. It’s required for passing in the HTTP_AUTHORIZATION env for S3, which is filtered out by Apache. The rewrite rule is not necessary for the OpenStack Swift-compatible interface.
You should configure Apache to allow encoded slashes, provide paths for log files and to turn off server signatures. See below for an exemplary embodiment of a gateway configuration for Debian/Ubuntu and CentOS/RHEL.

$vim /etc/apache2/sites-available/rgw.conf

FastCgiExternalServer /var/www/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock

<VirtualHost *:80>

 ServerName node29
 ServerAlias *.node29
 ServerAdmin [email protected]
 DocumentRoot /var/www
 RewriteEngine On
 RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

 <IfModule mod_fastcgi.c>
 <Directory /var/www>
 Options +ExecCGI
 AllowOverride All
 SetHandler fastcgi-script
 Order allow,deny
 Allow from all
 AuthBasicAuthoritative Off
 </Directory>
 </IfModule>

 AllowEncodedSlashes On
 ErrorLog /var/log/apache2/error.log
 CustomLog /var/log/apache2/access.log combined
 ServerSignature Off
</VirtualHost>

9. 使rgw.conf配置生效
For Debian/Ubuntu distributions, enable the site for rgw.conf.
sudo a2ensite rgw.conf
Then, disable the default site.
sudo a2dissite default

RESTART SERVICES AND START THE GATEWAY
To ensure that all components have reloaded their configurations, we recommend restarting your ceph and apache services. Then, start up the radosgw service.

RESTART APACHE
On Debian/Ubuntu systems, use apache2. For example:

sudo service apache2 restart

START THE GATEWAY
On Debian/Ubuntu systems, use radosgw. For example:

sudo /etc/init.d/radosgw start

VERIFY THE RUNTIME
Once the service is up and running, you can make an anonymous GET request to see if the gateway returns a response. A simple HTTP request to the domain name should return the following:

可以用curl来执行：
curl node29
成功则显示以下相应，同时在日志中会有记录：

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>anonymous</ID>
 <DisplayName/>
 </Owner>
 <Buckets/>
</ListAllMyBucketsResult>

配置一个用户，配置s3cmd进行S3接口测试

CREATE A USER
To create a user (S3 interface), execute the following:
radosgw-admin user create --uid={username} --display-name="{display-name}" [--email={email}]
For example:
radosgw-admin user create --uid=cephtest --display-name="ceph test" [email protected]
显示：
{ "user_id": "cephtest",
 "display_name": "ceph test",
 "email": "[email protected]",
 "suspended": 0,
 "max_buckets": 1000,
 "auid": 0,
 "subusers": [],
 "keys": [
 { "user": "cephtest",
 "access_key": "8JGTP1714JRRAPRDLBI4",
 "secret_key": "i4O+yJMNii87ruLMwcIIYbjLmKIUaoSO1svQOoB9"}],
 "swift_keys": [],
 "caps": [],
 "op_mask": "read, write, delete",
 "default_placement": "",
 "placement_tags": [],
 "bucket_quota": { "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1},
 "user_quota": { "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1},
 "temp_url_keys": []}

注意：如果生成的key中有"\",最后把这个user删掉，再重新生成直到不含"\"：
radosgw-admin user rm --uid=cephtest

Important：Check the key output. Sometimes radosgw-admin generates a JSON escape (\) character, and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character (\), encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually.

安装和配置s3cmd
apt-get install s3cmd

进行~/.s3cfg配置, 将之前radosgw-admin创建的user的access_key和secret_key，实际的host地址进行赋值：

$vim ~/.s3cfg 

[default]
access_key = PPA789F4W5ANH3COC51O
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2010-07-15/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = node29
host_bucket = %(bucket)s.node29
human_readable_sizes = False
list_md5 = False
log_target_prefix =
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = kqHLxnI14WqSl0Eh5akr73evrqmFZjfxYxUmG04h
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 10
urlencoding_mode = normal
use_https = False
verbosity = WARNING

测试是否配置成功：
s3cmd --configure

一路选确定，因为我们已经配置过了，直到最后一步Test选y：
Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

大功告成！

后记
简单的测试命令：
帮助命令：

s3cmd --help

创建一个名为rgw的bucket：
s3cmd mb s3://rgw

将tempfile写入到rgw这个bucket中：
s3cmd put tempfile s3://rgw

列出rgw这个bucket中的文件列表：
s3cmd ls s3://rgw

读取bucket中的tempfile文件到本地：
s3cmd get s3://rgw/tempfile tempfile_new