ceph 对象存储网关rados gateway和S3接口测试详细安装配置文档
给印象笔记做个小广告：
https://app.yinxiang.com/shard/s41/sh/d0d2d6fe-af84-4f66-8dd0-d5b6ab7c940e/4ec4059af3a7a656ae560157023a511a
前言
环境 ubuntu 14.04，ceph 0.80.5，安装主要流程当然是根据ceph官网得来，如果本文写的不明白请留言或者参考官网。
安装apache2，libapache2-mod-fastcgi，CEPH OBJECT GATEWAY
1. 安装apache2 和libapache2-mod-fastcgi
需要先添加这两个包的源，主要是第二个的源是默认没有的，apt-get不到
wget -q -O- https://raw.github.com/ceph/ceph/master/keys/autobuild.asc | sudo apt-key add -
echo deb http://gitbuilder.ceph.com/apache2-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-apache.list
echo deb http://gitbuilder.ceph.com/libapache-mod-fastcgi-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-fastcgi.list
sudo apt-get update && sudo apt-get install apache2 libapache2-mod-fastcgi
2. 配置apache/fastcgi
sudo vim /etc/apache2/apache2.conf
增加一行ServerName {fqdn}，我本机hostname是node29
ServerName node29
Enable the URL rewrite modules for Apache and FastCGI：
sudo a2enmod rewrite sudo a2enmod fastcgi
sudo service apache2 restart
3. ssl配置
Some REST clients use HTTPS by default. So you should consider enabling SSL for Apache.
这个配置没搞会，后续导致连不上，删了才弄好的，所以如果不需要https就不要搞了
4.ADD WILDCARD TO DNS
这个地方也不太明白，没有搞，后面倒也不是必须
5.INSTALL CEPH OBJECT GATEWAY
To install the Ceph Object Gateway daemon：
sudo apt-get install radosgw
To install the Ceph Object Gateway synchronization agent：
sudo apt-get install radosgw-agent
配置 CEPH OBJECT GATEWAY
The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:
- A name for the gateway instance. We use gateway in this guide.
- A storage cluster user name with appropriate permissions in a keyring.
- Pools to store its data.
- A data directory for the gateway instance.
- An instance entry in the Ceph Configuration file.
- A configuration file for the web server to interact with FastCGI.
1. CREATE A USER AND KEYRING
Create a keyring for the gateway：
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
2. Generate a Ceph Object Gateway user name and key for each instance
For exemplary purposes, we will use the name gateway after client.radosgw:
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
Add capabilities to the key. See Configuration Reference - Pools for details on the effect of write permissions for the monitor and creating pools.
sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
3. 将key添加到ceph集群中
Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
注意：实际在我的环境上路径是这个：
sudo ceph -k /etc/ceph/keyring.client.admin auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
4. 如果想在多个节点都装客户端
Distribute the keyring to the node with the gateway instance：
sudo scp /etc/ceph/ceph.client.radosgw.keyring ceph@{hostname}:/home/ceph
ssh {hostname}
sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring
5. 创建pool
Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. 在有些情况下有些pool是可以在执行过程中默认创建的，但是有些可能创建不成功，
所以建议手工创建吧：
ceph osd pool create .rgw 128 128
ceph osd pool create .rgw.root 128 128
ceph osd pool create .rgw.control 128 128
ceph osd pool create .rgw.gc 128 128
ceph osd pool create .rgw.buckets 128 128
ceph osd pool create .rgw.buckets.index 128 128
ceph osd pool create .log 128 128
ceph osd pool create .intent-log 128 128
ceph osd pool create .usage 128 128
ceph osd pool create .users 128 128
ceph osd pool create .users.email 128 128
ceph osd pool create .users.swift 128 128
ceph osd pool create .users.uid 128 128
6. 添加一个gateway配置到ceph.conf
[client.radosgw.{instance-name}]
host = {host-name}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.{instance-name}.fastcgi.sock
log file = /var/log/ceph/client.radosgw.{instance-name}.log
这里的{instance-name}是gateway：
[client.radosgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
以下配置为非必须：
CONFIGURING PRINT CONTINUE
On CentOS/RHEL distributions, turn off print continue. If you have it set to true, you may encounter problems with PUT operations.
rgw print continue = false
CONFIGURING OPERATIONS LOGGING
In early releases of Ceph (v0.66 and earlier), the Ceph Object Gateway will log every successful operation in the Ceph Object Gateway backend by default. This means that every request, whether it is a read request or a write request will generate a gateway operation that writes data. This does not come without cost, and may affect overall performance. Turning off logging completely can be done by adding the following config option to the Ceph configuration file:
rgw enable ops log = false
Another way to reduce the logging load is to send operations logging data to a UNIX domain socket, instead of writing it to the Ceph Object Gateway backend:
rgw ops log rados = false
rgw enable ops log = true
rgw ops log socket path =
When specifying a UNIX domain socket, it is also possible to specify the maximum amount of memory that will be used to keep the data backlog:
rgw ops log data backlog =
Any backlogged data in excess to the specified size will be lost, so the socket needs to be read constantly.
7. 使能S3的子域名调用，这一步不确定是不是必须
ENABLING SUBDOMAIN S3 CALLS
To use a Ceph Object Gateway with subdomain S3 calls (e.g., http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.radosgw.gateway] section of your Ceph configuration file:
还需要在/etc/ceph.conf的gateway配置中添加：
[client.radosgw.gateway]
...
rgw dns name = {hostname}
You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:
安装dnsmasq：
$ sudo apt-get install dnsmasq
You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:
address=/{hostname}/{host-ip-address}
listen-address={client-loopback-ip}
我这里的配置：
$ vim /etc/dnsmasq.conf中添加：
address=/node29/10.2.0.29
listen-address=127.0.0.1
Then, add the {client-loopback-ip} IP address as the first DNS nameserver on client the machine(s).
See Add Wildcard to DNS for details.
sudo /etc/init.d/dnsmasq restart
8. 增加一个ceph对象网关脚本
ADD A CEPH OBJECT GATEWAY SCRIPT
Add a s3gw.fcgi file (use the same name referenced in the first line of rgw.conf). For Debian/Ubuntu distributions, save the file to the /var/www directory. For CentOS/RHEL distributions, save the file to the /var/www/html directory. Assuming a cluster named ceph (default), and the user created in previous steps, the contents of the file should include:
增加一个/var/www/s3gw.fcgi配置文件（use the same name referenced in the first line of rgw.conf）
$vim /var/www/s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
对s3gw.fcgi增加可执行权限：
sudo chmod +x s3gw.fcgi
On some distributions, you must also change the ownership to apache.
sudo chown apache:apache s3gw.fcgi
注意：ubuntu 默认的apache用户是www-data
sudo chown www-data:www-data s3gw.fcgi
创建数据目录：
CREATE DATA DIRECTORY
Deployment scripts may not create the default Ceph Object Gateway data directory. Create data directories for each instance of a radosgw daemon (if you haven’t done so already). The host variables in the Ceph configuration file determine which host runs each instance of a radosgw daemon. The typical form specifies the radosgw daemon, the cluster name and the daemon ID.
sudo mkdir -p /var/lib/ceph/radosgw/{$cluster}-{$id}
Using the exemplary ceph.conf settings above, you would execute the following:
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway
创建一个网关配置文件：
CREATE A GATEWAY CONFIGURATION
On the host where you installed the Ceph Object Gateway, create an rgw.conf file. For Debian/Ubuntu systems, place the file in the /etc/apache2/sites-available directory. For CentOS/RHEL systems, place the file in the /etc/httpd/conf.d directory.
We recommend deploying FastCGI as an external server, because allowing Apache to manage FastCGI sometimes introduces high latency. To manage FastCGI as an external server, use the FastCgiExternalServer directive. See FastCgiExternalServer for details on this directive. See Module mod_fastcgi for general details. See Apache Virtual Host documentation for details onformat and settings. See Directive for additional details.
Ceph Object Gateway requires a rewrite rule for the Amazon S3-compatible interface. It’s required for passing in the HTTP_AUTHORIZATION env for S3, which is filtered out by Apache. The rewrite rule is not necessary for the OpenStack Swift-compatible interface.
You should configure Apache to allow encoded slashes, provide paths for log files and to turn off server signatures. See below for an exemplary embodiment of a gateway configuration for Debian/Ubuntu and CentOS/RHEL.
$vim /etc/apache2/sites-available/rgw.conf
FastCgiExternalServer /var/www/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
ServerName node29
ServerAlias *.node29
ServerAdmin [email protected]
DocumentRoot /var/www
RewriteEngine On
RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
Options +ExecCGI
AllowOverride All
SetHandler fastcgi-script
Order allow,deny
Allow from all
AuthBasicAuthoritative Off
AllowEncodedSlashes On
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined
ServerSignature Off
9. 使rgw.conf配置生效
For Debian/Ubuntu distributions, enable the site for rgw.conf.
sudo a2ensite rgw.conf
Then, disable the default site.
sudo a2dissite default
RESTART SERVICES AND START THE GATEWAY
To ensure that all components have reloaded their configurations, we recommend restarting your ceph and apache services. Then, start up the radosgw service.
RESTART APACHE
On Debian/Ubuntu systems, use apache2. For example:
sudo service apache2 restart
START THE GATEWAY
On Debian/Ubuntu systems, use radosgw. For example:
sudo /etc/init.d/radosgw start
VERIFY THE RUNTIME
Once the service is up and running, you can make an anonymous GET request to see if the gateway returns a response. A simple HTTP request to the domain name should return the following:
可以用curl来执行：
curl node29
成功则显示以下相应，同时在日志中会有记录：
anonymous
配置一个用户，配置s3cmd进行S3接口测试
CREATE A USER
To create a user (S3 interface), execute the following:
radosgw-admin user create --uid={username} --display-name="{display-name}" [--email={email}]
For example:
radosgw-admin user create --uid=cephtest --display-name="ceph test" [email protected]
显示：
{ "user_id": "cephtest",
"display_name": "ceph test",
"email": "[email protected]",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{ "user": "cephtest",
"access_key": "8JGTP1714JRRAPRDLBI4",
"secret_key": "i4O+yJMNii87ruLMwcIIYbjLmKIUaoSO1svQOoB9"}],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"user_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"temp_url_keys": []}
注意：如果生成的key中有"\",最后把这个user删掉，再重新生成直到不含"\"：
radosgw-admin user rm --uid=cephtest
Important：Check the key output. Sometimes radosgw-admin generates a JSON escape (\) character, and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character (\), encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually.
安装和配置s3cmd
apt-get install s3cmd
进行~/.s3cfg配置, 将之前radosgw-admin创建的user的access_key和secret_key，实际的host地址进行赋值：
$vim ~/.s3cfg
[default]
access_key = PPA789F4W5ANH3COC51O
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2010-07-15/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = node29
host_bucket = %(bucket)s.node29
human_readable_sizes = False
list_md5 = False
log_target_prefix =
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = kqHLxnI14WqSl0Eh5akr73evrqmFZjfxYxUmG04h
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 10
urlencoding_mode = normal
use_https = False
verbosity = WARNING
测试是否配置成功：
s3cmd --configure
一路选确定，因为我们已经配置过了，直到最后一步Test选y：
Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
大功告成！
后记
简单的测试命令：
帮助命令：
s3cmd --help
创建一个名为rgw的bucket：
s3cmd mb s3://rgw
将tempfile写入到rgw这个bucket中：
s3cmd put tempfile s3://rgw
列出rgw这个bucket中的文件列表：
s3cmd ls s3://rgw
读取bucket中的tempfile文件到本地：
s3cmd get s3://rgw/tempfile tempfile_new
https://app.yinxiang.com/shard/s41/sh/d0d2d6fe-af84-4f66-8dd0-d5b6ab7c940e/4ec4059af3a7a656ae560157023a511a