Ceph 通过radosgw提供RESTFul HTTP API接口支持对象存储能力,radosgw构建在librados之上,兼容Amazon S3以及Opensack Swift。
radosgw本质上是一个客户端程序,提供FastCGI 服务。作为一个客户端程序,需要满足如下要求:
radosgw支持以Apache、Civetweb、Nginx作为前端。Civetweb是默认前端,通过修改ceph.conf配置文件能够很容易的替换为Apache,通过配置能也以nginx作为前端。
下面分别给出centos7上rgw的安装配置过程
通过ceph-deploy可以方便的在rgw node上安装rgw包:
#> ceph-deploy --rgw install {rgw-node-name}
每个rgw实例都需要一个授权用户及key,下面的例子中创建了一个名为gateway的用户,并将密钥文件存储在/etc/ceph目录下
#> ceph auth get-or-create client.radosgw.gateway osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring
rgw需要存储池来存储数据,如果授权用户具有相关权限,rgw将会自动创建存储池,如果使用默认的区域(region)和可用区(zone),将包含如下的池:
.rgw.root
.rgw.control
.rgw.gc
.rgw.buckets
.rgw.buckets.index
.rgw.buckets.extra
.log
.intent-log
.usage
.users
.users.email
.users.swift
.users.uid
当然,您也可以手动创建各个存储池:
#> ceph osd pool create {poolname} {pg-num} {pgp-num} {replicated | erasure} [{erasure-code-profile}] {ruleset-name} {ruleset-number}
在ceph.conf中添加一个名为gateway的实例。
如果以civetweb作为前端,配置如下:
[client.radosgw.gateway]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/radosgw/client.radosgw.gateway-node1.log
rgw_frontends = civetweb port=80
civetweb默认监听在7480端口,上述的配置中显示指定监听端口为80(
port=80
)
如果以apache作为前端,配置如下:
[client.radosgw.gateway]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = ""
log file = /var/log/radosgw/client.radosgw.gateway.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
rgw print continue = false
配置apache服务,创建/etc/httpd/conf.d/rgw.conf
,并写入如下内容:
ServerName localhost
DocumentRoot /var/www/html
ErrorLog /var/log/httpd/rgw_error.log
CustomLog /var/log/httpd/rgw_access.log combined
# LogLevel debug
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
SetEnv proxy-nokeepalive 1
ProxyPass / fcgi://localhost:9000/
配置好之后,重启apache服务:systemctl restart httpd.service
ceph.conf配置如下:
[client.radosgw.gateway]
rgw_frontends = fastcgi
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path = /var/run/ceph/ceph.radosgw.gateway.sock
log_file = /var/log/ceph/radosgw.log
rgw_print_continue = false
rgw_content_length_compat = true
配置nginx服务,在/etc/nginx/nginx.conf
文件的http段下添加如下内容:
http {
server {
listen 80 default;
server_name {hostname};
location / {
fastcgi_pass_header Authorization;
fastcgi_pass_request_headers on;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param CONTENT_LENGTH $content_length;
if ($request_method = PUT) {
rewrite ^ /PUT$request_uri;
}
include fastcgi_params;
fastcgi_pass unix:/var/run/ceph/ceph.radosgw.gateway.sock;
}
location /PUT/ {
internal;
fastcgi_pass_header Authorization;
fastcgi_pass_request_headers on;
include fastcgi_params;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_pass unix:/var/run/ceph/ceph.radosgw.gateway.sock;
}
}
注意: fastcgi_pass 指向的路径需要与ceph.conf中配置的路径一致
让nginx配置生效:nginx -s reload
通过上面的安装->创建用户->创建存储池->配置
过程,rgw也就准备就绪了,可以通过下面的命令启动实例:
//radosgw -c {conf_file} -n {rgw-name}
#> radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
ceph rgw的测试方式有:s3cmd, cosbench,也可以通过python库boto自己写测试程序。个人感觉cosbench很不错,大家可以试试。
有时为了提高rgw的并发能力,需要部署多个rgw实例。其实也很简单,在多个节点上部署多个rgw实例:只需要安装rgw包,并将ceph.conf文件,密钥文件,前端配置文件拷贝到相应的节点,然后启动实例就好。
至此rgw的部署实践过程就介绍完了,如有问题欢迎留言。