ceph 对象功能部署

S3介绍	S3是亚马逊AWS提供的简单存储服务（可以理解为有公网域名的大容量高可用存储） S3配合CloudFront服务可作为CDN使用，它提供多节点全球发布 在使用过程中，一般是通过web页面上传内容到S3 Bucket（Bucket桶，S3里存内容的容器） 这个过程缓慢繁琐容易中断 推荐S3cmd这个软件用命令行实现 如下就是这个命令行工具的简明手册
S3cmd安装	安装s3cmd并设置好 下载下来之后，直接解压就可以使用其中的二进制可执行文件了 你可以安装到系统中去，这样在任何目录下都可以直接使用s3cmd 安装：   python setup.py install (注意此处的Python必须是2.6版本以上）   或者使用pip来安装，如果没有pip命令，就   yum -y install python-pip 然后 pip install s3cmd [root@cephclient ~]# pip install s3cmd Requirement already satisfied: s3cmd in /usr/lib/python2.7/site-packages Requirement already satisfied: python-dateutil in /usr/lib/python2.7/site-packages (from s3cmd) Requirement already satisfied: python-magic in /usr/lib/python2.7/site-packages (from s3cmd) [root@cephclient ~]# Or:   [root@ceph01 ~]# yum install s3cmd Installed:   s3cmd.noarch 0:1.6.1-1.el7                                                                                                                         Complete! [root@ceph01 ~]#   [root@cephclient ~]# yum update && sudo apt-get install s3cmd   验证是否安装成功:   [root@ceph01 ~]# s3cmd --version s3cmd version 1.6.1                            #表示安装成功 [root@ceph01 ~]#
为S3的访问创建账号： 在ceph 管理端操作	[root@cephclient init.d]# radosgw-admin user create --uid=guangzhou01 --display-name="guangzhou01" [email protected] {     "user_id": "guangzhou01",     "display_name": "guangzhou01",     "email": "[email protected]",     "suspended": 0,     "max_buckets": 1000,     "auid": 0,     "subusers": [],     "keys": [         {             "user": "guangzhou01",             "access_key": "RSZTJL9C8ESOCDMYEW87",             "secret_key": "RS41a6cYsAqfelCLvZnDCQDk8Ni8Qat4EM3p9b3N"         }     ],     "swift_keys": [],     "caps": [],     "op_mask": "read, write, delete",     "default_placement": "",     "placement_tags": [],     "bucket_quota": {         "enabled": false,         "max_size_kb": -1,         "max_objects": -1     },     "user_quota": {         "enabled": false,         "max_size_kb": -1,         "max_objects": -1     },     "temp_url_keys": [] }   [root@cephclient init.d]#     在创建用户时可以指定用的access key 和secret key，也可以让其自动产生，但是在返回的结果中可能包含转义字符，有些客户端不能识别，需要将其删除，或者重新生成。 同一个用户可以有多对access_key和secret_key,但是一个access_key只能对应一个用户。   S3 中access_key和secret_key分别扮演者用户名ID和用户密码的角色。 access_key–>用户名ID secret_key–>用户密码
配置S3 client：	[root@ceph01 ~]# s3cmd --configure   Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.   Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: E9501Q1YHS4ECXA6M0X5 Secret Key: guangzhou01 Default Region [US]:   Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]:   When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: no   On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name:   New settings:   Access Key: E9501Q1YHS4ECXA6M0X5   Secret Key: guangzhou01   Default Region: US   Encryption password:   Path to GPG program: /usr/bin/gpg   Use HTTPS protocol: False   HTTP Proxy server name:   HTTP Proxy server port: 0   Test access with supplied credentials? [Y/n] n   Save settings? [y/N] y Configuration saved to '/root/.s3cfg' [root@ceph01 ~]#
	在这个交互配置过程中，只配置了其中access_key和secret_key，如果要正常使用自己搭建的存储还有3项需要配置：   cloudfont_host host_base host_bucket 其对应的配置如下：
测试可以执行的配置文件 .s3cfg	[root@cephclient ~]# [root@cephclient ~]# [root@cephclient ~]# more /root/.s3cfg [default] access_key = RSZTJL9C8ESOCDMYEW87 access_token = add_encoding_exts = add_headers = bucket_location = US ca_certs_file = cache_file = check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cephclient default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = cephclient host_bucket = %(*)s.cephclient human_readable_sizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limitrate = 0 list_md5 = False log_target_prefix = long_listing = False max_delete = -1 mime_type = multipart_chunk_size_mb = 15 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 put_continue = False recursive = False recv_chunk = 4096 reduced_redundancy = False requester_pays = False restore_days = 1 secret_key = RS41a6cYsAqfelCLvZnDCQDk8Ni8Qat4EM3p9b3N send_chunk = 4096 server_side_encryption = False signature_v2 = False simpledb_host = cephclient skip_existing = False socket_timeout = 10 stop_on_error = False storage_class = urlencoding_mode = normal use_https = False use_mime_magic = True verbosity = WARNING website_endpoint = http://rgw.cephclient/ website_error = website_index = index.html   [root@cephclient ~]# [root@cephclient ~]# [root@cephclient ~]#
重启和查看网关radosgw服务	[root@cephclient init.d]# systemctl restart ceph-radosgw.target [root@cephclient init.d]# systemctl status ceph-radosgw.target
遇到的问题:	1 ) [root@ceph01 ceph]# s3cmd mb s3://test ERROR: [Errno -2] Name or service not known [root@ceph01 ceph]#   2 ) [root@cephclient conf.d]# s3cmd ls WARNING: Could not refresh role ERROR: [Errno 111] Connection refused   3 ) root@ceph03 ceph]# s3cmd mb s3://test   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!     An unexpected error has occurred.   Please try reproducing the error using   the latest s3cmd code from the git master   branch found at:     https://github.com/s3tools/s3cmd   and have a look at the known issues list:     https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions   If the error persists, please report the   following lines (removing any private   info as necessary) to:    [email protected]     !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!   Invoked as: /usr/bin/s3cmd mb s3://test Problem: gaierror: [Errno -2] Name or service not known S3cmd:   1.6.1 python:   2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] environment LANG=en_US.UTF-8   Traceback (most recent call last):   File "/usr/bin/s3cmd", line 2919, in     rc = main()   File "/usr/bin/s3cmd", line 2841, in main     rc = cmd_func(args)   File "/usr/bin/s3cmd", line 205, in cmd_bucket_create     response = s3.bucket_create(uri.bucket(), cfg.bucket_location)   File "/usr/lib/python2.7/site-packages/S3/S3.py", line 367, in bucket_create     response = self.send_request(request)   File "/usr/lib/python2.7/site-packages/S3/S3.py", line 1081, in send_request     conn = ConnMan.get(self.get_hostname(resource['bucket']))   File "/usr/lib/python2.7/site-packages/S3/ConnMan.py", line 192, in get     conn.c.connect()   File "/usr/lib64/python2.7/httplib.py", line 807, in connect     self.timeout, self.source_address)   File "/usr/lib64/python2.7/socket.py", line 553, in create_connection     for res in getaddrinfo(host, port, 0, SOCK_STREAM): gaierror: [Errno -2] Name or service not known   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!     An unexpected error has occurred.   Please try reproducing the error using   the latest s3cmd code from the git master   branch found at:     https://github.com/s3tools/s3cmd   and have a look at the known issues list:     https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions   If the error persists, please report the   above lines (removing any private   info as necessary) to:    [email protected] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!   [root@ceph03 ceph]#
上面的问题解决办法	当前测试的版本是: [root@cephclient ~]# s3cmd --version s3cmd version 1.5.2 [root@cephclient ~]# [root@ceph03 ceph]# s3cmd --version s3cmd version 1.6.1   解决办法： 修改 .s3cfg 配置文件！ 测试好多遍，重新拷贝上面的的.s3cfg 配置文件进行修改后，问题就解决了！
创建容器	[root@cephclient ~]# s3cmd mb s3://cephfs_data [root@cephclient ~]# s3cmd mb s3://guangzhou01 Bucket 's3://guangzhou01/' created
查看容器	[root@cephclient ~]# s3cmd ls 2017-04-22 13:16  s3://cephfs_data 2017-04-22 13:17  s3://guangzhou01 [root@cephclient ~]#
	[root@cephclient ~]# s3cmd mb s3://first_bucket Bucket 's3://first_bucket/' created [root@cephclient ~]# s3cmd ls 2017-04-22 13:16  s3://cephfs_data 2017-04-22 13:17  s3://guangzhou01 2017-04-22 14:27  s3://first_bucket [root@cephclient ~]#
删除bucket	[root@cephclient ~]# s3cmd rb s3://cephfs_data09 Bucket 's3://cephfs_data09/' removed
上传文件对象：	[root@cephclient ~]# s3cmd put calamari-1.5.5.tar.gz S3://first_bucket calamari-1.5.5.tar.gz -> s3://first_bucket/calamari-1.5.5.tar.gz  [1 of 1]  1320799 of 1320799   100% in    1s   781.02 kB/s  done [root@cephclient ~]#
查询上传的内容	[root@cephclient ceph]# s3cmd ls s3://guangzhou01/ 2017-04-22 15:04 1494531681   s3://guangzhou01/na1401_brocade_windows.zip [root@cephclient ceph]#
从S3 bucket上下载文件 下载单个文件	下载整个目录       [root@ceph03 ~]# s3cmd get --recursive s3://guangzhou01/s3test  （1） ERROR: Skipping ./s3test/: Is a directory download: 's3://guangzhou01/s3test/backup.bat' -> './s3test/backup.bat'  [2 of 5]  57 of 57   100% in    0s    15.12 kB/s  done download: 's3://guangzhou01/s3test/robocopy' -> './s3test/robocopy'  [3 of 5]  0 of 0     0% in    0s     0.00 B/s  done download: 's3://guangzhou01/s3test/test.bat' -> './s3test/test.bat'  [4 of 5]  44 of 44   100% in    0s    12.63 kB/s  done download: 's3://guangzhou01/s3test/test1.bat' -> './s3test/test1.bat'  [5 of 5]  57 of 57   100% in    0s    17.62 kB/s  done [root@ceph03 ~]# ls anaconda-ks.cfg  ceph03.txt  na1401_brocade_windows.zip  s3test [root@ceph03 ~]# cd s3test [root@ceph03 s3test]# ls backup.bat  robocopy  test1.bat  test.bat [root@ceph03 s3test]#     ---------------------------------------------------------------------------------------------------------------------------------------------------------- 注意下面两条命名的区别 get s3://public.s3tools.org/somewhere get s3://public.s3tools.org/somewhere/   第一条命令下载下来的东西会是/somewhere/dir1/** 即连同somewhere及其子目录一并下载，并在本地创建相同的目录结构 第二条下载下来的没有somewhere目录，即指下载该目录的内容，在本地并不创建somewhere目录，一般会使用recursive参数
删除操作:	删除 s3://guangzhou01/s3test 下的所有内容   [root@ceph03 ~]# s3cmd ls s3://guangzhou01/s3test                        DIR   s3://guangzhou01/s3test/ [root@ceph03 ~]# s3cmd del --recursive s3://guangzhou01/s3test/ delete: 's3://guangzhou01/s3test/' [root@ceph03 ~]# s3cmd ls s3://guangzhou01/s3test/ [root@ceph03 ~]#
创建SWIFT类型USER2.7 实际验证	编写了一段python代码，用于访问网关，并创建bucket，并通过list方法罗列出当前所有的bucket （官方示例）   依赖库安装
	[root@cephclient ceph]# vim s3test.py
启动radosgw 服务	开始 radosgw 服务 :   [root@cephclient conf.d]# radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway [root@cephclient conf.d]#       [root@cephclient init.d]# systemctl restart ceph-radosgw.target [root@cephclient init.d]# systemctl status ceph-radosgw.target   鈼?[0m ceph-radosgw.target - ceph target allowing to start/stop all [email protected] instances at once    Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw.target; enabled; vendor preset: enabled)    Active: active since Sat 2017-04-22 18:10:16 CST; 13s ago   Apr 22 18:10:16 cephclient systemd[1]: Stopping ceph target allowing to start/stop all [email protected] instanc... once. Apr 22 18:10:16 cephclient systemd[1]: Reached target ceph target allowing to start/stop all [email protected] i... once. Apr 22 18:10:16 cephclient systemd[1]: Starting ceph target allowing to start/stop all [email protected] instanc... once. Hint: Some lines were ellipsized, use -l to show in full. [root@cephclient init.d]#
运行python ,通过s3创建容器	[root@cephclient ceph]# python s3test.py my-new-bucket   2017-04-21T06:54:27.309Z [root@cephclient ceph]# [root@cephclient ceph]# python s3test.py my-new-bucket   2017-04-21T06:54:27.309Z my-new-bucket01 2017-04-21T07:26:26.866Z my-new-bucket02 2017-04-21T07:26:53.634Z
设置文件访问权限	[root@ceph1 cluster]# s3cmd put ceph.conf s3://S3test2 upload: 'ceph.conf' -> 's3://S3test2/ceph.conf'  [1 of 1]  229 of 229   100% in    0s   840.05 B/s  done [root@ceph1 cluster]# s3cmd setacl s3://S3test2/ceph.conf --acl-public --preserve s3://S3test2/ceph.conf: ACL set to Public  [1 of 1] [root@ceph1 cluster]#