post_object for AWS S3 ruby SDK
前言
S3（Simple Storage Service）http://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html 是AWS上线的第一个服务，从此开启了AWS得云计算生态。作为公有云的领跑者，S3的API接口也默默的自然成了对象存储行业内的“标准接口”。一些的公司或者厂商(如Dropbox)后端直接采用S3存储系统来存放文件。很多的云存储服务虽然有自己的存储API接口，同时也会提供一套S3 API兼容接口。其实不难理解，如果某一天你的产品具有一定的市场竞争力，后端存储服务的切换就可以做到无缝连接。
项目组功能开发框架采用ruby编码。然后把AWS S3 ruby SDK集成进现有的测试框架。对项目中提供的兼容性S3 API进行自动化测试用例开发。开发过程中，发现当时拿到的版本（最后修改：2013-11-27）AWS S3 ruby SDK没有提供直接的post object API接口。
AWS S3 API
S3提供三类API：
Service:只有一个Get API,用来获取用户所有bucket列表
Bucket:相当于一个存储容器,容器里放Object
Object:对象存储单元，包括目录和文件
关于S3的API，需要有些Restful和HTTP基本概念的了解，想深入直接看文档（http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html）。S3里给我印象比较深的几个API操作是，multipart uploads（文件分块，多线程上传，提高效率）, object copy（对象的复制，减少不必要上传）, post object（通过浏览器上传资源到Amazon S3）。
Post object
Post object应该算得上S3 API里高级点的接口，主要优点有：
用户可以通过浏览器上传对象到Amazon S3
与Put上传相比，可以绕过web服务器，文件直接上传到Amazon S3
Post object 实现
利用SDK里已经提供了一个生成form的类PresignedPost，结合curb类库里的post方法，写了一个post_object API接口。接口参数和S3的其他API保持一致：
def extend_s3(obj)
if obj.instance_of?(AWS::S3::Client::V20060301)
# Reopen the class AWS::S3::Client::V20060301
obj.class.class_eval do
# Post object for multipart form-data
# @overload post_object(options = {})
# @param [Hash] options
# @option options [required,Bucket] :bucket
# @option options [String] :key Upload object name.
# @option options [String] :file Upload local file path or text.
# @option options [Boolean] :verbose Flag for debugging output.
# @option options [Boolean] :raised Flag for raising the network error.
# @option options [Boolean] :secure By setting this to false, you can cause
# {#url} to return an HTTP URL. By default it returns an HTTPS URL.
# @option options [Srting] :cache_control Sets the Cache-Control header stored with the object.
# @option options [String] :content_type Sets the Content-Type header stored with the object.
# @option options [String] :content_disposition Sets the Content-Disposition header stored with the object.
# @option options [String] :expires_header Sets the Expires header stored with the object.
# @option options [Symbol] :acl A canned access control policy. Valid values are:
# * `:private`
# * `:public_read`
# * `:public_read_write`
# * `:authenticated_read`
# * `:bucket_owner_read`
# * `:bucket_owner_full_control`
# @option options [Symbol] :server_side_encryption (nil) If this
# option is set, the object will be stored using server side
# encryption. The only valid value is `:aes256`, which
# specifies that the object should be stored using the AES
# encryption algorithm with 256 bit keys. By default, this
# option uses the value of the `:s3_server_side_encryption`
# option in the current configuration.
# @option options [String] :success_action_redirect The URL to
# which the client is redirected upon successful upload.
# @option options [Integer] :success_action_status The status
# code returned to the client upon successful upload if
# `:success_action_redirect` is not specified. Accepts the
# values 200, 201, or 204 (default).
# If the value is set to 200 or 204, Amazon S3 returns an
# empty document with a 200 or 204 status code.
# If the value is set to 201, Amazon S3 returns an XML
# document with a 201 status code. For information on the
# content of the XML document, see
# [POST Object](http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/index.html?RESTObjectPOST.html).
# @option options [Time, DateTime, Integer, String] :expires The time
# at which the signature will expire. By default the signature will
# expire one hour after it is generated
# @option options [Hash] :metadata A hash of the metadata fields included in the
# signed fields. Additional metadata fields may be provided with the upload
# as long as they satisfy the conditions specified for the upload.
# @option options [Range] :content_length The range of acceptable object sizes
# for the upload. By default any size object may be uploaded.
# @option options [Array<String>] :ignore Additional fields which may be sent
# with the upload. These will be included in the policy so
# that they can be sent with any value. S3 will ignore them.
# @option options [Hash] :conditions upload condition. Valid example:
# key starts with "image/"
# * :conditions => {:key => [["starts-with", "$key", "image/"]]}
# @return [Core::Response]
def post_object options
bucket = options.delete(:bucket)
raise TypeError, "Bucket instance required for :bucket!" if not bucket.instance_of?(AWS::S3::Bucket)
verbose = config.http_wire_trace
file = options.delete(:file)
raised = options.delete(:raised) || true
form = bucket.presigned_post(options)
fields = form.fields
# Multipart form data post by curb
url = form.url.to_s
curl = Curl::Easy.new(url)
curl.ssl_verify_peer = false
curl.multipart_form_post = true
curl.verbose = verbose
# Provide the ability outside to do something on curl and fields,
#-eg: modify policy to a wrong policy, add content_length field, etc.
yield(curl, fields) if block_given?
post_data = fields.map {|k, v| Curl::PostField.content(k, v.to_s)}
if file.instance_of? Array
file.each do |f|
post_data << Curl::PostField.file('file', f)
end
else
post_data << Curl::PostField.file('file', file) if file
end
# New Core::Response
resp = AWS::Core::Response.new(AWS::Core::Http::Request.new,
AWS::Core::Http::Response.new)
resp.api_version = self.class::API_VERSION
resp.request_type = 'POST'
# Callback to process header
curl.on_header do |header_data|
if header_data =~ /:\s+/
name, value = header_data.strip.split(/:\s+/, 2)
resp.http_response.headers[name] = value
end
header_data.length
end
# Callback to process body
curl.on_complete do
resp.http_response.status = curl.response_code
resp.http_response.body = curl.body_str
resp.data = AWS::Core::XML::Parser.parse(curl.body_str) if curl.body_str
# Save conditions and post data
resp.data.update({'conditions' => form.send("generate_conditions"),
'fields' => fields,
'post_data' => post_data,
'file' => file})
# Debug the useful outputs
p resp.data if verbose
end
# Call http_post for mutipart form data uploaded
begin
curl.http_post(post_data)
rescue => e
raise e if raised
resp.http_response.network_error = e
end
curl.close
resp
end
end
end
end
下面我们把post object API加入到S3的对象里去。
AWS.config(...) s3 = AWS::S3::new client = s3.client #把post_object加入到client对象里 extend_s3(client) #通过client对象来访问S3 API client.put_object(...) client.post_object(:bucket => bucket_name, :key => object_name, :file => local_file_path)
总结
S3的自动化测试开发工作还是2014下半年的事情。对于ruby也是当时现学现用，感觉这语言太过于飘逸，写好的代码，一段时间不看就感觉有点生疏。可能还是用的时间不长的缘故。但ruby里的元编程是真心赞。目前还是用自己喜欢而熟悉的python搞测试框架开发。S3相关的工作去年就也告一个段落。现在把这部分开发代码整理出来，有需要的朋友可以拿过去直接用。