安装openstack对象存储swift

修改Vagrantfile文件如下

#_*_ mode: ruby _*_
#vi:set ft=ruby :
nodes={
        'controller' =>[1,200],
        'compute' => [1,201],
        'swift' => [1,210],
}
Vagrant.configure("2") do |config|
        config.vm.box="/root/vagrant/box/precise64.box"
        #Forescout NAC workaround
        config.vm.usable_port_range=2800..2900
        nodes.each do |prefix, (count, ip_start)|
                count.times do |i|
                        hostname="%s" % [prefix,(i+1)]
                        config.vm.define "#{hostname}" do |box|
                                box.vm.hostname="#{hostname}.book"
                                box.vm.network:"private_network",ip:
                                        "172.16.0.#{ip_start+i}",:netmask=>"255.255.0.0"
                                box.vm.network:"private_network",ip:
                                        "10.10.0.#{ip_start+i}",:netmask=>"255.255.0.0"
                                #Otherwise using VirtualBox
                                box.vm.provider :virtualbox do |vbox|
                                        #Defaults
                                        vbox.customize ["modifyvm",:id,"--memory",1024]
                                        if prefix == "swift"
                                                vbox.customize ["modifyvm",:id,"--memory",1024]
                                                vbox.customize ["modifyvm",:id,"--cpus",1]
                                                vbox.customize ["createhd","--filename",'swift_disk2.vdi',"--size",200*1024]
                                                vbox.customize ["storageattach",:id,'--storagectl','SATA Controller', '--port',1,'--device',0,'--type','hdd','--medium','swift_disk2.vdi']
                                        end
                                end
                        end
                end
        end
end
                                

然后使用vagrant up swift命令来启动这个虚拟机,并且用vagrant ssh swift登录到这个虚拟机。
首先参考这里添加ubuntu的cloud-archive。
安装必要的软件包

sudo apt-get install -y swift swift-proxy swift-account swift-container swift-object memcached xfsprogs curl python-webob ntp parted

然后编辑/etc/ntp.conf文件，修改下面的部分

server ntp.ubuntu.com
server 127.127.1.0
fudge 127.127.1.0 stratum 10

然后vagrant@swift:~$ sudo service ntp restart重启ntp服务
#创建模拟设备
使用vagrant@swift:~$ sudo fdisk /dev/sdb格式化磁盘
按顺序按下以下按键：
n——添加一个新分区
p——打印分区表
1——1号分区
enter——设置第一个sector的大小为默认2048mb

enter——设置最后一个sector的大小为默认值419430399mb
w——将分区表写入磁盘并退出
然后执行vagrant@swift:~$ sudo partprobe可以不重启就能看到新分区，然后使用下面的命令为新创建的分区创建xfs文件系统：

vagrant@swift:~$ sudo mkfs.xfs -i size=1024 /dev/sdb1

创建挂载点vagrant@swift:~$ sudo mkdir /mnt/sdb1
然后在文件/etc/fstab中添加下面的行

/dev/sdb1      /mnt/sdb1        xfs    noatime,nodiratime,nobarrier,logbufs=8 0 0

挂载这个区域

vagrant@swift:~$ sudo mount /dev/sdb1

接下来创建需要的文件结构：

vagrant@swift:~$ sudo mkdir /mnt/sdb1/{1..4}
vagrant@swift:~$ sudo chown swift:swift /mnt/sdb1/*
vagrant@swift:~$ sudo ln -s /mnt/sdb1/{1..4} /srv
vagrant@swift:~$ sudo mkdir -p /etc/swift/{object-server,container-server,account-server}
vagrant@swift:~$ for S in {1..4};do sudo mkdir -p /srv/${S}/node/sdb${S};done
vagrant@swift:~$ sudo mkdir -p /var/run/swift
vagrant@swift:~$ sudo chown -R swift:swift /etc/swift /srv/{1..4}/

为了保证openstack对象存储能在系统引导时启动，在/etc/rc.local文件中exit 0行前添加如下命令：

mkdir -p /var/run/swift
chown swift:swift /var/run/swift

上面这些代码创建的目录结构如下：

/etc/swift
    /object-server
    /container-server
    /account-server
/mnt/sdb1
    /1       /srv/1
    /2       /srv/2
    /3       /srv/3
    /4       /srv/4
/srv/1/node/sdb1
/srv/2/node/sdb2
/srv/3/node/sdb3
/srv/4/node/sdb4
/var/run/swift

在生产环境中，上面创建的这些设备文件目录会被物理服务器和服务器上的物理设备取代，并不需要这些目录结构。

#配置同步
创建/etc/rsyncd.conf文件，内容如下：

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 127.0.0.1

[account6012]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/account6012.lock

[account6022]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/account6022.lock

[account6032]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/account6032.lock

[account6042]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/account6042.lock

[container6011]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/account6011.lock

[container6021]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/account6021.lock

[container6031]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/account6031.lock

[container6041]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/account6041.lock

[object6010]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/account6010.lock

[object6020]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/account6020.lock

[object6030]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/account6030.lock

[object6040]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/account6040.lock

启用rsync并启动该服务

vagrant@swift:~$ sudo sed -i 's/=false/=true/' /etc/default/rsync
vagrant@swift:~$ sudo service rsync start

#配置openstack对象存储
首先生成一个随机数

vagrant@swift:~$ < /dev/urandom tr -dc A-Za-z0-9_ |head -c16;echo
w0uDYG_rbxMo3jd3

然后创建/etc/swift/swift.conf文件，填入以下内容

[swift-hash]
#Random unique string used on all nodes
swift_hash_path_suffix=w0uDYG_rbxMo3jd3

其中w0uDYG_rbxMo3jd3是上面一步生成的随机字符串。
##配置openstack对象存储代理服务器
首先创建/etc/swift/proxy-server.conf文件，内容如下：

[DEFAULT]
bind_port=8080
user=swift
swift_dir = /etc/swift

[pipeline:main]
#Order of execution of modules defined below
pipeline = catch_errors healthcheck cache authtoken keystone proxy-server

[app:proxy-server]
use=egg:swift#proxy
allow_account_management=true
account_autocreate=true
set log_name=swift-proxy
set log_facility=LOG_LOCAL0
set log_level=INFO
set access_log_name=swift-proxy
set access_log_facility=SYSLOG
set access_log_level=INFO
set log_headers=True

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:cache]
use = egg:swift#memcache
set log_name = cache

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_protocol = http
auth_host = 172.16.0.200
auth_port = 35357
auth_token = ADMIN
service_protocol = http
service_host = 172.16.0.200
service_port = 5000
admin_token =ADMIN
admin_tenant_name = service
admin_user = swift
admin_password = openstack
delay_auth_decision = 0
signing_dir = /tmp/keystone-signing-swift

[filter:keystone]
use = egg:swift#keystoneauth
operator_roles = admin,swiftoperator

##配置账户服务器
为第一个节点创建一个初始账户服务器配置文件/etc/swift/account-server/1.conf内容如下：

[DEFAULT]
devices = /srv/1/node
mount_check = false
bind_port = 6012
user = swift
log_facility =  LOG_LOCAL2

[pipeline:main]
pipeline=account-server

[app:account-server]
use = egg:swift#account

[account-replicator]
vm_test_mode = yes

[account-auditor]

[account-reaper]

再利用该文件创建其余三个节点：

vagrant@swift:~$ cd /etc/swift/account-server/
vagrant@swift:/etc/swift/account-server$ sed -e "s/srv\/1/srv\/2/" -e "s/601/602/" -e "s/LOG_LOCAL2/LOG_LOCAL3/" 1.conf|sudo tee -a 2.conf
vagrant@swift:/etc/swift/account-server$ sed -e "s/srv\/1/srv\/3/" -e "s/601/603/" -e "s/LOG_LOCAL2/LOG_LOCAL4/" 1.conf|sudo tee -a 3.conf
vagrant@swift:/etc/swift/account-server$ sed -e "s/srv\/1/srv\/4/" -e "s/601/604/" -e "s/LOG_LOCAL2/LOG_LOCAL5/" 1.conf|sudo tee -a 4.conf

##配置容器服务器
首先为第一个节点创建配置文件/etc/swift/container-server/1.conf：

[DEFAULT]
devices = /srv/1/node
mount_check = false
bind_port = 6011
user = swift
log_facility = LOG_LOCAL2

[pipeline:main]
pipeline = container-server

[app:container-server]
use = egg:swift#container

[account-replicator]
vm_test_mode = yes

[account-updater]

[account-auditor]

[account-sync]

[container-sync]

[container-auditor]

[container-replicator]

[container-updater]

然后利用它创建其余三个节点的配置文件：

vagrant@swift:~$ cd /etc/swift/container-server
vagrant@swift:/etc/swift/container-server$ sed -e "s/srv\/1/srv\/2/" -e "s/601/602/" -e "s/LOG_LOCAL2/LOG_LOCAL3/" 1.conf|sudo tee -a 2.conf
vagrant@swift:/etc/swift/container-server$ sed -e "s/srv\/1/srv\/3/" -e "s/601/603/" -e "s/LOG_LOCAL2/LOG_LOCAL4/" 1.conf|sudo tee -a 3.conf
vagrant@swift:/etc/swift/container-server$ sed -e "s/srv\/1/srv\/4/" -e "s/601/604/" -e "s/LOG_LOCAL2/LOG_LOCAL5/" 1.conf|sudo tee -a 4.conf

##配置对象服务器

首先为第一个节点创建对象服务器的配置文件/etc/swift/object-server/1.conf如下：

[DEFAULT]
devices = /srv/1/node
mount_check = false
bind_port = 6010
user = swift
log_facility = LOG_LOCAL2

[pipeline:main]
pipeline = object-server

[app:object-server]
use = egg:swift#object

[object-replicator]
vm_test_mode = yes

[object-updater]

[object-auditor]

然后利用它创建其余三个节点的配置文件：

vagrant@swift:~$ cd /etc/swift/object-server/
vagrant@swift:/etc/swift/object-server$ sed -e "s/srv\/1/srv\/2/" -e "s/601/602/" -e "s/LOG_LOCAL2/LOG_LOCAL3/" 1.conf|sudo tee -a 2.conf
vagrant@swift:/etc/swift/object-server$ sed -e "s/srv\/1/srv\/3/" -e "s/601/603/" -e "s/LOG_LOCAL2/LOG_LOCAL4/" 1.conf|sudo tee -a 3.conf
vagrant@swift:/etc/swift/object-server$ sed -e "s/srv\/1/srv\/4/" -e "s/601/604/" -e "s/LOG_LOCAL2/LOG_LOCAL5/" 1.conf|sudo tee -a 4.conf

##创建环
首先创建脚本/usr/local/bin/remakerings：

#!/bin/bash

cd /etc/swift
rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz

#Object Ring
swift-ring-builder object.builder create 18 3 1
swift-ring-builder object.builder add z1-127.0.0.1:6010/sdb1 1
swift-ring-builder object.builder add z2-127.0.0.1:6020/sdb2 1
swift-ring-builder object.builder add z3-127.0.0.1:6030/sdb3 1
swift-ring-builder object.builder add z4-127.0.0.1:6040/sdb4 1
swift-ring-builder object.builder rebalance

#Container Ring
swift-ring-builder container.builder create 18 3 1
swift-ring-builder container.builder add z1-127.0.0.1:6011/sdb1 1
swift-ring-builder container.builder add z2-127.0.0.1:6021/sdb2 1
swift-ring-builder container.builder add z3-127.0.0.1:6031/sdb3 1
swift-ring-builder container.builder add z4-127.0.0.1:6041/sdb4 1
swift-ring-builder container.builder rebalance

#Account Ring
swift-ring-builder account.builder create 18 3 1
swift-ring-builder account.builder add z1-127.0.0.1:6012/sdb1 1
swift-ring-builder account.builder add z2-127.0.0.1:6022/sdb2 1
swift-ring-builder account.builder add z3-127.0.0.1:6032/sdb3 1
swift-ring-builder account.builder add z4-127.0.0.1:6042/sdb4 1
swift-ring-builder account.builder rebalance

执行这个脚本：

vagrant@swift:~$ sudo chmod +x /usr/local/bin/remakerings
vagrant@swift:~$ sudo /usr/local/bin/remakerings

然后执行下面的代码启动对象存储服务

vagrant@swift:~$ sudo swift-init main start

vagrant@swift:~$ sudo swift-init rest start

如果报错ImportError: No module named keystoneclient.middleware.auth_token
或报错ImportError: No module named middleware.auth_token，
参考这里添加ubuntu的cloud-archive。之后执行sudo apt-get install python-keystoneclien

报错：
LookupError: Entry point ‘keystoneauth’ not found in egg ‘swift’
解决方法：添加了cloud-archive之后重新执行sudo apt-get install -y swift swift-proxy swift-account swift-container swift-object memcached xfsprogs curl python-webob ntp parted，这会将之前安装的软件包更新到最新的版本。

报错：Unable to find container-sync config section in /etc/swift/container-server.conf
在/etc/swift/container-server.conf中添加一个空的[container-sync] section

##用openstack身份认证服务配置openstack对象存储
###使用keystone客户端配置openstack身份认证服务所需的端点（endpoint）和账户
配置环境变量

vagrant@swift:~$ export ENDPOINT=172.16.0.200
vagrant@swift:~$ export SERVICE_TOKEN=ADMIN
vagrant@swift:~$ export SERVICE_ENDPOINT=http://${ENDPOINT}:35357/v2.0
vagrant@swift:~$ export SWIFT_PROXY_SERVER=172.16.0.210

配置openstack对象存储endpoint

vagrant@swift:~$ keystone --token $SERVICE_TOKEN --endpoint $SERVICE_ENDPOINT service-create --name swift --type object-store --description 'Openstack Storage Service'

获取对象存储服务的id

vagrant@swift:~$ ID=$(keystone service-list|awk '/\ swift\ /{print $2}')
vagrant@swift:~$ PUBLIC_URL="https://$SWIFT_PROXY_SERVER:443/v1/AUTH_\$(tenant_id)s"
vagrant@swift:~$ ADMIN_URL="https://$SWIFT_PROXY_SERVER:443/v1"
vagrant@swift:~$ INTERNAL_URL=$PUBLIC_URL
vagrant@swift:~$ keystone endpoint-create --region RegionOne --service_id $ID --publicurl $PUBLIC_URL --adminurl $ADMIN_URL --internalurl $INTERNAL_URL

###配置端点指向openstack对象存储服务，然后设置 swift用户使得代理服务器可以使用openstack身份认证服务进行认证

vagrant@swift:~$ SERVICE_TENANT_ID=$(keystone tenant-list|awk '/\ service\ /{print $2}')
vagrant@swift:~$ keystone user-create --name swift --pass swift --tenant_id $SERVICE_TENANT_ID --email swift@localhost --enabled true
vagrant@swift:~$ USER_ID=$(keystone user-list|awk '/\ swift\ /{print $2}')
vagrant@swift:~$ ROLE_ID=$(keystone role-list|awk '/\ admin\ /{print $2}')
vagrant@swift:~$ keystone user-role-add --user $USER_ID --role $ROLE_ID --tenant_id $SERVICE_TENANT_ID

###在openstack对象存储服务器上安装keystone python库，以便使用openstack身份认证服务

vagrant@swift:~$ sudo apt-get update
vagrant@swift:~$ sudo apt-get install python-keystone

###验证代理服务器配置
编辑文件/etc/swift/proxy-server.conf如下：

[DEFAULT]
bind_port=443
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
user=swift
#swift_dir = /etc/swift
log_facility = LOG_LOCAL1

[pipeline:main]
#Order of execution of modules defined below
pipeline = catch_errors healthcheck cache authtoken keystone proxy-server

[app:proxy-server]
use=egg:swift#proxy
allow_account_management=true
#account_autocreate=true
#set log_name=swift-proxy
#set log_facility=LOG_LOCAL0
#set log_level=INFO
#set access_log_name=swift-proxy
#set access_log_facility=SYSLOG
#set access_log_level=INFO
#set log_headers=True

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:cache]
use = egg:swift#memcache
#set log_name = cache

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
auth_protocol = http
auth_host = 172.16.0.200
auth_port = 35357
auth_token = ADMIN
service_protocol = http
service_host = 172.16.0.200
service_port = 5000
admin_token =ADMIN
admin_tenant_name = service
admin_user = swift
admin_password = swift
#delay_auth_decision = 0
#signing_dir = /tmp/keystone-signing-swift
cache = swift.cache

[filter:keystone]
#use = egg:swift#keystoneauth
paste.filter_factory = keystone.middleware.swift_auth:filter_factory
operator_roles = Member,admin

[filter:swift3]
use = egg:swift#swift3

重新启动使这些更改生效
vagrant@swift:~$ sudo swift-init proxy-server restart

报错ImportError: No module named swift_auth
解决方法：将文件/etc/swift/proxy-server.conf中的swift_auth改为auth_token

##设置SSL访问

vagrant@swift:~$ cd /etc/swift
vagrant@swift:/etc/swift$ sudo openssl req -new -x509 -nodes -out cert.crt -keyout cert.key
Generating a 1024 bit RSA private key
.++++++
....................++++++
writing new private key to 'cert.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GB
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cookbook
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:172.16.0.2
Email Address []:

文件/etc/swift/proxy-server.conf里的下面三行配置代理服务器使用认证的证书和密钥

bind_port=443
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key

重新启动代理服务器

vagrant@swift:~$ sudo swift-init proxy-server restart

##测试openstack对象存储
vagrant@swift:~$ swift -A http://172.16.0.200:5000/v2.0 -U service:swift -K swift -V 2.0 stat
如果出现无输出的现象，解决方案为：
查看/var/log/syslog可发现下面内容

报错ConfigurationError: unable to access signing dir /home/vagrant/keystone-signing
解决sudo chown swift:swift /home/vagrant/keystone-signing

报错 #012OSError: [Errno 13] Permission denied: '/var/cache/swift/account.recon'
解决 sudo chown swift:swift /var/cache/swift

使用sudo swift-init proxy-server restart重启代理服务

执行swift -A http://172.16.0.200:5000/v2.0 -U service:swift -K swift -V 2.0 stat

报错Account HEAD failed: https://172.16.0.210:443/v1/AUTH_be8bc34a3ec44980aead6b68dcbdf03d 401 Unauthorized
查看log发现下面内容

Jan 28 12:25:24 swift proxy-server Retrying on HTTP connection exception: [Errno 111] ECONNREFUSED
Jan 28 12:25:26 swift proxy-server HTTP connection exception: [Errno 111] ECONNREFUSED
Jan 28 12:25:26 swift proxy-server Authorization failed for token faea4672fffe452390c6bdc1fe5e18bf
Jan 28 12:25:26 swift proxy-server Invalid user token - rejecting request

然而并没有什么乱用
试试reboot

python-keystoneclient/precise-proposed uptodate 1:0.2.3-0ubuntu2.2~cloud0

sudo add-apt-repository cloud-archive:grizzly
sudo apt-get remove python-keystoneclient
vagrant@swift:~$ sudo rm /etc/apt/sources.list.d/grizzly.list
vagrant@swift:~$ sudo rm /etc/apt/sources.list.d/grizzly.list.save
vagrant@swift:~$ sudo apt-get update
sudo apt-get install python-keystoneclient