[置顶] OpenStack使用Bosh部署CloudFoundry(四)—使用Micro Bosh部署Bosh

版本说明：

CloudFoundry：V2版本-v149+（本文后续章节中的Manifest适用于149+版本的CF Release）

OpenStack：Grizzly版本+nova-network

CloudFoundry部署交流QQ群：176302388

本文介绍如何使用Micro Bosh来部署Bosh，部署需要创建8个虚拟机来部署8个Bosh组件，所以，至少得准备好8个Fixed IP，后边使用Bosh的时候需要访问Director组件及配置DNS，所以准备至少2个Floating IP，本文中为（10.68.19.131和10.68.19.132），Bosh组件的详细介绍可以参看下边连接。

BOSH组件介绍：http://docs.cloudfoundry.com/docs/running/bosh/components/

下面是部署步骤：

1、上传Bosh Stemcell到Micro Bosh中

root@bosh-cli:~/bosh-workspace/stemcells# bosh upload stemcell bosh-stemcell-latest-openstack-kvm-ubuntu.tgz

Verifying stemcell...
File exists and readable                                     OK
Using cached manifest...
Stemcell properties                                          OK

Stemcell info
-------------
Name:    bosh-openstack-kvm-ubuntu
Version: 1478

Checking if stemcell already exists...
No

Uploading stemcell...

latest-bosh-s: 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| 251.3MB  11.3MB/s Time: 00:00:22

Director task 1

Update stemcell
  extracting stemcell archive (00:00:06)
  verifying stemcell manifest (00:00:00)
  checking if this stemcell already exists (00:00:00)
  uploading stemcell bosh-stemcell/776 to the cloud (00:01:53)
  save stemcell bosh-stemcell/776 (f207bfcd-f0cb-4ea2-b753-022e6234ebfa) (00:00:00)
Done                    5/5 00:01:59

Task 1 done
Started         2013-12-16 09:51:59 UTC
Finished        2013-12-16 09:53:58 UTC
Duration        00:01:59

Stemcell uploaded and created

2、下载Bosh Release代码并打包

root@bosh-cli:~# cd ~/bosh-workspace
root@bosh-cli:~# git clone git://github.com/cloudfoundry/bosh.git

耗费时间较长，请耐心等待。

root@bosh-cli:~# cd ~/bosh-workspace/bosh
root@bosh-cli:~# apt-get install libxslt-dev libxml2-dev libmysql-ruby libmysqlclient-dev libpq-dev
root@bosh-cli:~# bundle install --local
root@bosh-cli:~# bundle exec rake release:create_dev_release

可以先修改~/bosh-workspace/bosh/Gemfile的source为淘宝源：

source 'http://ruby.taobao.org'

打包结束得到以下输出：

bosh create release --force
Syncing blobs...

Building DEV release
---------------------------------

....

Release version: 13.1-dev
Release manifest: /root/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml

3、上传Bosh Release包到Micro Bosh中

root@bosh-cli:~/bosh-workspace# bosh upload release ~/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml

成功后得到以下输出：

....

Creating new jobs
  blobstore/3.1-dev (00:00:00)
  powerdns/3.1-dev (00:00:00)
  redis/3 (00:00:00)
  nats/5 (00:00:00)
  postgres/4.1-dev (00:00:00)
  director/11.2-dev (00:00:00)
  health_monitor/5.1-dev (00:00:00)
  registry/0.1-dev (00:00:00)
Done                    8/8 00:00:00

Release has been created
  bosh/13.1-dev (00:00:00)
Done                    1/1 00:00:00

Task 2 done
Started         2013-12-16 10:06:26 UTC
Finished        2013-12-16 10:06:46 UTC
Duration        00:00:20

Release uploaded

4、确认已经上传的stemcell和release

root@bosh-cli:~# bosh stemcells

+---------------------------+---------+--------------------------------------+
| Name                      | Version | CID                                  |
+---------------------------+---------+--------------------------------------+
| bosh-openstack-kvm-ubuntu | 1478    | e64c6da5-f2ab-48f2-b03d-79c29d1acc43 |
+---------------------------+---------+--------------------------------------+

Stemcells total: 1
root@bosh-cli:~# bosh releases

+------+-----------+-------------+
| Name | Versions  | Commit Hash |
+------+-----------+-------------+
| bosh | 13.1-dev* | a8e43f41+   |
+------+-----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 1

5、创建部署Bosh的Manifest文件

创建目录

root@bosh-cli:~# mkdir -p ~/bosh-workspace/deployments/bosh-openstack
root@bosh-cli:~# cd ~/bosh-workspace/deployments/bosh-openstack

复制模板文件

root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-dynamic.yml bosh-openstack.yml

如果OpenStack中采用的是quantum网络组件，则使用下边的模板文件

root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-manual.yml bosh-openstack.yml

本次安装中使用nova-network网络组件，所以采用bosh-openstack-dynamic.yml，修改其中标记有# CHANGE 的配置项，下边是本环境中的配置，大家可参考：

---
name: bosh-openstack
director_uuid: 5d8c682c-290a-4194-bfc6-60248567ceab # CHANGE

release:
  name: bosh
  version: latest

compilation:
  workers: 3
  network: default
  reuse_compilation_vms: true
  cloud_properties:
    instance_type: v1.compile # CHANGE

update:
  canaries: 1
  canary_watch_time: 3000-120000
  update_watch_time: 3000-120000
  max_in_flight: 4
  max_errors: 1

networks:
  - name: floating
    type: vip
    cloud_properties: {}
  - name: default
    type: dynamic
    cloud_properties: {}

resource_pools:
  - name: common
    network: default
    size: 8
    stemcell:
      name: bosh-openstack-kvm-ubuntu
      version: latest
    cloud_properties:
      instance_type: v1.micro # CHANGE

jobs:
  - name: nats
    template: nats
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: redis
    template: redis
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: postgres
    template: postgres
    instances: 1
    resource_pool: common
    persistent_disk: 16384
    networks:
      - name: default
        default: [dns, gateway]

  - name: powerdns
    template: powerdns
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
      - name: floating
        static_ips:
          - 10.68.19.131 # CHANGE

  - name: blobstore
    template: blobstore
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: director
    template: director
    instances: 1
    resource_pool: common
    persistent_disk: 16384
    networks:
      - name: default
        default: [dns, gateway]
      - name: floating
        static_ips:
          - 10.68.19.132 # CHANGE

  - name: registry
    template: registry
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: health_monitor
    template: health_monitor
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

properties:
  nats:
    address: 0.nats.default.bosh-openstack.microbosh
    user: nats
    password: nats

  redis:
    address: 0.redis.default.bosh-openstack.microbosh
    password: redis

  postgres: &bosh_db
    host: 0.postgres.default.bosh-openstack.microbosh
    user: postgres
    password: postgres
    database: bosh

  dns:
    address: 10.68.19.131 # CHANGE
    db: *bosh_db
    recursor: 10.68.19.130 # CHANGE

  blobstore:
    address: 0.blobstore.default.bosh-openstack.microbosh
    agent:
      user: agent
      password: agent
    director:
      user: director
      password: director

  director:
    name: bosh
    address: 0.director.default.bosh-openstack.microbosh
    db: *bosh_db

  registry:
    address: 0.registry.default.bosh-openstack.microbosh
    db: *bosh_db
    http:
      user: registry
      password: registry

  hm:
    http:
      user: hm
      password: hm
    director_account:
      user: admin
      password: admin
    resurrector_enabled: true

  ntp:
    - 0.north-america.pool.ntp.org
    - 1.north-america.pool.ntp.org

  openstack:
    auth_url: http://10.68.19.61:5000/v2.0 # CHANGE  
    username: VF # CHANGE  
    api_key: 1 # CHANGE  
    tenant: VF_CF # CHANGE  
    region: myregion # CHANGE  
    default_security_groups: ["default"] # CHANGE  
    default_key_name: vkey # CHANGE

注意：director_uuid是Micro Bosh的Director的UUID，可以使用bosh status查看并复制到Manifest文件中。

6、部署Bosh

设置Bosh的部署文件

root@bosh-cli:~/bosh-workspace/deployments# bosh deployment ~/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml
Deployment set to `/root/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml'

执行部署命令

root@bosh-cli:~/bosh-workspace/deployments# bosh deploy

部署时可能会遇到错误RateLimit报错，报错内容如下：

E, [2013-12-15T06:13:32.599434 #6477] [task:3] ERROR -- : OpenStack API Request Entity Too Large error. Check task debug log for details.
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:20:in `cloud_error'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:39:in `rescue in with_openstack'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:25:in `with_openstack'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:512:in `block in set_vm_metadata'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:511:in `set_vm_metadata'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/metadata_helper.rb:14:in `update_vm_metadata'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:225:in `block (2 levels) in compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:290:in `prepare_vm'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:224:in `block in compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:89:in `block in with_compile_lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock.rb:58:in `lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:88:in `with_compile_lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:217:in `compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:205:in `block (2 levels) in process_task'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/event_log.rb:58:in `track'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:203:in `block in process_task'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:199:in `process_task'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:174:in `block (4 levels) in compile_packages'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `call'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `block (2 levels) in create_thread'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `loop'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `block in create_thread'

解决办法：请查看本人博客：http://blog.csdn.net/tiger435/article/details/8808623#t5，部署成功反馈：

root@bosh-cli:~/bosh-workspace/deployments# bosh deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully
Deploying `bosh-openstack.yml' to `microbosh-openstack' (type 'yes' to continue): yes

Director task 89

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  
....

Updating job nats
  nats/0 (canary) (00:00:29)
Done                    1/1 00:00:29

Updating job redis
  redis/0 (canary) (00:00:24)
Done                    1/1 00:00:24

Updating job postgres
  postgres/0 (canary) (00:00:44)
Done                    1/1 00:00:44

Updating job powerdns
  powerdns/0 (canary) (00:00:25)
Done                    1/1 00:00:25

Updating job blobstore
  blobstore/0 (canary) (00:00:25)
Done                    1/1 00:00:25

Updating job director
  director/0 (canary) (00:00:58)
Done                    1/1 00:00:58

Updating job registry
  registry/0 (canary) (00:00:33)
Done                    1/1 00:00:33

Updating job health_monitor
  health_monitor/0 (canary) (00:00:28)
Done                    1/1 00:00:28

Task 3 done
Started         2013-12-17 02:41:15 UTC
Finished        2013-12-17 02:57:08 UTC
Duration        00:15:53

Deployed `bosh-openstack.yml' to `microbosh-openstack'

8、验证部署是否成功

root@bosh-cli:~/bosh-workspace/deployments/bosh-openstack# bosh vms
Deployment `bosh-openstack'

Director task 90

Task 90 done

+------------------+---------+---------------+--------------------------+
| Job/index        | State   | Resource Pool | IPs                      |
+------------------+---------+---------------+--------------------------+
| blobstore/0      | running | common        | 50.50.0.11               |
| director/0       | running | common        | 50.50.0.10, 10.68.19.132 |
| health_monitor/0 | running | common        | 50.50.0.13               |
| nats/0           | running | common        | 50.50.0.5                |
| postgres/0       | running | common        | 50.50.0.7                |
| powerdns/0       | running | common        | 50.50.0.9, 10.68.19.131  |
| redis/0          | running | common        | 50.50.0.6                |
| registry/0       | running | common        | 50.50.0.12               |
+------------------+---------+---------------+--------------------------+

VMs total: 8

如上，所有Job的状态都是running，则部署成功。