Openstack(两控制节点+四计算节点)-1

      Openstack(两控制节点+四计算节点)系列四篇文章中openstack采用E版本，主要测试计算节点和控制节点宕机后的恢复过程，在计算节点采用iscsi模拟共享存储，在控制节点采用heartbeat+pacemaker监控控制节点的主要服务，所以控制节点是主备双机，需要设置vip，openstack的网络模式采用Flat模式。

主控制节点：10.1.6.186

备控制节点：10.1.6.188

heartbeat vip：10.1.6.100

计算节点：10.1.6.142、10.1.6.152、10.1.6.162、10.1.6.172

共享存储：10.1.6.39

先操作主control-node

1 安装网桥相关软件

apt-get install -y bridge-utils

2 修改网卡配置:/etc/network/interfaces 

auto lo
iface lo inet loopback

auto br100
iface br100 inet static
address 10.1.6.186
netmask 255.255.255.0
gateway 10.1.6.254
dns-nameservers 10.1.1.2

bridge_ports em1
bridge_hello 2
bridge_maxage 12
bridge_fd 0
bridge_stp off

3 重启网卡，使网桥配置生效

/etc/init.d/networking restart

4 作为控制节点需要安装NTP服务

apt-get install -y ntp

5 修改/etc/ntp.conf，在server ntp.ubuntu.com下添加如下内容：

server 127.127.1.0
fudge 127.127.1.0 stratum 10

6 重启ntp服务，使配置生效

/etc/init.d/ntp restart

7 安装消息队列、缓存、kvm等软件

apt-get install -y rabbitmq-server memcached python-memcache kvm libvirt-bin curl

8 安装iscsi、lvm2软件，在控制节点上volume会使用到(在本系列中属于选做)

apt-get install -y tgt lvm2

9 挑选一个分区或者盘做LVM卷(/dev/sda5)，卷名为nova-volumes(在本系列中属于选做)

pvcreate /dev/sda3
vgcreate nova-volumes /dev/sda3

10 安装mysql数据库，数据库管理员密码设置为:my_password

apt-get install -y mysql-server python-mysqldb

11 修改mysql配置文件:/etc/mysql/my.cnf

#监听0.0.0.0
bind-address            = 0.0.0.0
#启动binlog，为以后主从做好准备
server-id               = 1
log_bin                 = /var/log/mysql/mysql-bin.log

12 重启mysql使配置生效

/etc/init.d/mysql restart

13 创建openstack管理所需要的相关库:nova、keystone、glance

CREATE DATABASE nova;
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'my_password';
CREATE DATABASE glance;
GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY 'my_password';
CREATE DATABASE keystone;
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%'IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'vm.my.com' IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'vm.my.com' IDENTIFIED BY 'my_password';
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'vm.my.com'IDENTIFIED BY 'my_password';
FLUSH PRIVILEGES;

14 安装keystone相关组件

apt-get install -y keystone python-keystone python-keystoneclient

15 修改/etc/keystone/keystone.conf配置文件

[DEFAULT]
bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
admin_token = my_cloud

[sql]
connection = mysql://keystone:[email protected]/keystone

    期中配置vm.my.com通过DNS指向主控制节点上面heartbeat的vip地址上面

16 重启keystone服务

/etc/init.d/keystone restart

17 初始化keystone库

keystone-manage db_sync

18 为了安装设置keystone组件方便快捷，提前设置好环境变量

export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export SERVICE_PASSWORD=password
export FIXED_RANGE=10.1.6.0/24
export OS_AUTH_URL="http://vm.my.com:5000/v2.0/"
export SERVICE_ENDPOINT="http://vm.my.com:35357/v2.0"
export SERVICE_TOKEN=my_cloud
export MASTER="vm.my.com"
export OS_NO_CACHE=1

    为了避免每次ssh连接时需要设置环境变量，可以把以上设置加入.bashrc文件中

19 设置users and tenants and services，用脚本设置keystone.sh

#!/bin/bash -x
#
# Initial data for Keystone using python-keystoneclient
#
# Tenant               User      Roles
# ------------------------------------------------------------------
# admin                admin     admin
# service              glance    admin
# service              nova      admin, [ResellerAdmin (swift only)]
# service              quantum   admin        # if enabled
# service              swift     admin        # if enabled
# demo                 admin     admin
# demo                 demo      Member, anotherrole
# invisible_to_admin   demo      Member
#
# Variables set before calling this script:
# SERVICE_TOKEN - aka admin_token in keystone.conf
# SERVICE_ENDPOINT - local Keystone admin endpoint
# SERVICE_TENANT_NAME - name of tenant containing service accounts
# ENABLED_SERVICES - stack.sh's list of services to start
# DEVSTACK_DIR - Top-level DevStack directory


ADMIN_PASSWORD=${ADMIN_PASSWORD:-$OS_PASSWORD}
SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}
ENABLED_SERVICES="swift"

function get_id () {
    echo `$@ | awk '/ id / { print $4 }'`
}

# Tenants
ADMIN_TENANT=$(get_id keystone tenant-create --name=admin)
SERVICE_TENANT=$(get_id keystone tenant-create --name=$SERVICE_TENANT_NAME)


# Users
ADMIN_USER=$(get_id keystone user-create --name=admin \
                                         --pass="$ADMIN_PASSWORD" \
                                         [email protected])

# Roles
ADMIN_ROLE=$(get_id keystone role-create --name=admin)
MEMBER_ROLE=$(get_id keystone role-create --name=Member)
KEYSTONEADMIN_ROLE=$(get_id keystone role-create --name=KeystoneAdmin)
KEYSTONESERVICE_ROLE=$(get_id keystone role-create --name=KeystoneServiceAdmin)

# Add Roles to Users in Tenants
keystone user-role-add --user-id $ADMIN_USER --role-id $ADMIN_ROLE --tenant_id $ADMIN_TENANT

#keystone user-role-add --user $DEMO_USER --role $MEMBER_ROLE --tenant_id $DEMO_TENANT

# Configure service users/roles
NOVA_USER=$(get_id keystone user-create --name=nova \
                                        --pass="$SERVICE_PASSWORD" \
                                        --tenant_id $SERVICE_TENANT \
                                        [email protected])
keystone user-role-add --tenant_id $SERVICE_TENANT \
                       --user-id $NOVA_USER \
                       --role-id $ADMIN_ROLE

GLANCE_USER=$(get_id keystone user-create --name=glance \
                                          --pass="$SERVICE_PASSWORD" \
                                          --tenant_id $SERVICE_TENANT \
                                          [email protected])
keystone user-role-add --tenant_id $SERVICE_TENANT \
                       --user-id $GLANCE_USER \
                       --role-id $ADMIN_ROLE


if [[ "$ENABLED_SERVICES" =~ "swift" ]]; then
    SWIFT_USER=$(get_id keystone user-create --name=swift \
                                             --pass="$SERVICE_PASSWORD" \
                                             --tenant_id $SERVICE_TENANT \
                                             [email protected])
    keystone user-role-add --tenant_id $SERVICE_TENANT \
                           --user-id $SWIFT_USER \
                           --role-id $ADMIN_ROLE
    # Nova needs ResellerAdmin role to download images when accessing
    # swift through the s3 api. The admin role in swift allows a user
    # to act as an admin for their tenant, but ResellerAdmin is needed
    # for a user to act as any tenant. The name of this role is also
    # configurable in swift-proxy.conf
    RESELLER_ROLE=$(get_id keystone role-create --name=ResellerAdmin)
    keystone user-role-add --tenant_id $SERVICE_TENANT \
                           --user-id $NOVA_USER \
                           --role-id $RESELLER_ROLE
fi

if [[ "$ENABLED_SERVICES" =~ "quantum" ]]; then
    QUANTUM_USER=$(get_id keystone user-create --name=quantum \
                                               --pass="$SERVICE_PASSWORD" \
                                               --tenant_id $SERVICE_TENANT \
                                               [email protected])
    keystone user-role-add --tenant_id $SERVICE_TENANT \
                           --user-id $QUANTUM_USER \
                           --role-id $ADMIN_ROLE
fi

20 设置endpoint服务，使用endpoint.sh脚本设置

#!/bin/bash -x

# Author:       Martin Gerhard Loschwitz
# (c) 2012      hastexo Professional Services GmbH

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# 
# On Debian-based systems the full text of the Apache version 2.0 
# license can be found in `/usr/share/common-licenses/Apache-2.0'.

# MySQL definitions
MYSQL_USER=keystone
MYSQL_DATABASE=keystone
MYSQL_PASSWORD=my_password
MYSQL_HOST=vm.my.com
MASTER=vm.my.com

# Keystone definitions
KEYSTONE_REGION=RegionOne
SERVICE_ENDPOINT="http://vm.my.com:35357/v2.0"

# other definitions

while getopts "u:D:p:m:K:R:E:S:T:vh" opt; do
  case $opt in
    u)
      MYSQL_USER=$OPTARG
      ;;
    D)
      MYSQL_DATABASE=$OPTARG
      ;;
    p)
      MYSQL_PASSWORD=$OPTARG
      ;;
    m)
      MYSQL_HOST=$OPTARG
      ;;
    K)
      MASTER=$OPTARG
      ;;
    R)
      KEYSTONE_REGION=$OPTARG
      ;;
    E)
      export SERVICE_ENDPOINT=$OPTARG
      ;;
    S)
      SWIFT_MASTER=$OPTARG
      ;;
    T)
      export SERVICE_TOKEN=$OPTARG
      ;;
    v)
      set -x
      ;;
    h)
      cat <<EOF
Usage: $0 [-m mysql_hostname] [-u mysql_username] [-D mysql_database] [-p mysql_password]
       [-K keystone_master ] [ -R keystone_region ] [ -E keystone_endpoint_url ] 
       [ -S swift_master ] [ -T keystone_token ]
          
Add -v for verbose mode, -h to display this message.
EOF
      exit 0
      ;;
    \?)
      echo "Unknown option -$OPTARG" >&2
      exit 1
      ;;
    :)
      echo "Option -$OPTARG requires an argument" >&2
      exit 1
      ;;
  esac
done  

if [ -z "$KEYSTONE_REGION" ]; then
  echo "Keystone region not set. Please set with -R option or set KEYSTONE_REGION variable." >&2
  missing_args="true"
fi

if [ -z "$SERVICE_TOKEN" ]; then
  echo "Keystone service token not set. Please set with -T option or set SERVICE_TOKEN variable." >&2
  missing_args="true"
fi

if [ -z "$SERVICE_ENDPOINT" ]; then
  echo "Keystone service endpoint not set. Please set with -E option or set SERVICE_ENDPOINT variable." >&2
  missing_args="true"
fi

if [ -z "$MYSQL_PASSWORD" ]; then
  echo "MySQL password not set. Please set with -p option or set MYSQL_PASSWORD variable." >&2
  missing_args="true"
fi

if [ -n "$missing_args" ]; then
  exit 1
fi
 
keystone service-create --name nova --type compute --description 'OpenStack Compute Service'
keystone service-create --name volume --type volume --description 'OpenStack Volume Service'
keystone service-create --name glance --type image --description 'OpenStack Image Service'
keystone service-create --name swift --type object-store --description 'OpenStack Storage Service'
keystone service-create --name keystone --type identity --description 'OpenStack Identity'
keystone service-create --name ec2 --type ec2 --description 'OpenStack EC2 service'

create_endpoint () {
  case $1 in
    compute)
    keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s' --adminurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s' --internalurl 'http://'"$MASTER"':8774/v2/%(tenant_id)s'
    ;;
    volume)
    keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s' --adminurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s' --internalurl 'http://'"$MASTER"':8776/v1/%(tenant_id)s'
    ;;
    image)
    keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':9292/v1' --adminurl 'http://'"$MASTER"':9292/v1' --internalurl 'http://'"$MASTER"':9292/v1'
    ;;
    object-store)
    if [ $SWIFT_MASTER ]; then
      keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$SWIFT_MASTER"':8080/v1/AUTH_%(tenant_id)s' --adminurl 'http://'"$SWIFT_MASTER"':8080/v1' --internalurl 'http://'"$SWIFT_MASTER"':8080/v1/AUTH_%(tenant_id)s'
    else
      keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8080/v1/AUTH_%(tenant_id)s' --adminurl 'http://'"$MASTER"':8080/v1' --internalurl 'http://'"$MASTER"':8080/v1/AUTH_%(tenant_id)s'
    fi
    ;;
    identity)
    keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':5000/v2.0' --adminurl 'http://'"$MASTER"':35357/v2.0' --internalurl 'http://'"$MASTER"':5000/v2.0'
    ;;
    ec2)
    keystone endpoint-create --region $KEYSTONE_REGION --service_id $2 --publicurl 'http://'"$MASTER"':8773/services/Cloud' --adminurl 'http://'"$MASTER"':8773/services/Admin' --internalurl 'http://'"$MASTER"':8773/services/Cloud'
    ;;
  esac
}

for i in compute volume image object-store identity ec2; do
  id=`mysql -h "$MYSQL_HOST" -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE" -ss -e "SELECT id FROM service WHERE type='"$i"';"` || exit 1
  create_endpoint $i $id
done

21 验证设置是否正确

keystone tenant-list
keystone user-list
keystone role-list

22 安装glance服务

apt-get install -y glance glance-api glance-client glance-common glance-registry python-glance

23 修改配置文件/etc/glance/glance-api.conf 和/etc/glance/glance-registry.conf

#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = glance
admin_password = password

24 修改/etc/glance/glance-registry.conf使用mysql连接

sql_connection = mysql://glance:[email protected]/glance

25 修改/etc/glance/glance-registry.conf和/etc/glance/glance-api.conf

[paste_deploy]
flavor = keystone

26 重启glance相关服务

/etc/init.d/glance-api restart
/etc/init.d/glance-registry restart

27 初始化glance库

glance-manage version_control 0
glance-manage db_sync

28 再次重启glance相关服务

/etc/init.d/glance-api restart
/etc/init.d/glance-registry restart

29 测试glance，F版中会提示一个警告，E版中则无输出

glance index

WARNING! This tool is deprecated in favor of python-glanceclient (see http://github.com/openstack/python-glanceclient).

30 上传自制的Debian6镜像

glance add name="debian6 initrd" disk_format=qcow2 container_format=ovf is_public=true < initrd.img-2.6.32-5-amd64

#上面命令返回ID:9fd89cc3-c479-4544-aeae-6201bfbd504b

glance add name="debian6 vmlinuz" disk_format=qcow2 container_format=ovf is_public=true < vmlinuz-2.6.32-5-amd64

#上面命令返回ID:fd855e9b-465f-43d7-b2c1-73b8af289097

glance add name="debian6 OS" disk_format=qcow2 container_format=ovf is_public=true  ramdisk_id=9fd89cc3-c479-4544-aeae-6201bfbd504b kernel_id=fd855e9b-465f-43d7-b2c1-73b8af289097 < debian6.img

31 测试glance，glance index会看到上面三个镜像

glance index

    glance中存储镜像的本地存储目录用NFS代替，因为后面要模拟控制节点宕机。

32 安装nova组件

apt-get install -y nova-api nova-cert nova-common nova-objectstore nova-scheduler nova-volume nova-consoleauth  novnc python-nova python-novaclient nova-compute nova-compute-kvm nova-network

33 修改/etc/nova/api-paste.ini文件

#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = nova
admin_password = password

34 修改/etc/nova/nova.conf配置文件

[DEFAULT]
###### LOGS/STATE
#verbose=True
verbose=False

###### AUTHENTICATION
auth_strategy=keystone

###### SCHEDULER
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_driver=nova.scheduler.simple.SimpleScheduler

###### VOLUMES
volume_group=nova-volumes
volume_name_template=volume-%08x
iscsi_helper=tgtadm

###### DATABASE
sql_connection=mysql://nova:[email protected]/nova

###### COMPUTE
libvirt_type=kvm
#libvirt_type=qemu
connection_type=libvirt
instance_name_template=instance-%08x
api_paste_config=/etc/nova/api-paste.ini
allow_resize_to_same_host=True
libvirt_use_virtio_for_bridges=true
start_guests_on_host_boot=true
resume_guests_state_on_host_boot=true

###### APIS
osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
allow_admin_api=true
s3_host=vm.my.com
cc_host=vm.my.com

###### RABBITMQ
rabbit_host=vm.my.com

###### GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=vm.my.com:9292

###### NETWORK
network_manager=nova.network.manager.FlatManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
public_interface=em1
flat_interface=em1
flat_network_bridge=br100
fixed_range=10.1.6.0/24
multi_host=true

###### NOVNC CONSOLE
novnc_enabled=true
novncproxy_base_url= http://vm.my.com:6080/vnc_auto.html
vncserver_proxyclient_address=vm.my.com
vncserver_listen=vm.my.com

########Nova
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova

#####MISC
use_deprecated_auth=false
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

35 重启nova相关服务

service rabbitmq-server restart
service libvirt-bin restart
service nova-scheduler restart
service nova-network restart
service nova-cert restart
service nova-compute restart
service nova-api restart
service nova-objectstore restart
service nova-volume restart

36 初始化nova库，会有警告提示

nova-manage db sync

2012-11-10 10:28:04 WARNING nova.common.deprecated [-] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave.

37 创建fix ip

nova-manage network create private --fixed_range_v4=10.1.6.0/24 --num_networks=1 \
--bridge=br100 --bridge_interface=br100 --network_size=256 --multi_host=T

38 再次 重启nova相关服务

service rabbitmq-server restart
service libvirt-bin restart
service nova-scheduler restart
service nova-network restart
service nova-cert restart
service nova-compute restart
service nova-api restart
service nova-objectstore restart
service nova-volume restart

39 测试nova服务是否正常启动

nova-manage service list

40 停止不必要的服务

nova-manage service disable --host=control-1-186 --service=nova-compute
nova-manage service disable --host=control-1-186 --service=nova-volume
nova-manage service disable --host=control-1-186 --service=nova-network

40 安装horizon

apt-get install -y apache2 libapache2-mod-wsgi openstack-dashboard

41 安装vnc

apt-get install -y nova-novncproxy nova-xvpvncproxy novnc python-novnc

42 添加安全组策略

nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

43 访问dashboard(E版本中直接my.vm.com访问)

http://my.vm.com/horizon

44 在控制节点上停止nova-network和nova-compute服务，因为要让vm运行在计算节点

/etc/init.d/nova-network stop
/etc/init.d/nova-compute stop

45 跳转至第二章安装compute-1节点，并设置模拟存储配置

46 compute-1一切妥当后，转到control-node上面，查看有那些镜像和网络可以使用，在创建VM时需要提供相关ID

nova image-list
nova-manage network list

47 启动你的第一台VM

http://my.oschina.net/guol/blog/90134