How to set up live-migration environment for Openstack with devstack
1) Install OS
1.1) Install Ubuntu 12.04
1.2) Prepare at least 2 hosts for live migration, the README is using 3 hosts, one nova-controller node and two nova-compute node
1.3) Suppose the host name for the README are as following:
1.3.1) nova-controller: devstack12/172.17.27.12
1.3.2) nova-compute1: devstack13/172.17.27.13
1.3.3) nova-compute2: devstack14/172.17.27.14
1.4) Make sure the three hosts are DNS resolvable by each other.
2) Install required packages for all hosts
2.1) apt-get update
2.2) apt-get install openssh-server
2.3) apt-get install -y git
2.4) apt-get install libxml2-dev -y libxslt-dev
2.5) apt-get install -y libmysqlclient-dev
3) Check out devstack with the version of Essex
2.1) cd /root
2.2) git clone -b "stable/essex" git://github.com/openstack-dev/devstack.git
2.3) A folder named as devstack will be created under /root/devstack
4) Configure localrc for both nova-controller and nova-compute node
4.1) nova-controller node
4.1.1) Logon to nova-controller node
4.1.2) cd /root/devstack
4.1.3) vim localrc
4.2) nova-compute node
4.2.1) Logon to nova-compute node (Need to configure for both of the two nova-compute nodes)
4.2.2) cd /root/devstack
4.2.3) vim localrc
5) Install devstack on both nova-controller node and nova-compute node
6) Once install finished for both nova-controller node and nova-compute node, check if all nova service are happy face.
7) Configure NFS for live migration, you can create the NFS server anywhere as you like, for this README, we are creagting it on nova-controller node.
7.1) Condfigure nova-controller node as NFS server
7.1.1) apt-get install -y nfs-kernel-server
7.1.2) mkdir -p /export/instances/
7.1.3) chmod 777 -R /export/
7.1.4) mkdir -p /dev/md1
7.1.5) mkdir -p /raid1-1
7.1.7) vim /etc/default/nfs-kernel-server
NEED_SVCGSSD=no # no is default
7.1.8) vim /etc/default/nfs-common
NEED_IDMAPD=yes
NEED_GSSD=no # no is default
7.1.9) vim /etc/idmapd.conf
[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup
7.1.10) vim /etc/idmapd.conf -- only for JNDI
[Translation]
Method = nsswitch
7.1.11) sudo vim /etc/exports
/export 172.17.27.0/255.255.255.0(rw,sync,fsid=0,crossmnt,no_subtree_check)
/export/instances 172.17.27.0/255.255.255.0(rw,sync,no_subtree_check)
7.1.12) exportfs -ra
7.1.13) /etc/init.d/nfs-kernel-server restart
7.1.14) /etc/init.d/idmapd restart
7.2) Configure each nova-compute node as NFS client
7.2.1) apt-get install -y nfs-common
7.2.2) vim /etc/default/nfs-common
NEED_IDMAPD=yes
NEED_GSSD=no # no is default
7.2.3) modprobe nfs
7.2.4) mkdir -p /export/instances
7.2.5) vim /etc/fstab
172.17.27.12:/instances /export/instances nfs4 rw,hard,intr,rsize=32768,wsize=32768,nosuid,nodev 0 0
7.2.6) vim /etc/hosts.deny
rpcbind : ALL
7.2.7) vim /etc/hosts.allow
rpcbind : 172.17.27.12
7.2.8) mount -a -v
7.2.9) df -k
7.2.10) mount -t nfs4 172.17.27.12:/instances /export/instances
8) Config libvirt for nova-compute, this configure need to apply to all nova-compute nodes
8.1) vim /etc/libvirt/libvirtd.conf
8.1.1) Change "#listen_tls = 0" to "listen_tls = 0"
8.1.2) Change "#listen_tcp = 1" to "listen_tcp = 1"
8.1.3) Add "auth_tcp = "none""
8.2) vim /etc/init/libvirt-bin.conf
8.2.1) Change "env libvirtd_opts="-d"" to "env libvirtd_opts="-d -l""
8.3) vim /etc/default/libvirt-bin
8.3.1) Change "libvirtd_opts="-d"" to "libvirtd_opts="-d -l""
8.4) vim /etc/libvirt/qemu.conf
8.4.1) Change "#user = "root"" to "user = "root""
8.4.2) Change "#group = "root"" to "group = "root""
8.5) Restart libvirtd
stop libvirt-bin && start libvirt-bin
9) Update configuration for nova-compute nodes
9.1) copy /opt/stack/nova/etc/policy.json to /root
9.2) Add "state_path=/export" to /etc/nova/nova.conf
10) Restart nova-compute service for all nova-compute service.
10.1) ps -ef | gerp nova-compute
10.2) kill nova-compute
11.1) Create a new instance
11.2) Migrate the instance
1.1) Install Ubuntu 12.04
root@nova:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"1.2) Prepare at least 2 hosts for live migration, the README is using 3 hosts, one nova-controller node and two nova-compute node
1.3) Suppose the host name for the README are as following:
1.3.1) nova-controller: devstack12/172.17.27.12
1.3.2) nova-compute1: devstack13/172.17.27.13
1.3.3) nova-compute2: devstack14/172.17.27.14
1.4) Make sure the three hosts are DNS resolvable by each other.
2) Install required packages for all hosts
2.1) apt-get update
2.2) apt-get install openssh-server
2.3) apt-get install -y git
2.4) apt-get install libxml2-dev -y libxslt-dev
2.5) apt-get install -y libmysqlclient-dev
3) Check out devstack with the version of Essex
2.1) cd /root
2.2) git clone -b "stable/essex" git://github.com/openstack-dev/devstack.git
2.3) A folder named as devstack will be created under /root/devstack
4) Configure localrc for both nova-controller and nova-compute node
4.1) nova-controller node
4.1.1) Logon to nova-controller node
4.1.2) cd /root/devstack
4.1.3) vim localrc
HOST_IP=172.17.27.12 (Your nova-controller IP)
FLAT_INTERFACE=eth0
FIXED_RANGE=10.4.128.0/20
FIXED_NETWORK_SIZE=4096
FLOATING_RANGE=172.17.27.255/24
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
MYSQL_PASS=supersecret
RABBIT_PASS=supersecrete
SERVICE_TOKEN=xyzpdqlazydog
ADMIN_PASSWORD=nova
MYSQL_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_PASSWORD=nova
STATE_PATH=/export
LOCK_PATH=/tmp4.2) nova-compute node
4.2.1) Logon to nova-compute node (Need to configure for both of the two nova-compute nodes)
4.2.2) cd /root/devstack
4.2.3) vim localrc
HOST_IP=172.17.27.13 (Your nova-compute node IP)
FLAT_INTERFACE=eth0
FIXED_RANGE=10.4.128.0/20
FIXED_NETWORK_SIZE=4096
FLOATING_RANGE=172.17.27.255/24
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=nova
MYSQL_PASS=supersecret
RABBIT_PASS=supersecrete
MYSQL_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_PASSWORD=nova
SERVICE_TOKEN=xyzpdqlazydog
MYSQL_HOST=172.17.27.12 (Your nova-controller IP)
RABBIT_HOST=172.17.27.12
GLANCE_HOSTPORT=172.17.27.12:9292
ENABLED_SERVICES=n-cpu,n-net,n-api,n-vol
STATE_PATH=/export
LOCK_PATH=/tmp5) Install devstack on both nova-controller node and nova-compute node
6) Once install finished for both nova-controller node and nova-compute node, check if all nova service are happy face.
root@devstack12:~# nova-manage service list
2012-08-15 15:22:34 DEBUG nova.utils [req-ac296f2f-cf31-43b3-8b2c-bc140cc81b9b None None] backend from (pid=21252) __get_backend /opt/stack/nova/nova/utils.py:658
Binary Host Zone Status State Updated_At
nova-compute devstack12 nova enabled :-) 2012-08-16 02:10:00
nova-cert devstack12 nova enabled :-) 2012-08-16 02:09:58
nova-volume devstack12 nova enabled :-) 2012-08-16 02:09:58
nova-network devstack12 nova enabled :-) 2012-08-16 02:09:58
nova-scheduler devstack12 nova enabled :-) 2012-08-16 02:10:05
nova-consoleauth devstack12 nova enabled :-) 2012-08-16 02:10:02
nova-compute devstack14 nova enabled :-) 2012-08-16 02:09:47
nova-volume devstack14 nova enabled :-) 2012-08-16 02:09:52
nova-network devstack14 nova enabled :-) 2012-08-16 02:09:46
nova-compute devstack13 nova enabled :-) 2012-08-16 02:09:49
nova-volume devstack13 nova enabled :-) 2012-08-16 02:09:50
nova-network devstack13 nova enabled :-) 2012-08-16 02:09:49 7) Configure NFS for live migration, you can create the NFS server anywhere as you like, for this README, we are creagting it on nova-controller node.
7.1) Condfigure nova-controller node as NFS server
7.1.1) apt-get install -y nfs-kernel-server
7.1.2) mkdir -p /export/instances/
7.1.3) chmod 777 -R /export/
7.1.4) mkdir -p /dev/md1
7.1.5) mkdir -p /raid1-1
7.1.6)
vim /etc/fstab
/dev/md1 /raid1-1 ext3 data=journal,relatime,nosuid,noexec,nodev 1 2
/raid1-1/instances /export/instances none bind 0 07.1.7) vim /etc/default/nfs-kernel-server
NEED_SVCGSSD=no # no is default
7.1.8) vim /etc/default/nfs-common
NEED_IDMAPD=yes
NEED_GSSD=no # no is default
7.1.9) vim /etc/idmapd.conf
[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup
7.1.10) vim /etc/idmapd.conf -- only for JNDI
[Translation]
Method = nsswitch
7.1.11) sudo vim /etc/exports
/export 172.17.27.0/255.255.255.0(rw,sync,fsid=0,crossmnt,no_subtree_check)
/export/instances 172.17.27.0/255.255.255.0(rw,sync,no_subtree_check)
7.1.12) exportfs -ra
7.1.13) /etc/init.d/nfs-kernel-server restart
7.1.14) /etc/init.d/idmapd restart
7.2) Configure each nova-compute node as NFS client
7.2.1) apt-get install -y nfs-common
7.2.2) vim /etc/default/nfs-common
NEED_IDMAPD=yes
NEED_GSSD=no # no is default
7.2.3) modprobe nfs
7.2.4) mkdir -p /export/instances
7.2.5) vim /etc/fstab
172.17.27.12:/instances /export/instances nfs4 rw,hard,intr,rsize=32768,wsize=32768,nosuid,nodev 0 0
7.2.6) vim /etc/hosts.deny
rpcbind : ALL
7.2.7) vim /etc/hosts.allow
rpcbind : 172.17.27.12
7.2.8) mount -a -v
7.2.9) df -k
7.2.10) mount -t nfs4 172.17.27.12:/instances /export/instances
8) Config libvirt for nova-compute, this configure need to apply to all nova-compute nodes
8.1) vim /etc/libvirt/libvirtd.conf
8.1.1) Change "#listen_tls = 0" to "listen_tls = 0"
8.1.2) Change "#listen_tcp = 1" to "listen_tcp = 1"
8.1.3) Add "auth_tcp = "none""
8.2) vim /etc/init/libvirt-bin.conf
8.2.1) Change "env libvirtd_opts="-d"" to "env libvirtd_opts="-d -l""
8.3) vim /etc/default/libvirt-bin
8.3.1) Change "libvirtd_opts="-d"" to "libvirtd_opts="-d -l""
8.4) vim /etc/libvirt/qemu.conf
8.4.1) Change "#user = "root"" to "user = "root""
8.4.2) Change "#group = "root"" to "group = "root""
8.5) Restart libvirtd
stop libvirt-bin && start libvirt-bin
9) Update configuration for nova-compute nodes
9.1) copy /opt/stack/nova/etc/policy.json to /root
9.2) Add "state_path=/export" to /etc/nova/nova.conf
10) Restart nova-compute service for all nova-compute service.
10.1) ps -ef | gerp nova-compute
10.2) kill nova-compute
10.3) Start up nova-compute: (You can put "setsid" at the begining of the command to make it run as a daemon)
python /opt/stack/nova/bin/nova-compute --flagfile=/etc/nova/nova.conf11.1) Create a new instance
root@devstack12:/export/instances# euca-run-instances -t m1.tiny ami-00000003 --availability-zone xxx:devstack13
RESERVATION r-xpwbbzlv 9a2f4355c3b94cfe9a37ad2e8d6cc60d default
INSTANCE i-00000006 ami-00000003 server-6 server-6 pending None (9a2f4355c3b94cfe9a37ad2e8d6cc60d, None) 0 m1.tiny 2012-08-15T19:43:33.000Z unknown zone aki-00000001 ari-00000002 monitoring-disabled instance-store
root@devstack12:/export/instances# euca-describe-instances
RESERVATION r-xpwbbzlv 9a2f4355c3b94cfe9a37ad2e8d6cc60d default
INSTANCE i-00000006 ami-00000003 server-6 server-6 running None (9a2f4355c3b94cfe9a37ad2e8d6cc60d, devstack13) 0 m1.tiny 2012-08-15T19:43:33.000Z nova aki-00000001 ari-00000002 monitoring-disabled 10.4.128.2 10.4.128.2 instance-store
root@devstack12:/export/instances# nova list
+--------------------------------------+----------+--------+--------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+--------------------+
| f3a44421-39a3-477a-9b6d-819d98557953 | Server 6 | ACTIVE | private=10.4.128.2 |
+--------------------------------------+----------+--------+--------------------+
root@devstack12:/export/instances# nova show f3a44421-39a3-477a-9b6d-819d98557953
+-------------------------------------+----------------------------------------------------------+
| Property | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-SRV-ATTR:host | devstack13 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000006 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2012-08-15T19:43:33Z |
| flavor | m1.tiny |
| hostId | 2a2355e6f881cedd4e638be9b32970b804bd1f05db4582de2ae2730f |
| id | f3a44421-39a3-477a-9b6d-819d98557953 |
| image | cirros-0.3.0-x86_64-uec |
| key_name | |
| metadata | {} |
| name | Server 6 |
| private network | 10.4.128.2 |
| progress | 0 |
| status | ACTIVE |
| tenant_id | 9a2f4355c3b94cfe9a37ad2e8d6cc60d |
| updated | 2012-08-15T19:43:32Z |
| user_id | c604b35cce7e4b53a24cb3828c4b197c |
+-------------------------------------+----------------------------------------------------------+11.2) Migrate the instance
root@devstack12:/export/instances# nova live-migration f3a44421-39a3-477a-9b6d-819d98557953 devstack1411.3) Check the instance
root@devstack12:/export/instances# nova show f3a44421-39a3-477a-9b6d-819d98557953
+-------------------------------------+----------------------------------------------------------+
| Property | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-SRV-ATTR:host | devstack14 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000006 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2012-08-15T19:43:33Z |
| flavor | m1.tiny |
| hostId | 6a4b39673a99492bd2c461e5d0f616181d77fa83c9d3b3f44a890ff6 |
| id | f3a44421-39a3-477a-9b6d-819d98557953 |
| image | cirros-0.3.0-x86_64-uec |
| key_name | |
| metadata | {} |
| name | Server 6 |
| private network | 10.4.128.2 |
| progress | 0 |
| status | ACTIVE |
| tenant_id | 9a2f4355c3b94cfe9a37ad2e8d6cc60d |
| updated | 2012-08-15T19:44:23Z |
| user_id | c604b35cce7e4b53a24cb3828c4b197c |
+-------------------------------------+----------------------------------------------------------+