Xen Live Migration with iSCSI
This is a document about how to set up XEN and iSCSI on 3 Debian Stable (3.1 / Sarge) machines so that you can have a virtual host that can be migrated live between 2 of the machines. The third machine will be the common disk for the virtual machine that is seen via the 2 machines running XEN. I have assumed you have some knowledge of Linux specifically Debian, but not limited to. You may need to solve some dependencies that I have left out by yourself. I can't think of everything, and I didn't start with completely empty machines.
In this example we will use dangermouse character names for the machine names. Buggles and Dangermouse will be the XEN hosts, and Penfold will house the disk that is common for both. Virtual01 will be the name we give to the virtual machine, that can be live migrated between dangermouse and buggles, using the disk on penfold over iSCSI. That was a mouthful, but don't worry we will break it down into what is hopefully simple steps.
Assuming you have already got running debian stable machines using the 2.6.8 kernel that was shipped with Debian, we are ready to move on to installing XEN from backports.org. This process needs to be done on all three machines.
Prepare the apt sources by adding the backports repositories to the file /etc/apt/sources.list
dangermouse:~# echo "deb http://www.backports.org/debian/ sarge-backports main" >> /etc/apt/sources
Now tell apt that we don't normally want to install packages from backports.org unless we specifically ask.
dangermouse:~# cat <<! >> /etc/apt/preferencesPackage: *Pin: release a=sarge-backportsPin-Priority: 200!
Update the apt cache:
dangermouse:~# apt-get update
Now we want to install the xen kernel and hypervisor on every machine. This is so we get a consistent 2.6.16 kernel on every machine including the machine we will use for the disk so we can use the latest version of iSCSI Target software from IET.
dangermouse:~# aptitude -t sarge-backports install linux-image-2.6.16-1-xen-686 linux-modules-2.6.16-1-xen-686 grub xen-hypervisor-3.0-i386 xen-utils-3.0 udev
There is probably a few other packages that I have left out that will get automatically updated as well. They should all be fine.
We need to disable tls as it is not supported by XEN at this time. There may be a libc686-xen package available, and I would suggest installing that if possible instead of disabling /lib/tls
.
dangermouse:~# mv /lib/tls /lib/tls.disabled
The Debian package of XEN kernel or hypervisor doesn't yet make an initrd image, so we need to make one manually.
dangermouse:~# mkinitrd -o /boot/initrd.img-2.6.16-1-xen-686 2.6.16-1-xen-686
Now prepare the boot loader. Firstly run update-grub to automatically populate /boot/grub/menu.1st
, then fixup the created XEN entries, by adding two module lines and removing the initrd line.
dangermouse:~# update-grub
turn this:
kernel /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 roinitrd /boot/initrd.img-2.6.16-1-xen-686
into this in the /boot/grub/menu.1st
:
kernel /boot/xen-3.0-i386.gz console=vga dom0_mem=256000module /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 ro xencons=offmodule /boot/initrd.img-2.6.16-1-xen-686
Replace /dev/hda1
with whatever is your root partition. dom0_mem is the amount of memory available to the host virtual machine or domain0. xencons turns off the use of the serial port. I need the serial ports for other purposes, so this one is up to you.
Now you need to reboot and confirm that the XEN hypervisor boots and domain0 starts as well. If this fails go over the steps and make sure you didn't leave a step out. Because you still have your Debian standard kerenl installed you can always recover to that with no ill affect
Now that you have XEN installed and you are successfully booting off the XEN Hypervisor we can configure the xen daemon. The configuration file is /etc/xen/xend-config.sxp.
(xend-relocation-server yes)(xend-relocation-hosts-allow '')(network-script 'network-bridge netdev=eth0')(vif-script vif-bridge)(dom0-min-mem 196)(dom0-cpus 0)
Now restart the xend process.
dangermouse:~# /etc/init.d/xend restart
We setup udev to make the iSCSI devices a consistent name across all the machines, and you don't end up with /dev/sda
on one machine and /dev/sdb
on another.
On each of the clients do the following:
dangermouse:~# echo "BUS==\"scsi\", SYSFS{vendor}==\"IET\", SYSFS{model}==\"VIRTUAL-DISK\", \KERNEL==\"sd*\", NAME=\"%k\", PROGRAM=\"/lib/udev/getlun.sh $id\", SYMLINK+=\"iscsi_%c%n\"" \>>/etc/udev/rules.d/10_persistant_scsi.rulesdangermouse:~# cat <<! >> /lib/udev/getlun.sh#!/bin/bashecho $1 | awk -F":" '{print $NF}'!
What this will do is create a link in /dev
called iscsi_01 for instance, where 0 is the LUN number, and 1 is the first partition, which might be a link to say /dev/sdb1
.
Now we need to set up the target on the machine which won't run the virtual machine. This is penfold.
I used a block device for this on a SATA raid controller, but you could just use an ordinary file or a normal disk, or logical volume. The iSCSI Target documentation will give you more information on setting up something other than a block device.
penfold:~# apt-get install linux-headers-2.6.16-1-xen-686 openssl libssl-dev
Get the latest IET source code from your nearest sourceforge mirror, then uncompress and extract the files from the archive ready to build.
penfold:~# cd /usr/srcpenfold:~# wget http://sf.net...../iscsitarget-0.4.13.tar.gzpenfold:~# tar -zxf iscsitarget-0.4.13.tar.gzpenfold:~# cd iscsitarget-0.4.13penfold:~# makepenfold:~# make install
edit /etc/ietd.conf
and make it look like the following.
IncomingUser joe secretOutgoingUser jack secret2 Target iqn.2006-05.com.performancemagic:penfold.disk1 IncomingUser jim othersecret OutgoingUser james yetanothersecret Lun 0 Path=/dev/sdb,Type=fileio Alias Test HeaderDigest None DataDigest None MaxConnections 1 InitialR2T Yes ImmediateData No MaxRecvDataSegmentLength 8192 MaxXmitDataSegmentLength 8192 MaxBurstLength 262144 FirstBurstLength 65536 DefaultTime2Wait 2 DefaultTime2Retain 20 MaxOutstandingR2T 8 DataPDUInOrder Yes DataSequenceInOrder Yes ErrorRecoveryLevel 0
Now start the iSCSI target.
penfold:~# /etc/init.d/iscsi-target startStarting iSCSI enterprise target service: succeeded.
(dmesg output abreviated)
penfold:~# dmesgiSCSI Enterprise Target Software - version 0.4.13iotype_init(90) register fileioiotype_init(90) register nulliosess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1sess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1
Now lets build and setup the iSCSI initiator.
dangermouse:~# cd /usr/srcdangermouse:~# svn co svn://svn.berlios.org/open-iscsi/
The revision I got was 581
dangermouse:~# cd open-iscsidangermouse:~# makedangermouse:~# make installdangermouse:~# echo "iqn.2006-05.com.performancemagic:dangermouse.01" > /etc/initiatorname.iscsi
Change dangermouse in the above example to buggles for the other XEN host.
Edit /etc/iscsid.conf
and make it look like the following.
node.active_cnx = 1node.startup = manualnode.session.auth.username = jimnode.session.auth.password = othersecretnode.session.timeo.replacement_timeout = 120node.session.err_timeo.abort_timeout = 10node.session.err_timeo.reset_timeout = 30node.session.iscsi.InitialR2T = Nonode.session.iscsi.ImmediateData = Yesnode.session.iscsi.FirstBurstLength = 262144node.session.iscsi.MaxBurstLength = 16776192node.session.iscsi.DefaultTime2Wait = 0node.session.iscsi.DefaultTime2Retain = 0node.session.iscsi.MaxConnections = 1node.conn[0].iscsi.HeaderDigest = Nonenode.conn[0].iscsi.DataDigest = Nonenode.conn[0].iscsi.MaxRecvDataSegmentLength = 65536discovery.sendtargets.auth.username = joediscovery.sendtargets.auth.password = secret
Start the initiator.
dangermouse:~# /etc/init.d/open-iscsi startStarting iSCSI initiator service: succeeded.
(dmesg output abreviated)
dangermouse:~# dmesgSCSI subsystem initializediscsi: registered transport (tcp)
Now we have the target setup and the initiator started, we need to find the targets on the initiator.
dangermouse:~# iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260[0f6cce] 127.0.0.1:3260,1 iqn.2006-05.com.performancemagic:penfold.disk1
now take the number in the [ ] brackets and login.
dangermouse:~# iscsiadm -m node -r 0f0cce -l
(dmesg output abreviated)
dangermouse:~# dmesgscsi0 : iSCSI Initiator over TCP/IP, v1.0-574sess_param(173) 1 0 1 8192 8192 262144 65536 2 0 1 1 1 0 1 1Vendor: IET Model: VIRTUAL-DISK Rev: 0Type: Direct-Access ANSI SCSI revision: 04SCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backSCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backsda: sda1sd 0:0:0:1: Attached scsi disk sda
Now that you can see the iSCSI device we can create partitions for the virtual machine to use. We just need to create the simplest of patitions, one for the root file-system of the virtual machine and one for swap. You can use fdisk or cfdisk for this, or any other partitioning programme. Your partitions in fdisk might look something like this:
dangermouse:~# fdisk /dev/iscsi_0Command (m for help): pDisk /dev/iscs_0: 60.0 GB, 60011642880 bytes255 heads, 63 sectors/track, 7296 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/iscsi_01 1 249 2000061 82 Linux swap / Solaris/dev/iscsi_02 250 7296 56605027+ 83 Linux
Now that we have created the partitions we need to create file-systems on them.
dangermouse:~# mkswap /dev/iscsi_01Setting up swapspace version 1, size = 5115 kBno label, UUID=055d2ab4-d2ab-4779-818c-71d4950c08dfdangermouse:~# mke2fs -j /dev/iscsi_02Filesystem label=OS type: LinuxBlock size=1024 (log=0)Fragment size=1024 (log=0)1256 inodes, 5000 blocks250 blocks (5.00%) reserved for the super userFirst data block=1Maximum filesystem blocks=52428801 block group8192 blocks per group, 8192 fragments per group1256 inodes per groupWriting inode tables: doneCreating journal (1024 blocks): doneWriting superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 28 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.
I am going to create a Debian virtual machine using debootstrap.
dangermouse:~# mount /dev/iscsi_02 /mntdangermouse:~# debootstrap sarge /mntI: Retrieving ReleaseI: Retrieving Packages......
Once debootstrap is completed we need to setup the machine a little bit. We need to disable tls again, setup the /etc/fstab
and the network interfaces and create tty and console devices. We also need to copy the kernel modules to the virtual machine.
dangermouse:~# mv /mnt/lib/tls /mnt/lib/tls.disableddangermouse:~# cat <<! > /mnt/etc/fstab# /etc/fstab: static file system information.## <file system> <mount point> <type> <options> <dump> <pass>proc /proc proc defaults 0 0/dev/hda1 none swap sw 0 0/dev/hda2 / ext3 defaults,errors=remount-ro 0 1!dangermouse:~# echo "virtaul01" > /mnt/etc/hostnamedangermouse:~# cat <<! >/mnt/etc/network/interfacesauto lo eth0iface lo inet loopbackiface eth0 inet static address 192.168.1.10 netmask 255.255.255.0 broadcast 192.168.1.255 network 192.168.1.0 gateway 192.168.1.254!dangermouse:~# cd /mnt/devdangermouse:/mnt/dev# ./MAKEDEV console tty1 tty2 tty3 tty4 tty5 tty6dangermouse:~# cp -a /lib/modules/`uname -r` /mnt/lib/modulesdangermouse:~# umount /mnt
Make sure you unmount the disk ready for XEN to run the machine. If you don't you will destroy the filesystem.
Now that we have a basic Debian machine ready, we need to setup the XEN virtual machine config file. This file needs to be on both clients, buggles and dangermouse.
dangermouse:~# cat <<! >/etc/xen/virtual01kernel = "/boot/vmlinuz-2.6.16-1-xen-686"memory = 128name = "virtual01"vif = [ 'ip=192.168.1.10' ]disk = [ 'phy:/dev/iscsi_02,hda2,w','phy:/dev/iscsi_01,hda1,w' ]root = "/dev/hda2 ro"extra = "2"ramdisk = '/boot/initrd.img-2.6.16-1-xen-686'!
Now that we have a config file lets try starting the virtual machine on both of the clients one after the other, shuting it down each time.
dangermouse:~# xm create -c virtual01
This command should produce what looks to be a machine booting, and hopefully it will get all the way to the console where you can log in as root. Once logged in, shut the machine down in the normal way, eg shutdown -h now.
Celebrate! Well done we have a working XEN virtual machine, also known as a domain U. Now repeat the xm create -c virtual01 command on buggles (the other client) to ensure everything is setup and working on there. In each instance test you can ping the ethernet interface from a remote machine.
Now for what is probably the easiest part of the whole process. Now that we know that you have a working iSCSI configuration and a working XEN virtual machine on both clients we are ready to test the live migration process.
With the virtual machine running on one of the clients, (dangermouse in this example) run this command as root on that same machine:
dangermouse:~# xm migrate --live virtual01 buggles
Hopefully after a few seconds the command should return without printing anything. If that is that case then that is probably a very good thing. Check on both clients using xm list to see that the virtual machine has indeed migrated to the second node. If it has, congratulations!
转自:www.performancemagic.com/iscsi-xen-howto/index.html