Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18 kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460c blades.
Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hg and http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you.
Install prerequisite packages
| yum -y groupinstall "Development Libraries" |
| yum -y groupinstall "Development Tools" |
| yum -y installtransfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-devel |
Install rpmforge repository. Follow the instructions on CentOS Wiki.
| wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm |
| rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt |
| rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm |
| rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm |
Install mercurial and git
| yum -y install mercurial git |
Compile Xen
| hg clone http://xenbits.xen.org/xen-4.0-testing.hg |
Compile pvops kernel
This is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. To modify it, change to build-xxx directory and run make menuconfig.
Make sure to have the following in your config file
| # These are important for CentOS/RHEL |
| CONFIG_SYSFS_DEPRECATED=y |
| CONFIG_SYSFS_DEPRECATED_V2=y |
| CONFIG_XEN_MAX_DOMAIN_MEMORY=32 |
| CONFIG_XEN_SAVE_RESTORE=y |
| CONFIG_XEN_PRIVILEGED_GUEST=y |
| CONFIG_XEN_BLKDEV_FRONTEND=m |
| CONFIG_XEN_NETDEV_FRONTEND=m |
| CONFIG_XEN_KBDDEV_FRONTEND=m |
| CONFIG_XEN_FBDEV_FRONTEND=m |
| CONFIG_XEN_BLKDEV_BACKEND=y |
| CONFIG_XEN_NETDEV_BACKEND=y |
| CONFIG_XEN_COMPAT_XENFS=y |
| CONFIG_XEN_XENBUS_FRONTEND=m |
| CONFIG_XEN_PCIDEV_FRONTEND=y |
If you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config.
| CONFIG_XEN_BLKDEV_BACKEND=m |
| CONFIG_XEN_NETDEV_BACKEND=m |
Install Xen and kernel
| make install-tools PYTHON_PREFIX_ARG= |
For make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required.
Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case)
| mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21 |
Update grub
| kernel /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all |
| module /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8 |
| module /initrd-2.6.32.21.img |
Add the following to /etc/modprobe.conf to load all the Xen modules
| install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/true |
OR, add the following to /etc/rc.local (I had trouble making the modprobe work)
| if( uname-r | grep"2.6.32.21") ; then |
| formod inxen-evtchn xen-netback xen-blkback xenfs; do |
Add xenfs to /etc/fstab
| none /proc/xen xenfs defaults 0 0 |
Test it
| echo "savedefault --default=X --once"| grub --batch (where X is the entry forXen) |
Troubleshooting The biggest problem will be kernel not booting properly. There might be various reasons. Make sure to use the right set of configuration options for your hardware Make sure to have all Xen options correctly compiled out_of_memory kernel OOPS CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number Add more memory to dom0 with dom0_mem=4096M Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted Missing /dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run
| modprobe xen-evtchn xen-gntdev |
See step 9 above to update /etc/modprobe.conf
If you see the following in /var/log/messages
| xenconsoled: Failed to contact xenstore (Connection refused) |
Do step 2 to insert all the appropriate modules, and verify that you can start xend
Installing VMs First, verify that the initrd includes all the necessary Xen frontend modules. This
is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all
the right netfront and blkfront modules installed. A sample mkinitrd for domU is
below
| mkinitrd -v-f /boot/initrd-2.6.32.21-domU.img --with=blktap \ |
| --with=xen-blkback --with=xenfs 2.6.32.21 |
Install virt-manager
Configure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines
| (xend-relocation-port 8002) |
Restart xend
Start virt-manager as root Install a VM as usual Referenceshttp://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling the pvops kernel. http://wiki.xensource.com/xenwiki/Xen4.0 http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)
