Memory Allocation Problems on Xen Server & Fix

Recently I started seeing this error while provisioning Xen VPSes on a server with 8GB of RAM:

[2009-03-23 15:51:40 xend.XendDomainInfo 3310] DEBUG (XendDomainInfo:1603) XendDomainInfo.destroy: domid=None
[2009-03-23 15:51:40 xend 3310] ERROR (xmlrpclib2:184) (12, 'Cannot allocate memory')

I checked the memory, and sure enough, all of the VPSes were using up the memory. More specifically, dom0 (the main server) had ‘ballooned’ and begun to take up 3GB of RAM, this is more than the system should need:

[2009-03-22 18:03:08 xend.XendDomainInfo 3310] DEBUG (XendDomainInfo:1113) Setting memory target of domain Domain-0 (0) to 2903 MiB.
[2009-03-22 18:03:09 xend 3310] DEBUG (balloon:127) Balloon: 537840 KiB free; need 537600; done.

Basically when memory ballooning is set, the memory available to the main hypervisor will have a minimum value set and will balloon according to the current requirements. This setting is in the config file which for me is at /etc/xen/xend-config.sxp:

# Dom0 will balloon out when needed to free memory for domU.
# dom0-min-mem is the lowest memory level (in MB) dom0 will get down to.
# If dom0-min-mem=0, dom0 will never balloon out.
(dom0-min-mem 256)

I changed this line to:

(dom0-min-mem 0)

I also modified grub to set an absolute amount of memory to dom0:

title CentOS (2.6.18-92.1.22.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-92.1.22.el5 dom0_mem=1536M
module /vmlinuz-2.6.18-92.1.22.el5xen ro root=/dev/vg_root/lv_root
module /initrd-2.6.18-92.1.22.el5xen.img

The value for dom0_mem sets the system memory at boot time. This will be set on the kernel line. Make sure this is large enough to handle system tasks; 1.5G seems to be good for me right now.

Reboot the server to set the memory for dom0.

You could also set the memory usage for dom0 temporarily by using the ‘xm mem-set’ command:

Usage: xm mem-set <Domain> <Mem>

This would not apply after a reboot. Good luck!