CentOS6下安装基于Xen的Eucalyptus 2.0.3

Eucalyptus

• 官网 http://open.eucalyptus.com/downloads
• 手册 http://open.eucalyptus.com/wiki
• 服务器角色
  • Cloud controller(CLC)，控制调度云的资源，同时提供web界面以及兼容EC2的SOAP服务
  • Walrus，实现了基于桶的存储，同时提供兼容S3的SOAP服务以及REST接口
  • Cluster controller(CC)，实现以集群为单位的调度
  • Storage controller(SC)，基于块的存储控制器，EBS-style
  • Node controller(NC)，虚拟节点，运行xen及虚拟机的机器

版本

• CentOS 6 Minimal install
• Xen 4.1.1 NC服务器上面需要完全安装，其余服务器需要几个编译好的rpm，参考 http://blog.csdn.net/phoenixie/article/details/6885313
• Eucalyptus 2.0.3，下载offline源码包, http://eucalyptussoftware.com/downloads/releases/eucalyptus-2.0.3-src-offline.tar.gz , 依赖的库 http://eucalyptussoftware.com/downloads/releases/eucalyptus-2.0.3-src-deps.tar.gz
• euca2ools 1.3.1, 使用fedora的，下载 http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1-fedora-x86_64.tar.gz
• dhcpd 3.0.5, CC上需要安装, 使用旧版本rhel5的, eucalyptus 2.0.3对稍微高版本的ISC dhcpd支持有问题, 下载http://ftp.redhat.com/redhat/linux/enterprise/5Server/en/os/SRPMS/dhcp-3.0.5-29.el5_7.1.src.rpm

准备工作

 1. 下载各个软件包

 2. 除了NC以外的服务器安装CentOS 6 Minimal方式

 3. NC服务器按照参考文档安装配置Xen环境，注意在Xen的内核配置里面添加一项  max_loop=256

 4. 配置好所有机器的NTP，使它们时钟保持同步

安装过程

 1. 创建 /etc/yum.repos.d/euca.repo 

[euca]
name=Eucalyptus
baseurl=http://www.eucalyptussoftware.com/downloads/repo/eucalyptus/2.0.3/yum/centos/x86_64
gpgcheck=0

 2. 通过yum安装必须的软件包 (所有机器上)

yum -y install perl-Convert-ASN1.noarch scsi-target-utils httpd java-1.6.0-openjdk-devel ant ant-nodeps libvirt-devel curl-devel httpd-devel apr-devel openssl-devel libxml2 libxml2-devel gnutls gnutls-devel libgcrypt-devel zlib-devel perl-Convert-ASN1 perl-Crypt-OpenSSL-RSA perl-Crypt-OpenSSL-Random chkfontpath fuse-libs swig gcc make patch compat-expat1 compat-db43 lzo2 vblade m2crypto lvm2 rsync ntp vconfig groff rpm-build

 3. 安装之前编译好的Xen的rpm包(给控制服务器，节点服务器已经装过xen的就不需要了)

rpm -Uvh xen-licenses-*.x86_64.rpm xen-devel-*.x86_64.rpm xen-libs-*.x86_64.rpm 

 4. 安装dhcpd

wget http://ftp.redhat.com/redhat/linux/enterprise/5Server/en/os/SRPMS/dhcp-3.0.5-29.el5_7.1.src.rpm
  
rpm -i dhcp-3.0.5-29.el5_7.1.src.rpm
  
cd ~/rpmbuild/SPECS/
rpmbuild -bb dhcp.spec  # 如果编译不过去，可能需要手动去掉-Werror编译选项
  
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh dhcp-3.0.5-29.el6.1.x86_64.rpm # 如果已经装了dhcp，先手动删除掉原来的

 5. 安装Eucalyptus的依赖软件(eucalyptus-2.0.3-src-deps.tar.gz)

export EUCALYPTUS=/home/public/eucalyptus-2.0.3
export APACHE_INCLUDES=/usr/include/httpd/
export APR_INCLUDES=/usr/include/apr-1/
export AXIS2C_HOME=${EUCALYPTUS}/packages/axis2c-1.6.0

mkdir -p ${EUCALYPTUS}/packages/

tar xzf eucalyptus-2.0.3-src-deps.tar.gz
cd eucalyptus-src-deps

tar xzf axis2-1.4.tgz -C ${EUCALYPTUS}/packages/
cd ${EUCALYPTUS}/packages/
ln -sf axis2-1.4 axis2
cd -

tar xzf axis2c-src-1.6.0.tar.gz
cd axis2c-src-1.6.0
CFLAGS="-w" ./configure --prefix=${AXIS2C_HOME} -with-apache2=${APACHE_INCLUDES} --with-apr=${APR_INCLUDES} --enable-multi-thread=no
make
make install
cd -
rm -rf axis2c-src-1.6.0

export LD_LIBRARY_PATH=${AXIS2C_HOME}/lib
tar xzf rampartc-src-1.3.0-0euca2.tar.gz
cd rampartc-src-1.3.0
./configure --prefix=${AXIS2C_HOME} --enable-static=no --with-axis2=${AXIS2C_HOME}/include/axis2-1.6.0/
make
make install
rm -rf rampartc-src-1.3.0

cd ${EUCALYPTUS}/packages
ln -sf axis2c-1.6.0 axis2c
cd -
sed -i "s;<\!--phase name=\"Security\"/-->;<phase name=\"Security\"/>;g" ${AXIS2C_HOME}/axis2.xml

cd ..
rm -rf eucalyptus-src-deps

 6. 安装Eucalyptus (eucalyptus-2.0.3-src-offline.tar.gz)

export EUCALYPTUS=/home/public/eucalyptus-2.0.3
export AXIS2_HOME=${EUCALYPTUS}/packages/axis2
export AXIS2C_HOME=${EUCALYPTUS}/packages/axis2c
export JAVA_HOME="/usr/lib/jvm/java-openjdk/"
export JAVA="$JAVA_HOME/JRE/bin/java"

tar xzf eucalyptus-2.0.3-src-offline.tar.gz
cd eucalyptus-2.0.3-src-offline
./configure --with-axis2=${AXIS2_HOME} --with-axis2c=${AXIS2C_HOME} --enable-debug --prefix=${EUCALYPTUS}
make
make install
cd -
rm -rf eucalyptus-2.0.3-src-offline

cd /home/public/
ln -sf eucalyptus-2.0.3 eucalyptus

export EUCALYPTUS=/home/public/eucalyptus
ln -sf $EUCALYPTUS/etc/init.d/eucalyptus-cloud /etc/init.d/eucalyptus-cloud
ln -sf $EUCALYPTUS/etc/init.d/eucalyptus-cc /etc/init.d/eucalyptus-cc
ln -sf $EUCALYPTUS/etc/init.d/eucalyptus-nc /etc/init.d/eucalyptus-nc

chkconfig eucalyptus-cloud off
chkconfig eucalyptus-cc off
chkconfig eucalyptus-nc off

 7. 安装工具(euca2ools-1.3.1-fedora-x86_64.tar.gz)

tar xzf euca2ools-1.3.1-fedora-x86_64.tar.gz
cd euca2ools-1.3.1-fedora-x86_64

rpm -Uvh euca2ools-1.3.1-1.x86_64.rpm

cd -
rm -rf euca2ools-1.3.1-fedora-x86_64

配置

 1. 创建一个专属用户 

adduser eucalyptus

 2. 初始化

export EUCALYPTUS=/home/public/eucalyptus
$EUCALYPTUS/usr/sbin/euca_conf \
 -d $EUCALYPTUS \
 --hypervisor xen \
 --instances /home/eucalyptus/instances \
 --user eucalyptus \
 --setup
# -d: Eucalyptus路径
# --hypervisor: 使用的hypervisor类型，可以是kvm或xen
# --instances: instance存储路径
# --user: 服务使用的用户
# --setup: 初始化

service eucalyptus-nc start
chkconfig eucalyptus-nc on

 3. 编辑 /home/public/eucalyptus/etc/eucalyptus/eucalyptus.conf，按自己的需求修改，其中 VNET_BRIDGE 是xen要使用的bridge名字，其余的VNET_MODE等请参考配置文件的说明，里面写的很详细。
 4. 复制eucalyptus安装目录到所有机器上，建立init.d的链接，添加service的服务，然后还需要运行一次初始化
 5. 启动NC

service eucalyptus-nc start # NC的Web service默认端口8775
chkconfig eucalyptus-nc on

 6. 启动CC

service eucalyptus-cc start # CC的Web service默认端口8774
chkconfig eucalyptus-cc on

 7. 启动相关服务，不同的服务可以部署在不同机器上面

export EUCALYPTUS=/home/public/eucalyptus
$EUCALYPTUS/usr/sbin/euca_conf -d $EUCALYPTUS --enable cloud
$EUCALYPTUS/usr/sbin/euca_conf -d $EUCALYPTUS --enable walrus
$EUCALYPTUS/usr/sbin/euca_conf -d $EUCALYPTUS --enable sc

service eucalyptus-cloud start # 三个服务都通过这个脚本启动，http端口8080，https端口8443，Web service端口8773
chkconfig eucalyptus-cloud on

 8. 注册服务

$EUCALYPTUS/usr/sbin/euca_conf --register-walrus $IP
$EUCALYPTUS/usr/sbin/euca_conf --register-cluster OurCloud $IP # 注册一个Cluster，名字叫OurCloud，默认端口8774
$EUCALYPTUS/usr/sbin/euca_conf --register-sc OurCloud $IP # 给Cluster添加一个SC

 9. 注册节点

$EUCALYPTUS/usr/sbin/euca_conf --register-nodes $IP

 10. 访问Cloud(CLC)所在IP的https://$IP:8443，用户名admin，密码admin，第一次登陆成功后会要求马上修改密码
 11. 在管理界面的导航栏找到Credentials，进入，点击按钮"Download Credentials"，会下载一个zip文件
 12. 把zip文件传到安装了euca2ools的服务器上，建一个目录解压进去，以后每次使用euca2ools前需要执行一次下面的命令

source eucarc

 13. 安装完成

实例1: 使用官网提供的Ubuntu image

 1. 下载地址  http://open.eucalyptus.com/wiki/EucalyptusUserImageCreatorGuide 2. 添加 (euca-ubuntu-9.04-x86_64.tar.gz) 

tar xzf euca-ubuntu-9.04-x86_64.tar.gz
cd euca-ubuntu-9.04-x86_64

# 进入运行环境
source ~/.euca/eucarc

# 上传内核，得到一个eki-XXXXXXXX
euca-bundle-image -d ./tmp/ -i xen-kernel/vmlinuz-2.6.27.21-0.1-xen --kernel true
euca-upload-bundle -b ubuntu9-bucket -m ./tmp/vmlinuz-2.6.27.21-0.1-xen.manifest.xml
euca-register ubuntu9-bucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml

# 上传ramdisk，得到一个eri-XXXXXXXX
euca-bundle-image -d ./tmp/ -i xen-kernel/initrd-2.6.27.21-0.1-xen --ramdisk true
euca-upload-bundle -b ubuntu9-bucket -m ./tmp/initrd-2.6.27.21-0.1-xen.manifest.xml
euca-register ubuntu9-bucket/initrd-2.6.27.21-0.1-xen.manifest.xml

# 上传根分区，用上面两个命令得到的数字执行
euca-bundle-image -d ./tmp/ -i ubuntu.9-04.x86-64.img --kernel eki-XXXXXXXX --ramdisk eri-XXXXXXXX
euca-upload-bundle -b ubuntu9-bucket -m ./tmp/ubuntu.9-04.x86-64.img.manifest.xml
euca-register ubuntu9-bucket/ubuntu.9-04.x86-64.img.manifest.xml

 3. 创建一个登录虚拟机用的key

euca-add-keypair euca-key > ~/euca-key.private
chmod 0600 ~/euca-key.private

 4. 使用刚才上传虚拟机时候得到的emi值启动一个虚拟机实例

euca-run-instances -k euca-key -n 1 emi-XXXXXXXX

 5. 查看运行状态，没有dhcpd的话，这时拿不到IP

>euca-describe-instances

RESERVATION r-4F030A57 admin default
INSTANCE i-35C505C3 emi-XXXXXXXX 0.0.0.0 0.0.0.0 running euca-key 0 m1.small 2011-11-01T10:48:56.831Z OurCloud eki-XXXXXXXX eri-XXXXXXXX

实例2: 使用dhcpd分配IP

 0. CC服务器, IP eth3 192.168.11.3/24; NC服务器, IP eth3 192.168.11.4/24, Xen的bridge xenbr0; 实现从NC上面运行的虚拟机可以通过CC访问到外面的网络
 1. 配置CC的/home/public/eucalyptus/etc/eucalyptus/eucalyptus.conf

VNET_PUBINTERFACE="eth3"
VNET_PRIVINTERFACE="eth3"

VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_DHCPUSER="eucalyptus"

VNET_MODE="MANAGED-NOVLAN"
VNET_SUBNET="192.168.101.0"
VNET_NETMASK="255.255.255.0"
VNET_DNS="192.168.11.2"
VNET_ADDRSPERNET="16"   # 这个数字如果太大会无法创建虚拟机
VNET_PUBLICIPS="192.168.11.80-192.168.11.120"
VNET_CLOUDIP="192.168.11.3"
#VNET_MODE="SYSTEM"

 2. 配置NC的/home/public/eucalyptus/etc/eucalyptus/eucalyptus.conf

VNET_PUBINTERFACE="xenbr0"
VNET_PRIVINTERFACE="xenbr0"
VNET_BRIDGE="xenbr0"
VNET_MODE="MANAGED_NOVLAN"
#VNET_MODE="SYSTEM"

 3. 重新启动CC

service eucalyptus-cc cleanrestart

 4. 创建一个虚拟机看看 

>euca-run-instances -k euca-key -n 1 emi-XXXXXXXX 
RESERVATION r-502007DA admin admin-default
INSTANCE i-440907AE emi-XXXXXXXX 0.0.0.0 0.0.0.0 pending euca-key 2011-11-04T11:01:17.656Z eki-XXXXXXXX eri-XXXXXXXX

>euca-describe-instances 
RESERVATION r-502007DA admin default
INSTANCE i-440907AE emi-XXXXXXXX 192.168.11.101 192.168.101.131 running euca-key 0 m1.small 2011-11-04T11:01:17.656Z OurCloud eki-XXXXXXXX eri-XXXXXXXX

>ssh -i ~/euca-key.private 192.168.11.101
root@ubuntu:~#

>ip addr show eth0
UNKNOWN qlen 1000
    link/ether d0:0d:44:09:07:ae brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.131/28 brd 192.168.101.143 scope global eth0
    inet6 fe80::d20d:44ff:fe09:7ae/64 scope link 
       valid_lft forever preferred_lft forever

>ping -c 1 192.168.11.2 
PING 192.168.11.2 (192.168.11.2) 56(84) bytes of data.
64 bytes from 192.168.11.2: icmp_seq=1 ttl=63 time=1.26 ms

--- 192.168.11.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.264/1.264/1.264/0.000 ms

实例3: 自己建立CentOS6虚拟机

 1. 使用virtinst以minimal模式安装一个CentOS6虚拟机，硬盘弄一个分区就行了，其他分区都可省，在虚拟机里面安装配置好需要的软件，关掉虚拟机(安装参数可参考 http://blog.csdn.net/phoenixie/article/details/6885313)

 2. 提取虚拟机 / 分区，假设硬盘文件名为sda.img，2GB

> parted sda.img
GNU Parted 2.1
Using /home/vms/centos6/sda.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) U
Unit?  [compact]? b                                                       
(parted) p
Model:  (file)
Disk /home/vms/centos6/sda.img: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start     End          Size         Type     File system  Flags
 1      1048576B  2147483647B  2146435072B  primary  ext3         boot

(parted) quit
> dd if=sda.img of=rootfs.img bs=1M skip=1 count=2047
2047+0 records in
2047+0 records out
2146435072 bytes (2.1 GB) copied, 37.3425 s, 57.5 MB/s

 3. 配置根分区文件

# 加载根分区
mkdir /mnt/rootfs
mount -o loop rootfs.img /mnt/rootfs/

# 复制出来kernel和initramfs
cp /mnt/rootfs/boot/vmlinuz-* ./kernel
cp /mnt/rootfs/boot/initramfs-* ./ramdisk

# 配置网卡
rm udev/rules.d/70-persistent-net.rules
vi /mnt/rootfs/etc/udev/rules.d/60-net.rules
ACTION=="add", SUBSYSTEM=="net", IMPORT{program}="/lib/udev/rename_device"
SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/net.hotplug"

vi /mnt/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet

# 关闭selinux
vi /mnt/rootfs/etc/sysconfig/selinux 
SELINUX=disabled

# 修改fstab，去掉所有的UUID方式
vi /mnt/rootfs/etc/fstab
/dev/xvda1         /             ext3     defaults,errors=remount-ro 0 0
/dev/xvda2         /mnt          ext3     defaults                   0 0
/dev/xvda3         swap          swap     defaults                   0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

vi /mnt/rootfs/etc/mtab
/dev/sda1 / ext3 rw,errors=remount-ro 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0

# 启动时自动下载可登录的ssh-rsa的public key
vi /mnt/rootfs/etc/rc.local 
# simple attempt to get the user ssh key using the meta-data service
mkdir -p /root/.ssh
echo >> /root/.ssh/authorized_keys
curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /root/.ssh/authorized_keys
echo "AUTHORIZED_KEYS:"
echo "************************"
cat /root/.ssh/authorized_keys
echo "************************"

# 删除密码
vi /mnt/rootfs/etc/shadow
root:!:15280:0:99999:7:::

# 卸载
umount /mnt/rootfs

 4. 修改NC上面的 /home/public/eucalyptus/usr/share/eucalyptus/gen_libvirt_xml, 把所有的sda改为xvda, 这个修改后，目前官网提供的image将不能使用

 5. 最后, 上传

source ~/.euca/eucarc
# 上传内核，得到一个eki-KKKKKKKK
euca-bundle-image -d ./tmp/ -i ./kernel --kernel true
euca-upload-bundle -b centos6-bucket -m ./tmp/kernel.manifest.xml 
euca-register centos6-bucket/kernel.manifest.xml
# 上传ramdisk，得到一个eri-RRRRRRRR
euca-bundle-image -d ./tmp/ -i ./ramdisk --ramdisk true
euca-upload-bundle -b centos6-bucket -m ./tmp/ramdisk.manifest.xml
euca-register centos6-bucket/ramdisk.manifest.xml
# 上传根分区
euca-bundle-image -d ./tmp/ -i ./rootfs.img --kernel  eki-KKKKKKKK --ramdisk eri-RRRRRRRR
euca-upload-bundle -b centos6-bucket -m ./tmp/rootfs.img.manifest.xml 
euca-register centos6-bucket/rootfs.img.manifest.xml

 6. 现在可以试试了，需要注意的即将创建的虚拟机的磁盘，不能小于根分区文件的字节数，否则不能建立虚拟机；而且设定的虚拟机的内存，不能小于内核的最低需要；这两个参数可以在管理界面里面调整。