云安全：Cloudstack云平台搭建详解及经验总结

前言：

正在学习云安全，不搭建云平台，总感觉学的有些虚，没落到实处。在网上查资料了解到，Cloudstack产品比较成熟，已经有成熟的企业应用。而且相对OpenStack配置没那么复杂，所以就先从简单入门，不过实际搭建的时候还是遇到了很多坑。下面先写能够搭建的步骤，最后再写自己的失败尝试。

Cloudstack介绍：

What is Apache CloudStack?

Apache CloudStack is an open source Infrastructure-as-a-Service platform that manages and orchestrates pools of storage, network, and computer resources to build a public or private IaaS compute cloud.

 Cloud Infrastructure Overview：

• Zones: Typically, a zone is equivalent to a single datacenter. A zone consists of one or more pods and secondary storage.
• Pods: A pod is usually a rack, or row of racks that includes a layer-2 switch and one or more clusters.
• Clusters: A cluster consists of one or more homogenous hosts and primary storage.
• Host: A single compute node within a cluster; often a hypervisor.
• Primary Storage: A storage resource typically provided to a single cluster for the actual running of instance disk images. (Zone-wide primary storage is an option, though not typically used.)
• Secondary Storage: A zone-wide resource which stores disk templates, ISO images, and snapshots.

 

以上就是cloudstack官方的介绍 

References by:http://docs.cloudstack.apache.org/en/latest/conceptsandterminology/concepts.html

简单理解：

Zone，作为一个数据中心，如果将Zone比作一个仓库，那么，Pod就可以理解为仓库里的一个货架，一个货架有好多层，Cluster就是其中的一层，每层有好多主机，Host就是其中的一个主机。Primary Storage通常和Cluster是联系在一起的，它存放Cluster所有运行着的虚拟机的虚拟磁盘。Secondary Storage存放模板、ISO镜像以及快照。

 

图片来源：https://www.bmc.com/blogs/saas-vs-paas-vs-iaas-whats-the-difference-and-how-to-choose/

SaaS（软件即服务），PaaS（平台即服务）和IaaS（基础架构即服务），用户的自由度依次递增。百度的VPS应该就属于IaaS，云虚拟主机应该属于PaaS，SaaS用户连应用都不用安装，只需要通过浏览器访问和使用即可。

 

 

Cloudstack云平台搭建

环境准备

安装需求：

1. At least one computer which supports and has enabled hardware virtualization.
2. An CentOS 7.5 x86_64 install ISO, on bootable media
3. A /24 network with the gateway being at xxx.xxx.xxx.1, no DHCP should be on this network and none of the computers running CloudStack will have a dynamic address. Again this is done for the sake of simplicity.

感觉单纯做练习的话还是使用虚拟机，方便些，毕竟有快照。

虚拟机开启硬件虚拟化的方法，以VMware Workstation为例：

虚拟机----->设置------>处理器----->勾选虚拟化引擎中的虚拟化Intel VT----->确定即可

进入虚拟机后可使用命令这条命令查看是否开启虚拟化

[root@localhost upload]# lsmod | grep kvm
kvm_intel             188644  6 
kvm                   621480  1 kvm_intel
irqbypass              13503  13 kvm

 Operating System

官网的推荐是最好更新一下

If your network interface was configured to grant the server internet access, it is always wise to update the system before starting:

 但是个人感觉，还是不要更新的好，我一开始也更新了一次结果安装过程中出现各种问题。

yum -y update

网络配置

这个是必须要做的，我有一次安装的时候偷懒没有设置桥接网络，结果就报错了，cloudstack-agent在初始化的时候会检查网络是不是桥接

Before going any further, make sure that “brctl” is installed and available:

yum install bridge-utils -y

Create and open /etc/sysconfig/network-scripts/ifcfg-cloudbr0 and add the following settings:

 

DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
IPADDR=172.16.10.2
GATEWAY=172.16.10.1
NETMASK=255.255.255.0
DNS1=8.8.8.8
DNS2=8.8.4.4
STP=yes
USERCTL=no
NM_CONTROLLED=no

#这一步中需要注意的是，对于IPADDR, GATEWAY, DNS应该按照自己的真实网络来配置
比如我的是：

IPADDR=192.168.75.136
GATEWAY=192.168.75.2
NETMASK=255.255.255.0
DNS1=192.168.75.2
DNS2=192.168.1.1

官网的DNS好像是谷歌的，我用的是VMware的网关.网关和DNS设置成一样的就行，没有必要特地去上网查一些DNS服务器(其实我就查了つ﹏⊂)。

VMware虚拟网关查看方法：

编辑--->虚拟网络编辑器---->NAT设置

 

Open the configuration file of your interface and configure it as follows:

这里意思是设置自己虚拟机中原有的网卡，我的命令是vi /etc/sysconfig/network-scripts/ifcfg-ens33。根据自己的网卡名称改变

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
NAME=eth0
DEVICE=eth0
ONBOOT=yes
BRIDGE=cloudbr0

到这里后，官网的建议是

Now that we have the configuration files properly set up, we need to run a few commands to start up the network:

# systemctl enable network

# systemctl restart network

其实这里按照如果真的按照官网的建议去做，就会发现虚拟机无法联网了，而且会报错：

Failed to start LSB: Bring up/down networking

这是由于network服务和NetwrokManager服务冲突导致的，关闭NetworkManager即可

# systemctl stop NetworkManager

# systemctl disable NetworkManager

 

 完成后可以ping一下百度。

ping baidu.com

Hostname

# hostname --fqdn
At this point it will likely return:
localhost

这一步一般是没有问题的，如果出错了，参考官方文档的纠正方法

http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html#hostname

安全配置

由于只是练习，安全配置全都关掉好了，不必增加负担。

systemctl stop firewalld
systemctl disable firewalld

setenforce permissive
#修改文件使SELINUX=permissive
vi /etc/selinux/config


systemctl disable iptables
systemctl stop iptables

时间同步

NTP configuration is a necessity for keeping all of the clocks in your cloud servers in sync.

十分简单

# yum -y install ntp
# systemctl enable ntpd
# systemctl start ntpd

NFS

NFS(network file system)网络文件系统的搭建也十分简单

安装nfs

Our configuration is going to use NFS for both primary and secondary storage. We are going to go ahead and setup two NFS shares for those purposes.

# yum -y install nfs-utils

添加共享文件目录

We now need to configure NFS to serve up two different shares. This is handled comparatively easily in the /etc/exports file.

vi /etc/exports
#添加以下两行即可
/export/secondary *(rw,async,no_root_squash,no_subtree_check)
/export/primary *(rw,async,no_root_squash,no_subtree_check)

创建共享文件目录

We’ll go ahead and create those directories

# mkdir -p /export/primary
# mkdir /export/secondary

配置NFS

Now you’ll need to add the configuration values at the bottom in the file /etc/sysconfig/nfs

vi /etc/sysconfig/nfs
#觉得取消注释麻烦的话，直接把下面几行粘贴到文件中就行

LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
MOUNTD_PORT=892
RQUOTAD_PORT=875
STATD_PORT=662
STATD_OUTGOING_PORT=2020

启动与开机自启

We now need to configure the nfs service to start on boot and actually start it

# systemctl enable rpcbind
# systemctl enable nfs
# systemctl start rpcbind
# systemctl start nfs

 

数据库

先换源

这是清华开源镜像站，里面有mysql换源的方法

https://mirrors.tuna.tsinghua.edu.cn/help/mysql/

Note：这里有一个大坑

如果这里不对.repo做变动，会导致安装的时候，安装的是最新版mysql8，而开发团队其实使用的是mysql5.1或mysql5.5

这里清华的镜像源并没有mysql5.5但5.6也能用。

更重要的是，如果这里不管的话，到cloudstack-management安装完成时，会发现无法通过web UI访问，报错：

HTTP ERROR 503

Problem accessing /client/. Reason:

    Service Unavailable

如果只是单纯的去搜索HTTP ERROR 503的解决办法是完全没有用的。

查看日志

vi /var/log/cloudstack/management/management-server.log

就会发现这样一条错误

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:101)
Caused by: java.lang.IllegalArgumentException: Can not set long field com.cloud.upgrade.dao.VersionVO.id to java.math.BigInteger
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at sun.reflect.UnsafeLongFieldAccessorImpl.set(UnsafeLongFieldAccessorImpl.java:102)
... 65 more

这里导致503问题真正原因是

Looks like the new version (8.0.x) is stricter with conversions from Long to BigIntfor example. Auto Increment fields in the database are of type BigInt, but in code we are using Long. 

https://github.com/apache/cloudstack/issues/3843

于此同时mysql-connector-java 版本也要修改，mysql官方提供的是8.0清华镜像中的也是8.0，可以用epel-release中的软件旧版

解决的办法就是修改下载的mysql-community.repo，要禁用mysql80, 57, mysql，mysql-connector-java

[root@localhost upload]# cat /etc/yum.repos.d/mysql-community.repo 
[mysql-connectors-community]
name=MySQL Connectors Community
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-connectors-community-el7/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
exclude=mysql-connector-java*

[mysql-tools-community]
name=MySQL Tools Community
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-tools-community-el7/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

[mysql56-community]
name=MySQL 5.6 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql56-community-el7/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql57-community-el7/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

[mysql80-community]
name=MySQL 8.0 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql80-community-el7/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
[root@localhost upload]# 

同时记得安装epel-release，并换源：

# yum -y install epel-release
# curl -o /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
# rm /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel-testing.repo 

# 每次换源后都记得执行
# yum clean all
# yum makecache

 

回归正题，继续安装。

添加验证密钥

rpm --import http://repo.mysql.com/RPM-GPG-KEY-mysql

安装

# yum install mysql-server
# yum install mysql-connector-python

配置密码

# mysql_secure_installation

修改mysqld配置文件

With MySQL now installed we need to make a few configuration changes to /etc/my.cnf.

Specifically we need to add the following options to the [mysqld] section:

innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'

单主机模式配置

If you are running the KVM hypervisor on the same machine with the Management Server, edit /etc/sudoers and add the following line:

Defaults:cloud !requiretty

管理系统安装

接下来就是cloudstack-managenment的安装

一般yum源中是没有cloudstack的，官方文档是直接修改yum添加上它的库，详见

http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html#configuring-the-cloudstack-package-repository

但由于我这边外网下载速度着实令人捉急，在虚拟机里通过yum下载是行不通的。

这里可以根据官方的repo地址http://download.cloudstack.org/centos/7/4.11/

下载到本地再上传给虚拟机，同时这里也可以提前把后面要用到的系统模板给下载好。

现在需要下载并上传到虚拟机中的有四个文件

http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2

http://download.cloudstack.org/centos7/4.11/cloudstack-agent-4.11.3.0-1.el7.centos.x86_64.rpm

http://download.cloudstack.org/centos7/4.11/cloudstack-common-4.11.3.0-1.el7.centos.x86_64.rpm

http://download.cloudstack.org/centos7/4.11/cloudstack-management-4.11.3.0-1.el7.centos.x86_64.rpm

对于上传文件用xshell, 虚拟机安装lrzsz即可：

# yum -y install lrzsz

management安装：

localinstall 可以自动解决安装包的依赖问题

安装时每台主机都要安装common，且必须先安装，不然会提示依赖错误（以身试法/(ㄒoㄒ)/~~）

[root@localhost upload]# yum -y localinstall cloudstack-common-4.11.3.0-1.el7.centos.x86_64.rpm cloudstack-management-4.11.3.0-1.el7.centos.x86_64.rpm 

cloudstack-management与数据库的连接

With the application itself installed we can now setup the database, we’ll do that with the following command and options:

# cloudstack-setup-databases cloud:password@localhost --deploy-as=root:password

cloudstack-management的初始化

Now that the database has been created, we can take the final step in setting up the management server by issuing the following command:

[root@master cloudstack]# cloudstack-setup-management
Starting to configure CloudStack Management Server:
Configure Firewall ...        [OK]
Configure CloudStack Management Server ...[OK]
CloudStack Management Server setup is Done!
[root@master cloudstack]# 

提示完成，初始化实际上还要等一会才完成，直到执行

ss -nplt | grep 8080

时，出现

[root@localhost upload]# ss -nplt | grep 8080
LISTEN     0      50        [::]:8080                  [::]:*                   users:(("java",pid=2648,fd=42))

监听8080端口了才可以访问Web UI

这个时候如果，数据库安装有问题，访问时就会出现前面说的503错误

上传系统模板

# usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -f systemvmtemplate-4.11.2-kvm.qcow2.bz2 -h kvm -F

agent安装

agent如果不是和management安装在同一台机器的话，也是需要经历如下过程：

• 网络配置
• 验证hostname
• 安全配置
• 安装NTP

我是安装在同一台机器就省略了这些步骤

只需执行

[root@localhost upload]# yum localinstall cloudstack-common-4.11.3.0-1.el7.centos.x86_64.rpm cloudstack-management-4.11.3.0-1.el7.centos.x86_64.rpm -y

#单主机cloudstack-common实际上是不必要的

KVM配置

We have two different parts of KVM to configure, libvirt, and QEMU.

QEMU配置

KVM configuration is relatively simple at only a single item. We need to edit the QEMU VNC configuration. This is done by editing /etc/libvirt/qemu.conf and ensuring the following line is present and uncommented.

vnc_listen=0.0.0.0

Libvirt配置

Libvirt is a dependency of cloud-agent and should already be installed.

In order to have live migration working libvirt has to listen for unsecured TCP connections. We also need to turn off libvirts attempt to use Multicast DNS advertising. Both of these settings are in /etc/libvirt/libvirtd.conf

Set the following paramaters:

listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0

modify /etc/sysconfig/libvirtd:

取消注释

# LIBVIRTD_ARGS="--listen"

重启：

# systemctl restart libvirtd

验证：

# lsmod | grep kvm
kvm_intel              55496  0
kvm                   337772  1 kvm_intel

安装基本完成

Web UI

下面就是在浏览器上点点点而已：

Add Zone:

Add Pod:

Add Cluster

Add Primary Storage:

Add Secondary Storage:

Add Host

最后着一步是很容易失败的，原因很多，要多去看日志/var/log/cloudstack/agent/:

Lanch:

资源占用情况

 

安装过程中可能遇到的问题及解决办法：

无dna模块：

[root@master ~]# cloudstack-setup-management
Traceback (most recent call last):
  File "/usr/bin/cloudstack-setup-management", line 24, in 
    from cloudutils.serviceConfigServer import cloudManagementConfig
  File "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfigServer.py", line 17, in 
    from db import Database
  File "/usr/lib64/python2.7/site-packages/cloudutils/db.py", line 20, in 
    import mysql.connector
  File "/usr/lib64/python2.7/site-packages/mysql/connector/__init__.py", line 41, in 
    import dns.resolver
ImportError: No module named dns.resolver

解决办法：

yum -y install python-pip
pip install dnspython

mysql 验证码错误

GPG key retrieval failed: [Errno 14] curl#37 - "Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-mysql"

解决办法：

rpm --import http://repo.mysql.com/RPM-GPG-KEY-mysql

maven javac version 错误

这是我根据4.14.0.0的源码编译时，出现的错误，按照上面的步骤不会出现这种错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project cloud-framework-managed-context: Compilation failure
[ERROR] javac: invalid target release: 11
[ERROR] Usage: javac  
[ERROR] use -help for a list of possible options
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

解决办法：

sudo alternatives --config javac

Reference by:https://issues.apache.org/jira/browse/SPARK-21075

Add host时出错

错误原因可能很多，这只是一种

DEBUG:root:execute:hostname -f
DEBUG:root:execute:selinuxenabled
DEBUG:root:execute:setenforce 0
DEBUG:root:cloudbr0 is not a network device, is it down?
DEBUG:root:execute:route -n|awk '/^0.0.0.0/ {print $2,$8}'
DEBUG:root:execute:ifconfig ens33
DEBUG:root:[Errno 2] No such file or directory
  File "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfig.py", line 38, in configration
    result = self.config()
  File "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfig.py", line 309, in config
    super(networkConfigRedhat, self).cfgNetwork()
  File "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfig.py", line 108, in cfgNetwork
    device = self.netcfg.getDefaultNetwork()
  File "/usr/lib64/python2.7/site-packages/cloudutils/networkConfig.py", line 53, in getDefaultNetwork
    pdi = networkConfig.getDevInfo(dev)
  File "/usr/lib64/python2.7/site-packages/cloudutils/networkConfig.py", line 157, in getDevInfo
    elif networkConfig.isBridge(dev) or networkConfig.isOvsBridge(dev):

解决办法：

必须用桥接网卡

 

总结与收获：

这次学习Cloudstack花费的时间格外多，一开始是看一些视频教程，大致有个印象，然后开始实践，遇到坑，爬不上来。就来CSDN找文字教程，还是趴不上来，解决不了问题。又去官网，扒官方文档，但是按照官方文档的步骤来仍然出错（计算机发展的真快啊，大概19年九月的文章竟然过时了）。最后，在不存在的网站（https://github.com/apache/cloudstack/issues/3843）上找到了解答，是Mysql yum软件库最近的一次更新后，只提供8.0版本的mysql-connector-java导致的。

学习历程：视频教程->文字教程->官方文档->不存在的网站。