Cloudstack 安全组分析（1）
面向Cloudstack4.2 Security Group XenServer 6.x ,对于KVM原理一样，需要和Agent通信，调用security_group.py。
安全组介绍
安全组是AWS里的一种技术，用于隔离多租户虚拟机，Cloudstack也支持安全组，3.0.x版本只支持Basic Zone的安全组，4.1以后引入Advance Shared Zone的安全组，目前安全组支持的Hypervisor为KVM 和Xen系列，对于VMware可以使用PVLAN也可以使用vShield实现，需要独立开发。
首先要创建带安全组的Shared网络服务，创建Zone，创建VM时，选择新创建的安全组或默认的。如果不选择安全组，会使用默认的安全组。默认情况下，安全组会将VM的Deny All Ingress ，Allow ALL Egress.
XenServer前置条件
对于XenServer< 6.1,需要安装CSP包来支持安全组，从XenServer6.1开始，CSP已经集成到XenServer安装盘中。
XenServer 6.0.2:
http://download.cloud.com/releases/3.0.1/XS-6.0.2/xenserver-cloud-supp.tgz
XenServer 5.6 SP2:
http://download.cloud.com/releases/2.2.0/xenserver-cloud-supp.tgz
XenServer 6.0:
http://download.cloud.com/releases/3.0/xenserver-cloud-supp.tgz
安装：
#tar xf xenserver-cloud-supp.tgz #xe-install-supplemental-pack xenserver-cloud-supp.iso
另外需要将网络从OVS改为Bridge。在XenServer机器执行
#cat /etc/xensource/network.conf openvswitch
执行改为bridge或者手动改为bridge,并重启主机。
# xe-switch-network-backend bridge
另外需要确认以下这些值，如下表示OK。
#sysctl -a|grep bridge net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-arptables = 1
打开内核转发：
net.ipv4.ip_forward = 1
如果不是，修改
vi /etc/sysctl.conf #sysctl -p #生效
安全组默认规则代码分析
CitrixResourceBase.java 是负责和XenServer交互的底层资源类。在VM启动时，如果启用安全组，将建立默认规则(Ingress Allow,Egress Deny).
方法为 execute(StartCommand cmd),里面代码片段：
if (_canBridgeFirewall) {
String result = null;
if (vmSpec.getType() != VirtualMachine.Type.User) {
NicTO[] nics = vmSpec.getNics();
boolean secGrpEnabled = false;
for (NicTO nic : nics) {
if (nic.isSecurityGroupEnabled()
|| (nic.getIsolationUri() != null && nic.getIsolationUri().getScheme()
.equalsIgnoreCase(IsolationType.Ec2.toString()))) {
secGrpEnabled = true;
break;
}
}
if (secGrpEnabled) {
result = callHostPlugin(conn, "vmops", "default_network_rules_systemvm", "vmName", vmName);
if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
s_logger.warn("Failed to program default network rules for " + vmName);
} else {
s_logger.info("Programmed default network rules for " + vmName);
}
}
} else {
// For user vm, program the rules for each nic if the isolation
// uri scheme is ec2
NicTO[] nics = vmSpec.getNics();
for (NicTO nic : nics) {
if (nic.isSecurityGroupEnabled() || nic.getIsolationUri() != null
&& nic.getIsolationUri().getScheme().equalsIgnoreCase(IsolationType.Ec2.toString())) {
List<String> nicSecIps = nic.getNicSecIps();
String secIpsStr;
StringBuilder sb = new StringBuilder();
if (nicSecIps != null) {
for (String ip : nicSecIps) {
sb.append(ip).append(":");
}
secIpsStr = sb.toString();
} else {
secIpsStr = "0:";
}
result = callHostPlugin(conn, "vmops", "default_network_rules", "vmName", vmName, "vmIP",
nic.getIp(), "vmMAC", nic.getMac(), "vmID", Long.toString(vmSpec.getId()), "secIps",
secIpsStr);
if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
s_logger.warn("Failed to program default network rules for " + vmName + " on nic with ip:"
+ nic.getIp() + " mac:" + nic.getMac());
} else {
s_logger.info("Programmed default network rules for " + vmName + " on nic with ip:"
+ nic.getIp() + " mac:" + nic.getMac());
}
}
}
}
}
该方法将判断主机网络是否为bridge，如果不是就不会建立安全组规则。将遍历VM所有网卡，判断是否启用安全组隔离类型为EC2，即安全组隔离。
对于System VM（vrouter,ssvm,cpvm），要单独处理，如果是System VM,会调用主机plugin vmops里的方法default_network_rules_systemvm 参数为VM名字，如果是User VM,会调用default_network_rules，参数为 vmName, vmIP ，vmMAC,， secIps，secIps表示网卡的第二个IP地址，cloudstack支持一个网卡多个IP地址。
主机Plugin vmops 位于cloudstack 代码树中cloudstack/scripts/vm/hypervisor/xenserver,在Xenserver主机/etc/xapi.d/plugins/
Cloudstack对于Xenserver和KVM的安全组主要使用Iptables，ebtable ，ipset，arptables实现。
System VM
def default_network_rules_systemvm(session, args):
......
vmchain = chain_name(vm_name)
.......
delete_rules_for_vm_in_bridge_firewall_chain(vm_name)
#建立iptables chain
try:
util.pread2(['iptables', '-N', vmchain])
except:
util.pread2(['iptables', '-F', vmchain])
#允许所有出流量
allow_egress_traffic(session)
#遍历vif
for vif in vifs:
try:
util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', vif, '-j', vmchain])
util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', vmchain])
util.pread2(['iptables', '-I', vmchain, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', 'RETURN'])
except:
util.SMlog("Failed to program default rules")
return 'false'
util.pread2(['iptables', '-A', vmchain, '-j', 'ACCEPT'])
return 'true'
SSVM Chain规则
Chain s-6-VM (8 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
CPVM Chain规则
Chain v-2-VM (6 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
VRouter Chain规则
Chain r-4-VM (4 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
通过以上三个SystemVM可以看到，基本所有流量都RETURN到BRIDGE-FIREWALL处理
User VM
def default_network_rules(session, args):
vm_name = args.pop('vmName')
vm_ip = args.pop('vmIP')
vm_id = args.pop('vmID')#
vm_mac = args.pop('vmMAC')
sec_ips = args.pop("secIps")
action = "-A"
try:
vm = session.xenapi.VM.get_by_name_label(vm_name)
if len(vm) != 1:
util.SMlog("### Failed to get record for vm " + vm_name)
return 'false'
vm_rec = session.xenapi.VM.get_record(vm[0])
domid = vm_rec.get('domid')
except:
util.SMlog("### Failed to get domid for vm " + vm_name)
return 'false'
if domid == '-1':
util.SMlog("### Failed to get domid for vm (-1): " + vm_name)
return 'false'
vif = "vif" + domid + ".0"
tap = "tap" + domid + ".0"
vifs = [vif]
try:
util.pread2(['ifconfig', tap])
vifs.append(tap)
except:
pass
delete_rules_for_vm_in_bridge_firewall_chain(vm_name)
vmchain = chain_name(vm_name)
vmchain_egress = egress_chain_name(vm_name) #EGRESS
vmchain_default = chain_name_def(vm_name) #INGRESS
destroy_ebtables_rules(vmchain)
#建立三个chain: i-x-y-VM ,i-x-y-VM-eg,i-x-y-VM-def
try:
util.pread2(['iptables', '-N', vmchain])
except:
util.pread2(['iptables', '-F', vmchain])
try:
util.pread2(['iptables', '-N', vmchain_egress])
except:
util.pread2(['iptables', '-F', vmchain_egress])
try:
util.pread2(['iptables', '-N', vmchain_default])
except:
util.pread2(['iptables', '-F', vmchain_default])
vmipset = vm_name
#创建ipset
if create_ipset_forvm(vmipset) == False:
util.SMlog(" failed to create ipset for rule " + str(tokens))
return 'false'
#添加ip到ipset
if add_to_ipset(vmipset, [vm_ip], action ) == False:
util.SMlog(" failed to add vm " + vm_ip + " ip to set ")
return 'false'
#添加第二个ip到ipset
secIpSet = "1"
ips = sec_ips.split(':')
ips.pop()
if ips[0] == "0":
secIpSet = "0";
if secIpSet == "1":
util.SMlog("Adding ipset for secondary ips")
add_to_ipset(vmipset, ips, action)
if write_secip_log_for_vm(vm_name, sec_ips, vm_id) == False:
util.SMlog("Failed to log default network rules, ignoring")
keyword = '--' + get_ipset_keyword()
try:
for v in vifs:
util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-j', vmchain_default])
util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-j', vmchain_default])
#don't let vm spoof its ip address
for v in vifs:
#允许VM的DNS请求，RETURN到上级chain处理
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-p', 'udp', '--dport', '53', '-j', 'RETURN'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', '!', keyword, vmipset, 'src', '-j', 'DROP'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-m', 'set', '!', keyword, vmipset, 'dst', '-j', 'DROP'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-j', vmchain_egress])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-j', vmchain])
except:
util.SMlog("Failed to program default rules for vm " + vm_name)
return 'false'
default_arp_antispoof(vmchain, vifs, vm_ip, vm_mac)
#add default arp rules for secondary ips;
if secIpSet == "1":
util.SMlog("Adding arp rules for sec ip")
arp_rules_vmip(vmchain, vifs, ips, vm_mac, action)
default_ebtables_antispoof_rules(vmchain, vifs, vm_ip, vm_mac)
if write_rule_log__vm(vm_name, vm_id, vm_ip, domid, '_initial_', '-1', vm_mac) == False:
util.SMlog("Failed to log default network rules, ignoring")
util.SMlog("Programmed default rules for vm " + vm_name)
return 'true'
用户VM chain规则
Chain i-2-7-VM (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain i-2-7-VM-eg (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 Chain i-2-7-def (2 references) target prot opt source destination RETURN udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src udp dpt:53 DROP all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged !set i-2-7-VM src DROP all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged !set i-2-7-VM dst i-2-7-VM-eg all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src i-2-7-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged
对于User VM，会创建ebtables和ipset,arptable。并且允许DHCP 67/68端口,否则用户VM将无法自动获得IP。
ebtables
Bridge chain: FORWARD, entries: 3, policy: ACCEPT -j DEFAULT_EBTABLES -i vif21.0 -j i-2-7-VM #入接口 -o vif21.0 -j i-2-7-VM #出接口 Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: DEFAULT_EBTABLES, entries: 12, policy: ACCEPT #DHCP的广播目的地址是255.255.255.255,源地址是0.0.0.0 -p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 67 -j ACCEPT -p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 68 -j ACCEPT -p ARP --arp-op Request -j ACCEPT -p ARP --arp-op Reply -j ACCEPT -p IPv4 -d Broadcast -j DROP -p IPv4 -d Multicast -j DROP -p IPv4 --ip-dst 255.255.255.255 -j DROP -p IPv4 --ip-dst 224.0.0.0/4 -j DROP -p IPv4 -j RETURN -p IPv6 -j DROP -p 802_1Q -j DROP -j DROP Bridge chain: i-2-7-VM, entries: 2, policy: ACCEPT # drop fake dhcp and snooping of dhcp requests -p IPv4 -i vif21.0 --ip-proto udp --ip-dport 68 -j DROP -p IPv4 -o vif21.0 --ip-proto udp --ip-dport 67 -j DROP
IPSet
ipset是以VM名字命名的，member是VM的IP地址。
Name: i-2-7-VM Type: iphash References: 4 Header: hashsize: 1024 probes: 8 resize: 50 Members: 192.168.2xx.xxx
对于target 为RETURN 的会交给上级BRIDGE_FIREWALL 处理。
Chain BRIDGE_FIREWALL (1 references) target prot opt source destination BRIDGE-DEFAULT-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 i-2-7-def all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.1 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.0 --physdev-is-bridged v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.2 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.3 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.1 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.0 --physdev-is-bridged s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.2 --physdev-is-bridged r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif19.1 --physdev-is-bridged r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif19.0 --physdev-is-bridged i-2-7-def all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged
通过以下规则可以发现，安全组所有包都会经FORWARD BRIDGE-FIREWALL,而BRIDGE-FIREWALL的第一条是BRIDGE-DEFAULT-FIREWALL预先处理。默认DHCP的67，68端口允许。
Chain FORWARD (policy ACCEPT) target prot opt source destination BRIDGE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out eth1 --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out eth0 --physdev-is-bridged DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain BRIDGE-DEFAULT-FIREWALL (1 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged udp spt:68 dpt:67 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged udp spt:67 dpt:68
VM组播问题
通过以上规则可以发现，安全组对于组播包IGMP和除了DHCP广播包，其他都是是被丢弃的。
如果VM内部的应用需要组播，需要手动开启删除这个规则。
以上是VM默认规则，下篇将讲解用户自定义规则，涉及到Cloudstack 安全组的核心类 SecurityGroupManagerImpl.java和SecurityGroupManagerImpl2.java ，和一些相关API。