Cloudstack 安全组分析(1)

面向Cloudstack4.2 Security Group XenServer 6.x ,对于KVM原理一样,需要和Agent通信,调用security_group.py。

安全组介绍

安全组是AWS里的一种技术,用于隔离多租户虚拟机,Cloudstack也支持安全组,3.0.x版本只支持Basic Zone的安全组,4.1以后引入Advance Shared Zone的安全组,目前安全组支持的Hypervisor为KVM 和Xen系列,对于VMware可以使用PVLAN也可以使用vShield实现,需要独立开发。

首先要创建带安全组的Shared网络服务,创建Zone,创建VM时,选择新创建的安全组或默认的。如果不选择安全组,会使用默认的安全组。默认情况下,安全组会将VM的Deny All Ingress ,Allow ALL Egress. 

XenServer前置条件

对于XenServer< 6.1,需要安装CSP包来支持安全组,从XenServer6.1开始,CSP已经集成到XenServer安装盘中。

XenServer 6.0.2:
http://download.cloud.com/releases/3.0.1/XS-6.0.2/xenserver-cloud-supp.tgz
XenServer 5.6 SP2:
http://download.cloud.com/releases/2.2.0/xenserver-cloud-supp.tgz
XenServer 6.0:
http://download.cloud.com/releases/3.0/xenserver-cloud-supp.tgz

安装:

#tar xf xenserver-cloud-supp.tgz
#xe-install-supplemental-pack xenserver-cloud-supp.iso

另外需要将网络从OVS改为Bridge。在XenServer机器执行

#cat /etc/xensource/network.conf
openvswitch

执行改为bridge或者手动改为bridge,并重启主机。

# xe-switch-network-backend bridge

另外需要确认以下这些值,如下表示OK。

#sysctl -a|grep bridge 
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-arptables = 1

打开内核转发:

net.ipv4.ip_forward = 1

如果不是,修改

vi /etc/sysctl.conf
#sysctl -p #生效

安全组默认规则代码分析

CitrixResourceBase.java 是负责和XenServer交互的底层资源类。在VM启动时,如果启用安全组,将建立默认规则(Ingress Allow,Egress Deny).

方法为 execute(StartCommand cmd),里面代码片段:

if (_canBridgeFirewall) {
    String result = null;
    if (vmSpec.getType() != VirtualMachine.Type.User) {
        NicTO[] nics = vmSpec.getNics();
        boolean secGrpEnabled = false;
        for (NicTO nic : nics) {
            if (nic.isSecurityGroupEnabled()
                    || (nic.getIsolationUri() != null && nic.getIsolationUri().getScheme()
                            .equalsIgnoreCase(IsolationType.Ec2.toString()))) {
                secGrpEnabled = true;
                break;
            }
        }
        if (secGrpEnabled) {
            result = callHostPlugin(conn, "vmops", "default_network_rules_systemvm", "vmName", vmName);
            if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
                s_logger.warn("Failed to program default network rules for " + vmName);
            } else {
                s_logger.info("Programmed default network rules for " + vmName);
            }
        }

    } else {
        // For user vm, program the rules for each nic if the isolation
        // uri scheme is ec2
        NicTO[] nics = vmSpec.getNics();
        for (NicTO nic : nics) {
            if (nic.isSecurityGroupEnabled() || nic.getIsolationUri() != null
                    && nic.getIsolationUri().getScheme().equalsIgnoreCase(IsolationType.Ec2.toString())) {
                List<String> nicSecIps = nic.getNicSecIps();
                String secIpsStr;
                StringBuilder sb = new StringBuilder();
                if (nicSecIps != null) {
                    for (String ip : nicSecIps) {
                        sb.append(ip).append(":");
                    }
                    secIpsStr = sb.toString();
                } else {
                    secIpsStr = "0:";
                }
                result = callHostPlugin(conn, "vmops", "default_network_rules", "vmName", vmName, "vmIP",
                        nic.getIp(), "vmMAC", nic.getMac(), "vmID", Long.toString(vmSpec.getId()), "secIps",
                        secIpsStr);

                if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
                    s_logger.warn("Failed to program default network rules for " + vmName + " on nic with ip:"
                            + nic.getIp() + " mac:" + nic.getMac());
                } else {
                    s_logger.info("Programmed default network rules for " + vmName + " on nic with ip:"
                            + nic.getIp() + " mac:" + nic.getMac());
                }
            }
        }
    }
}

该方法将判断主机网络是否为bridge,如果不是就不会建立安全组规则。将遍历VM所有网卡,判断是否启用安全组隔离类型为EC2,即安全组隔离。

对于System VM(vrouter,ssvm,cpvm),要单独处理,如果是System VM,会调用主机plugin vmops里的方法default_network_rules_systemvm 参数为VM名字,如果是User VM,会调用default_network_rules,参数为 vmName, vmIP ,vmMAC,,  secIps,secIps表示网卡的第二个IP地址,cloudstack支持一个网卡多个IP地址。

主机Plugin vmops 位于cloudstack 代码树中cloudstack/scripts/vm/hypervisor/xenserver,在Xenserver主机/etc/xapi.d/plugins/

Cloudstack对于Xenserver和KVM的安全组主要使用Iptables,ebtable ,ipset,arptables实现。

System VM 

def default_network_rules_systemvm(session, args):
    ......
    
    vmchain = chain_name(vm_name)
   .......
    delete_rules_for_vm_in_bridge_firewall_chain(vm_name)
    #建立iptables chain
    try:
        util.pread2(['iptables', '-N', vmchain])
    except:
        util.pread2(['iptables', '-F', vmchain])
    #允许所有出流量
    allow_egress_traffic(session)
    #遍历vif
    for vif in vifs:
        try:
            util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', vif, '-j', vmchain])
            util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', vmchain])
            util.pread2(['iptables', '-I', vmchain, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', 'RETURN'])
        except:
            util.SMlog("Failed to program default rules")
            return 'false'
    util.pread2(['iptables', '-A', vmchain, '-j', 'ACCEPT'])
 
    return 'true'

 SSVM Chain规则

Chain s-6-VM (8 references)
pkts bytes target     prot opt in     out source    destination
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged
0     0 ACCEPT     all  --  *  * 0.0.0.0/0 0.0.0.0/0 

 CPVM  Chain规则

Chain v-2-VM (6 references)
pkts bytes target     prot opt in     out source   destination
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0     0.0.0.0/0   PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged
0     0 ACCEPT     all  --  *  * 0.0.0.0/0     0.0.0.0/0 

 VRouter Chain规则


Chain r-4-VM (4 references)
pkts bytes target     prot opt in  out source   destination
0     0 RETURN     all  --  *  * 0.0.0.0/0      0.0.0.0/0   PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged
0     0 RETURN     all  --  *  * 0.0.0.0/0      0.0.0.0/0   PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged
0     0 ACCEPT     all  --  *  * 0.0.0.0/0      0.0.0.0/0 

通过以上三个SystemVM可以看到,基本所有流量都RETURN到BRIDGE-FIREWALL处理

User VM 

def default_network_rules(session, args):
    vm_name = args.pop('vmName')
    vm_ip = args.pop('vmIP')
    vm_id = args.pop('vmID')#
    vm_mac = args.pop('vmMAC')
    sec_ips = args.pop("secIps")
    action = "-A"
    
    try:
        vm = session.xenapi.VM.get_by_name_label(vm_name)
        if len(vm) != 1:
             util.SMlog("### Failed to get record for vm  " + vm_name)
             return 'false'
        vm_rec = session.xenapi.VM.get_record(vm[0])
        domid = vm_rec.get('domid')
    except:
        util.SMlog("### Failed to get domid for vm " + vm_name)
        return 'false'
    if domid == '-1':     
        util.SMlog("### Failed to get domid for vm (-1):  " + vm_name)
        return 'false'
    
    vif = "vif" + domid + ".0"
    tap = "tap" + domid + ".0"
    vifs = [vif]
    try:
        util.pread2(['ifconfig', tap])
        vifs.append(tap)
    except:
        pass

    delete_rules_for_vm_in_bridge_firewall_chain(vm_name)

     
    vmchain =  chain_name(vm_name)
    vmchain_egress =  egress_chain_name(vm_name) #EGRESS
    vmchain_default = chain_name_def(vm_name) #INGRESS
    
    destroy_ebtables_rules(vmchain)
    
#建立三个chain: i-x-y-VM ,i-x-y-VM-eg,i-x-y-VM-def
    try:
        util.pread2(['iptables', '-N', vmchain])
    except:
        util.pread2(['iptables', '-F', vmchain])
    
    try:
        util.pread2(['iptables', '-N', vmchain_egress])
    except:
        util.pread2(['iptables', '-F', vmchain_egress])
        
    try:
        util.pread2(['iptables', '-N', vmchain_default])
    except:
        util.pread2(['iptables', '-F', vmchain_default])        

    vmipset = vm_name
    #创建ipset
    if create_ipset_forvm(vmipset) == False:
       util.SMlog(" failed to create ipset for rule " + str(tokens))
       return 'false'

    #添加ip到ipset
    if add_to_ipset(vmipset, [vm_ip], action ) == False:
       util.SMlog(" failed to add vm " + vm_ip + " ip to set ")
       return 'false'

    #添加第二个ip到ipset
    secIpSet = "1"
    ips = sec_ips.split(':')
    ips.pop()
    if ips[0] == "0":
        secIpSet = "0";

    if secIpSet == "1":
        util.SMlog("Adding ipset for secondary ips")
        add_to_ipset(vmipset, ips, action)
        if write_secip_log_for_vm(vm_name, sec_ips, vm_id) == False:
            util.SMlog("Failed to log default network rules, ignoring")

    keyword = '--' + get_ipset_keyword()

    try:
        for v in vifs:
            util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-j', vmchain_default])
            util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-j', vmchain_default])

        #don't let vm spoof its ip address
        for v in vifs:
            #允许VM的DNS请求,RETURN到上级chain处理
            util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-p', 'udp', '--dport', '53', '-j', 'RETURN'])
            util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', '!', keyword, vmipset, 'src', '-j', 'DROP'])
            util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-m', 'set', '!', keyword, vmipset, 'dst', '-j', 'DROP'])
            util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-j', vmchain_egress])
            util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v,  '-j', vmchain])
    except:
        util.SMlog("Failed to program default rules for vm " + vm_name)
        return 'false'
    
    default_arp_antispoof(vmchain, vifs, vm_ip, vm_mac)
    #add default arp rules for secondary ips;
    if secIpSet == "1":
        util.SMlog("Adding arp rules for sec ip")
        arp_rules_vmip(vmchain, vifs, ips, vm_mac, action)

    default_ebtables_antispoof_rules(vmchain, vifs, vm_ip, vm_mac)
    
    if write_rule_log__vm(vm_name, vm_id, vm_ip, domid, '_initial_', '-1', vm_mac) == False:
        util.SMlog("Failed to log default network rules, ignoring")
        
    util.SMlog("Programmed default rules for vm " + vm_name)
    return 'true'

用户VM chain规则

Chain i-2-7-VM (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain i-2-7-VM-eg (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain i-2-7-def (2 references)
target     prot opt source     destination
RETURN     udp  --  0.0.0.0/0  0.0.0.0/0    PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src udp dpt:53
DROP       all  --  0.0.0.0/0  0.0.0.0/0    PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged !set i-2-7-VM src
DROP       all  --  0.0.0.0/0  0.0.0.0/0    PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged !set i-2-7-VM dst
i-2-7-VM-eg  all  --  0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src
i-2-7-VM   all  --  0.0.0.0/0   0.0.0.0/0   PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged 


对于User VM,会创建ebtables和ipset,arptable。并且允许DHCP 67/68端口,否则用户VM将无法自动获得IP。

ebtables

Bridge chain: FORWARD, entries: 3, policy: ACCEPT
-j DEFAULT_EBTABLES
-i vif21.0 -j i-2-7-VM #入接口
-o vif21.0 -j i-2-7-VM  #出接口

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge chain: DEFAULT_EBTABLES, entries: 12, policy: ACCEPT
#DHCP的广播目的地址是255.255.255.255,源地址是0.0.0.0 
-p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 67 -j ACCEPT
-p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 68 -j ACCEPT
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p IPv4 -d Broadcast -j DROP
-p IPv4 -d Multicast -j DROP
-p IPv4 --ip-dst 255.255.255.255 -j DROP
-p IPv4 --ip-dst 224.0.0.0/4 -j DROP
-p IPv4 -j RETURN
-p IPv6 -j DROP
-p 802_1Q -j DROP
-j DROP

Bridge chain: i-2-7-VM, entries: 2, policy: ACCEPT
# drop fake dhcp and snooping of dhcp requests 
-p IPv4 -i vif21.0 --ip-proto udp --ip-dport 68 -j DROP
-p IPv4 -o vif21.0 --ip-proto udp --ip-dport 67 -j DROP

IPSet

ipset是以VM名字命名的,member是VM的IP地址。

Name: i-2-7-VM
Type: iphash
References: 4
Header: hashsize: 1024 probes: 8 resize: 50
Members:
192.168.2xx.xxx

对于target 为RETURN 的会交给上级BRIDGE_FIREWALL 处理。

Chain BRIDGE_FIREWALL (1 references)
target     prot opt source        destination
BRIDGE-DEFAULT-FIREWALL  all  --  0.0.0.0/0 0.0.0.0/0
i-2-7-def  all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged
r-4-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged
r-4-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif17.1 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif17.0 --physdev-is-bridged
v-2-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif17.2 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif18.3 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif18.1 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif18.0 --physdev-is-bridged
s-6-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif18.2 --physdev-is-bridged
r-4-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif19.1 --physdev-is-bridged
r-4-VM     all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif19.0 --physdev-is-bridged
i-2-7-def  all  --  0.0.0.0/0     0.0.0.0/0     PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged

通过以下规则可以发现,安全组所有包都会经FORWARD BRIDGE-FIREWALL,而BRIDGE-FIREWALL的第一条是BRIDGE-DEFAULT-FIREWALL预先处理。默认DHCP的67,68端口允许。

Chain FORWARD (policy ACCEPT)
target     prot opt source           destination
BRIDGE-FIREWALL  all  --  0.0.0.0/0  0.0.0.0/0 PHYSDEV match --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0        0.0.0.0/0    PHYSDEV match --physdev-out eth1 --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0        0.0.0.0/0    PHYSDEV match --physdev-out eth0 --physdev-is-bridged
DROP       all  --  0.0.0.0/0        0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain BRIDGE-DEFAULT-FIREWALL (1 references)
target     prot opt source         destination
ACCEPT     all  --  0.0.0.0/0      0.0.0.0/0    state RELATED,ESTABLISHED
ACCEPT     udp  --  0.0.0.0/0      0.0.0.0/0    PHYSDEV match --physdev-is-bridged udp spt:68 dpt:67
ACCEPT     udp  --  0.0.0.0/0      0.0.0.0/0    PHYSDEV match --physdev-is-bridged udp spt:67 dpt:68 

VM组播问题

通过以上规则可以发现,安全组对于组播包IGMP和除了DHCP广播包,其他都是是被丢弃的。

如果VM内部的应用需要组播,需要手动开启删除这个规则。

以上是VM默认规则,下篇将讲解用户自定义规则,涉及到Cloudstack 安全组的核心类 SecurityGroupManagerImpl.java和SecurityGroupManagerImpl2.java ,和一些相关API。


你可能感兴趣的:(CloudStack)