Neutron总结-linuxbridge+vxlan网络
本篇文章介绍如何规划及创建linuxbridge+vxlan网络，实现实例间及实例与外部的通讯。读者应对OpenStack及其网络服务Neutron有初步的了解。
1.规划网络
部署节点为一个controller节点(包含网络节点)，两个compute节点。controller节点有3个网卡，分别为eth0(管理和API网络，CIDR为192.168.128.0/24)、eth1(租户网络，CIDR为10.10.10.0/24)、eth2(外部网络，CIDR为11.11.11.0/24，不设置IP)；compute节点有2个网卡，分别为eth0(管理和API网络，CIDR为192.168.128.0/24)、eth1(租户网络，CIDR为10.10.10.0/24)。
这里用VMware Workstation下面的三个虚拟机分别作为controller、compute1、compute2节点，其网卡设置为：
- 每个主机的eth0网卡作为 API&Management 网络，使用的是“NAT模式”，可以连接到外网下载OpenStack软件
- 每个主机的eth1网卡作为 Tenant(VM) 网络，是承载VxLAN的底层网络，使用的是“仅主机模式 ”
- controller节点也作为网络节点，需要模拟OpenStack的外网，增加网卡eth2，这里也选择“仅主机模式”
具体IP设置为：
controller节点
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.31
netmask 255.255.255.0
gateway 192.168.128.2
# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.10
netmask 255.255.255.0
# The external network interface
auto eth2
iface eth2 inet manualcompute1节点
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.32
netmask 255.255.255.0
gateway 192.168.128.2
# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.11
netmask 255.255.255.0compute2节点
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.34
netmask 255.255.255.0
gateway 192.168.128.2
# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.12
netmask 255.255.255.0Neutron网络方案为：
- Neutron ML2的Type Driver为vxlan
- Neutron ML2的Mechanism Driver为LinuxBridge
- Neutron L2 Agent为LinuxBridge
2.创建虚拟网络
修改配置文件
/etc/neutron/plugins/ml2/ml2_conf.ini
/etc/neutron/l3_agent.ini
/etc/neutron/dhcp_agent.ini
controller节点 /etc/neutron/plugins/ml2/linuxbridge_agent.ini
compute节点 /etc/neutron/plugins/ml2/linuxbridge_agent.ini
securitygroup、vxlan的配置同上(注意local_ip是eth1网卡的IP地址)
配置文件修改后，需要重启网络相关服务。
登陆到dashboard，可以查看网络服务信息。
controller节点启动了 neturon-metadata-agent、neutron-l3-agent、neutron-linuxbridge-agent、neutron-dhcp-agent
compute节点启动了 neutron-linuxbridge-agent
创建vxlan100网络，网络地址为172.16.100.0/24
创建vxlan101网络，网络地址为172.16.101.0/24
创建外部网络（外部网络为flat网络），物理网络为external(和前面的配置一致，否则报错)，网络地址为11.11.11.0/24
创建路由器，把vxlan网络和外部网络连通
我们的网络环境就准备好了。
3.测试网络
创建3个实例，2个选择vxlan100网络，1个选择vxlan101网络。
其中：test1、test3被调度到compute1节点，test2被调度到compute2节点。
查看网络拓扑。两个vxlan网络均通过路由器连接到了外网，并且两个vxlan网络也通过路由器连接。所以理论上test1、test2、test3能互通，并且也能连通外部网络。
为了保证外部能ping通以及能ssh登陆到实例，需要在安全组那加两条规则
下面测试网络的连通性：
同一vxlan：172.16.100.11 ping 172.16.100.13
不同vxlan：172.16.100.11 ping 172.16.101.12
外网：172.16.100.11 ping 11.11.11.111 (确保外网相同网段有一台机器)
目前的网络结构如下
查看各个节点上面的虚拟网络设备
controller节点
root@controller:~# brctl show
bridge name bridge id STP enabled interfaces
brq02444a0c-47 8000.2a7c30cba574 no tap3de43a36-db
tapd018e93e-83
vxlan-100
brq662b5cb3-38 8000.000c29c2468d no eth2
tapec56cb87-d6
brq68a5be3b-16 8000.467e8814dbf9 no tapbc8ecf37-95
tapc095787f-94
vxlan-101
virbr0 8000.52540057a3c4 yes virbr0-niccompute1节点
root@compute1:~# brctl show
bridge name bridge id STP enabled interfaces
brq02444a0c-47 8000.7e309ef03102 no tapa4604188-24
vxlan-100
brq68a5be3b-16 8000.12614635535c no tapf830c325-3a
vxlan-101
virbr0 8000.52540066ffc3 yes virbr0-niccompute2节点
root@compute2:~# brctl show
bridge name bridge id STP enabled interfaces
brq02444a0c-47 8000.721f3ae01492 no tap73ed9130-1e
vxlan-100
brq68a5be3b-16 8000.8a1f5b8c25ce no vxlan-101
virbr0 8000.52540066ffc3 yes virbr0-niccontroller节点1个Router、2个DHCP分别在各自的namespace下
root@controller:~# ip netns
qdhcp-02444a0c-47d1-48ef-9de7-a8614a5fd799
qdhcp-68a5be3b-1696-4bd6-9785-1c9954a714e9
qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617可以通过exec查看router细节
root@controller:~# ip netns exec qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: qr-bc8ecf37-95@if10: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:b8:46:02 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.1/24 brd 172.16.101.255 scope global qr-bc8ecf37-95
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feb8:4602/64 scope link
valid_lft forever preferred_lft forever
3: qr-d018e93e-83@if11: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:81:ec:b1 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.1/24 brd 172.16.100.255 scope global qr-d018e93e-83
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe81:ecb1/64 scope link
valid_lft forever preferred_lft forever
4: qg-ec56cb87-d6@if12: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:01:36:c6 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.12/24 brd 11.11.11.255 scope global qg-ec56cb87-d6
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe01:36c6/64 scope link
valid_lft forever preferred_lft forever 查看router的路由
root@controller:~# ip netns exec qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617 route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 11.11.11.1 0.0.0.0 UG 0 0 0 qg-ec56cb87-d6
11.11.11.0 * 255.255.255.0 U 0 0 0 qg-ec56cb87-d6
172.16.100.0 * 255.255.255.0 U 0 0 0 qr-d018e93e-83
172.16.101.0 * 255.255.255.0 U 0 0 0 qr-bc8ecf37-95router的iptables
root@controller:~# ip netns exec qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i qg-ec56cb87-d6 ! -o qg-ec56cb87-d6 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-ec56cb87-d6 -j SNAT --to-source 11.11.11.12
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.12
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat可以通过exec查看dhcp细节
vxlan100的dhcp
root@controller:~# ip netns exec qdhcp-02444a0c-47d1-48ef-9de7-a8614a5fd799 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ns-3de43a36-db@if9: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:d5:60:09 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.10/24 brd 172.16.100.255 scope global ns-3de43a36-db
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-3de43a36-db
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fed5:6009/64 scope link
valid_lft forever preferred_lft forever vxlan101的dhcp
root@controller:~# ip netns exec qdhcp-68a5be3b-1696-4bd6-9785-1c9954a714e9 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ns-c095787f-94@if8: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:8a:e7:fb brd ff:ff:ff:ff:ff:ff
inet 172.16.101.10/24 brd 172.16.101.255 scope global ns-c095787f-94
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-c095787f-94
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe8a:e7fb/64 scope link
valid_lft forever preferred_lft forever 再看下更详细的网络结构
controller节点
compute节点
如果需要从外部网络访问虚拟机，则要添加Floating IP
test1添加floating IP为11.11.11.13
测试网络连通：
外网：11.11.11.111 ping 11.11.11.13
root@ubuntu:~# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:0f brd ff:ff:ff:ff:ff:ff
inet 192.168.128.10/24 brd 192.168.128.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d0f/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:19 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.10/24 brd 192.168.1.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d19/64 scope link
valid_lft forever preferred_lft forever
4: eth2: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:23 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.111/24 brd 11.11.11.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d23/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu:~# ping 11.11.11.13
PING 11.11.11.13 (11.11.11.13) 56(84) bytes of data.
64 bytes from 11.11.11.13: icmp_seq=1 ttl=63 time=9.12 ms
64 bytes from 11.11.11.13: icmp_seq=2 ttl=63 time=3.40 ms
64 bytes from 11.11.11.13: icmp_seq=3 ttl=63 time=2.74 ms
64 bytes from 11.11.11.13: icmp_seq=4 ttl=63 time=1.82 ms 外网：11.11.11.111 ssh登录到 11.11.11.13
root@ubuntu:~# ssh [email protected]
cirros@11.11.11.13's password:
$ ip a
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:08:df:ad brd ff:ff:ff:ff:ff:ff
inet 172.16.100.11/24 brd 172.16.100.255 scope global eth0
inet6 fe80::f816:3eff:fe08:dfad/64 scope link
valid_lft forever preferred_lft forever 查看controller节点router的变化
root@controller:~# ip netns exec qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: qr-bc8ecf37-95@if10: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:b8:46:02 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.1/24 brd 172.16.101.255 scope global qr-bc8ecf37-95
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feb8:4602/64 scope link
valid_lft forever preferred_lft forever
3: qr-d018e93e-83@if11: mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:81:ec:b1 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.1/24 brd 172.16.100.255 scope global qr-d018e93e-83
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe81:ecb1/64 scope link
valid_lft forever preferred_lft forever
4: qg-ec56cb87-d6@if12: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:01:36:c6 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.12/24 brd 11.11.11.255 scope global qg-ec56cb87-d6
valid_lft forever preferred_lft forever
inet 11.11.11.13/32 brd 11.11.11.13 scope global qg-ec56cb87-d6
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe01:36c6/64 scope link
valid_lft forever preferred_lft forever iptables
root@controller:~# ip netns exec qrouter-a88fa0ad-5e9d-4f63-9044-82d247c73617 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 11.11.11.13/32 -j DNAT --to-destination 172.16.100.11
-A neutron-l3-agent-POSTROUTING ! -i qg-ec56cb87-d6 ! -o qg-ec56cb87-d6 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 11.11.11.13/32 -j DNAT --to-destination 172.16.100.11
-A neutron-l3-agent-float-snat -s 172.16.100.11/32 -j SNAT --to-source 11.11.11.13
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-ec56cb87-d6 -j SNAT --to-source 11.11.11.12
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.12
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat这样我们就创建并且测试了LinuxBridge+vxlan网络，验证了网络的连通性。并对Neutron虚拟网络实现的细节进行了描述，希望对大家有帮助。
在Neutron的学习总结过程中，参考了网络上面大量有价值的文档，在这里对无私分享的同学们表示衷心感谢！尤其是《每天5分钟学习OpenStack》，写的非常详细，本篇文档主要参考了里面的内容，在这里重点推荐：）。