当前网络架构
物理环境: master01(172.31.133.26), node01(172.31.133.27), node02(172.31.133.28)
逻辑网络: 一个逻辑路由器router0,连接三个逻辑交换机bridge0,bridge1,bridge2。每个交换机上有一个端口,连接上pod或者vm。如下:
ovn-nbctl show
switch 679c8a34-5f60-42a8-91a3-5f727e41e2af (bridge1)
port bridge1-vm1
addresses: ["dynamic"]
port bridge1-router0
type: router
router-port: router0-bridge1
switch 7124d3c4-950a-4f4a-9733-db7f204a96c4 (bridge2)
port bridge2-router0
type: router
router-port: router0-bridge2
port bridge2-vm2
addresses: ["dynamic"]
switch 05b72423-5db2-4279-8b5f-5f4b5e8c1408 (bridge0)
port bridge0-router0
type: router
router-port: router0-bridge0
port bridge0-vm0
addresses: ["dynamic"]
router a96ea502-72df-41a2-ac27-0271e60661d2 (router0)
port router0-bridge1
mac: "04:ac:10:ff:34:01"
networks: ["172.66.2.10/24"]
port router0-bridge0
mac: "04:ac:10:ff:34:00"
networks: ["172.66.1.10/24"]
port router0-bridge2
mac: "04:ac:10:ff:34:02"
networks: ["172.66.3.10/24"]
ovs-vsctl show (node02上)
1bd24d64-5b67-4497-b972-5789ba8d4fa7
Bridge br-int
fail_mode: secure
Port "qtest3vm0mqm5g"
Interface "qtest3vm0mqm5g"
Port "ovn-aa8648-0"
Interface "ovn-aa8648-0"
type: geneve
options: {csum="true", key=flow, remote_ip="172.31.133.27"}
Port br-int
Interface br-int
type: internal
Port "ovn-1800fb-0"
Interface "ovn-1800fb-0"
type: geneve
options: {csum="true", key=flow, remote_ip="172.31.133.26"}
ovs_version: "2.11.2"
ovn 出网
相对物理网络
在集群每个节点的ovs网桥br-int上添加一个端口ovn0,通过路由规则,将ovn0作为外部流量出入口。
逻辑网络
添加逻辑交换机join,将各个节点的ovn0流量关联到交换机join的逻辑端口。然后将交换机连到逻辑路由器router0上,组成连通的ovn逻辑网络
组网
创建逻辑交换机jion,并连上逻辑路由器router0。
# 创建逻辑交换机jion
ovn-nbctl ls-add join
# 关联逻辑路由器router0
ovn-nbctl lrp-add router0 router0-join 04:ac:10:ff:35:02 172.66.4.10/24
ovn-nbctl lsp-add join join-router0
ovn-nbctl lsp-set-type join-router0 router
ovn-nbctl lsp-set-addresses join-router0 04:ac:10:ff:35:02
ovn-nbctl lsp-set-options join-router0 router-port=router0-join
# 删除端口
# ovn-nbctl lrp-del router0-jion
# ovn-nbctl lsp-del join-router0
为逻辑交换机jion添加端口
添加port
ovn-nbctl lsp-add join master01
ovn-nbctl lsp-set-addresses master01 "04:ac:10:ff:35:04 172.66.4.1"
ovn-nbctl lsp-add join node01
ovn-nbctl lsp-set-addresses node01 "04:ac:10:ff:35:05 172.66.4.2"
ovn-nbctl lsp-add join node02
ovn-nbctl lsp-set-addresses node02 "04:ac:10:ff:35:06 172.66.4.3"
# 删除port
# ovn-nbctl lsp-del master01
# ovn-nbctl lsp-del node01
# ovn-nbctl lsp-del node02
关联逻辑端口(在每个节点上执行),并设置mac和ip
# 在master01上
ovs-vsctl add-port br-int ovn0 -- set Interface ovn0 type=internal
ovs-vsctl set Interface ovn0 external_ids:iface-id=master01
ifconfig ovn0 172.66.4.1 netmask 255.255.255.0
ifconfig ovn0 hw ether 04:ac:10:ff:35:04
# 在node01上
ovs-vsctl add-port br-int ovn0 -- set Interface ovn0 type=internal
ovs-vsctl set Interface ovn0 external_ids:iface-id=node01
ifconfig ovn0 172.66.4.2 netmask 255.255.255.0
ifconfig ovn0 hw ether 04:ac:10:ff:35:05
# 在node02上
ovs-vsctl add-port br-int ovn0 -- set Interface ovn0 type=internal
ovs-vsctl set Interface ovn0 external_ids:iface-id=node02
ifconfig ovn0 172.66.4.3 netmask 255.255.255.0
ifconfig ovn0 hw ether 04:ac:10:ff:35:06
添加route
现在有三个用户交换机bridge0(172.66.1.10/24),bridge1(172.66.2.10/24),bridge2(172.66.3.10/24),也就是三个网段需要出网
添加宿主机路由
# 需要在每个节点上执行
# 在node01上,(node02 master01上同样操作)
route add -net 172.66.1.0/24 gw 172.66.4.10
route add -net 172.66.2.0/24 gw 172.66.4.10
route add -net 172.66.3.0/24 gw 172.66.4.10
添加ovn虚拟路由器router0路由
# 添加后容器内可以ping通宿主机
ovn-nbctl lr-route-add router0 172.31.133.27 172.66.4.2
ovn-nbctl lr-route-add router0 172.31.133.28 172.66.4.3
ovn-nbctl lr-route-add router0 172.31.133.26 172.66.4.1
展示
[root@master01 /]# ovn-nbctl lr-route-list router0
IPv4 Routes
172.31.133.26 172.66.4.1 dst-ip
172.31.133.27 172.66.4.2 dst-ip
172.31.133.28 172.66.4.3 dst-ip
上面的操作执行后,容器和宿主机可以互通。如果想要出公网,还需要添加如下路由:
添加ovn虚拟路由器router0路由
ovn-nbctl --policy=src-ip lr-route-add router0 172.66.1.12 172.66.4.3
ovn-nbctl --policy=src-ip lr-route-add router0 172.66.3.12 172.66.4.2
ovn-nbctl --policy=src-ip lr-route-add router0 172.66.2.12 172.66.4.3
如上配置的是分布式网关,也可以通过路由规则实现集中式网关如:ovn-nbctl --policy=src-ip lr-route-add router0 172.66.1.12/24 172.66.4.3
,让所有172.66.1.12/24网段的流量都走172.66.4.3出网
展示
[root@master01 /]# ovn-nbctl lr-route-list router0
IPv4 Routes
172.31.133.26 172.66.4.1 dst-ip
172.31.133.27 172.66.4.2 dst-ip
172.31.133.28 172.66.4.3 dst-ip
172.66.1.12 172.66.4.3 src-ip
172.66.2.12 172.66.4.3 src-ip
172.66.3.12 172.66.4.2 src-ip
添加snat
在每需要出网的节点上执行(node01,node02)
> iptables -t nat -A POSTROUTING -s 172.66.3.0/24 -o bond0.1331 -j MASQUERADE
> iptables -t nat -A POSTROUTING -s 172.66.1.0/24 -o bond0.1331 -j MASQUERADE
> iptables -t nat -A POSTROUTING -s 172.66.2.0/24 -o bond0.1331 -j MASQUERADE
测试
我这边逻辑交换机上连的是容器化的虚机
[root@master01 ovn]# virtctl console vm1
[root@vm1 ~]# ping 14.215.177.39
PING 14.215.177.39 (14.215.177.39) 56(84) bytes of data.
64 bytes from 14.215.177.39: icmp_seq=1 ttl=49 time=28.0 ms
64 bytes from 14.215.177.39: icmp_seq=2 ttl=49 time=25.9 ms
--- 14.215.177.39 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 25.943/26.989/28.036/1.059 ms
完美收工