VXLAN如何实现同网段跨节点通信

VXLAN是一种封装协议，将二层数据包封装在三层数据包中。出现主要为了解决vlan id不足问题。

vxlan报文

为了测试vxlan来实现跨节点间同子网通信，利用netns搭建如下网络环境，拓扑如下，设置命令见文末

+-----------------+                           +-----------------+
|node-1           |                           |node-2           |
| +---------+     |                           | +---------+     |
| |   br0   |     |                           | |   br0   |     |
| +---------+     |                           | +---------+     |
| |         |     |                           | |         |     |
| br0       |     |                           | br0       |     |
|4e:69:85:99:e0:04|                           |e6:39:1c:37:20:a2|
|192.168.101.2/24 |                           |192.168.101.3/24 |
|           |     |                           |           |     |
|         vxlan0  |                           |         vxlan0  |
|4e:69:85:99:e0:04|                           |e6:39:1c:37:20:a2|
|                 |          +-----+          |                 |
|node1-eth -------|----------| sw0 |----------|-------node-2-eth|
|ae:93:c0:59:9e:a0|          +-----+          |4e:8b:ee:bd:58:b5|
|192.168.1.2/24   |                           |192.168.1.3/24   |
+-----------------+                           +-----------------+

node-1和node-2分别有一个192.168.101.0/24的ip，vxlan如何实现这两个节点该网段的互通？
node-1，2上br0网桥的br0口上都配置192.168.101.0网段ip，并且添加一个vxlan0的tun设备，vxlan0配置remote和local及vni id。
当node-1中ping 192.168.101.3时：

1. 查本地路由表，匹配后走br0，所以sip=192.168.101.2 dip=192.168.101.3 smac=[MAC node-1 br0] dmac=[MAC node-2 br0]

   ➜  ~ ip netns exec node-1 ip r | grep 192.168.101.0
   192.168.101.0/24 dev br0 proto kernel scope link src 192.168.101.2
   

2. 数据包到br0后，br0是个switch，查fdb(forward database)决定数据包发给vxlan0的port

   ➜  ~ ip netns exec node-1 bridge fdb | grep e6:39:1c:37:20:a2
   e6:39:1c:37:20:a2 dev vxlan0 master br0
   e6:39:1c:37:20:a2 dev vxlan0 dst 192.168.1.3 self
   

3. 二层数据包到vxlan口后根据vxlan的配置，设置vxlan头，udp头，和ip头

   ➜  ~ ip netns exec node-1 ip -d l show vxlan0
   3: vxlan0:  mtu 1450 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
       link/ether 4e:69:85:99:e0:04 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535
       vxlan id 1111 remote 192.168.1.3 local 192.168.1.2 dev node1-eth srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
       bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.4e:69:85:99:e0:4 designated_root 8000.4e:69:85:99:e0:4 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
   

4. 封装后的三层数据包再通过查路由表后，将数据包封装以太网头部后从node1-eth口发出

   ➜  ~ ip netns exec node-1 ip r | grep 192.168.1.0
   192.168.1.0/24 dev node1-eth proto kernel scope link src 192.168.1.2
   

在node-2抓包如下:

➜  ~ ip netns exec node-2 tcpdump -i any -nnev
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
14:33:49.402662 node2-eth In  ifindex 10 ae:93:c0:59:9e:a0 ethertype IPv4 (0x0800), length 154: (tos 0x0, ttl 64, id 10391, offset 0, flags [none], proto UDP (17), length 134)
    192.168.1.2.45186 > 192.168.1.3.4789: VXLAN, flags [I] (0x08), vni 1111
4e:69:85:99:e0:04 > e6:39:1c:37:20:a2, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 6168, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.101.2 > 192.168.101.3: ICMP echo request, id 64021, seq 1, length 64
14:33:49.402662 vxlan0 In  ifindex 3 4e:69:85:99:e0:04 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 6168, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.101.2 > 192.168.101.3: ICMP echo request, id 64021, seq 1, length 64
14:33:49.402662 br0   In  ifindex 2 4e:69:85:99:e0:04 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 6168, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.101.2 > 192.168.101.3: ICMP echo request, id 64021, seq 1, length 64
14:33:49.402678 br0   Out ifindex 2 e6:39:1c:37:20:a2 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 26146, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.101.3 > 192.168.101.2: ICMP echo reply, id 64021, seq 1, length 64
14:33:49.402679 vxlan0 Out ifindex 3 e6:39:1c:37:20:a2 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 26146, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.101.3 > 192.168.101.2: ICMP echo reply, id 64021, seq 1, length 64
14:33:49.402680 node2-eth Out ifindex 10 4e:8b:ee:bd:58:b5 ethertype IPv4 (0x0800), length 154: (tos 0x0, ttl 64, id 2944, offset 0, flags [none], proto UDP (17), length 134)
    192.168.1.3.45186 > 192.168.1.2.4789: VXLAN, flags [I] (0x08), vni 1111
e6:39:1c:37:20:a2 > 4e:69:85:99:e0:04, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 26146, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.101.3 > 192.168.101.2: ICMP echo reply, id 64021, seq 1, length 64

6 packets captured
6 packets received by filter
0 packets dropped by kernel

---

参考文档
rfc7348 (ietf.org)
附录
环境设置脚本

brctl addbr sw0
ip l set sw0 up
ip netns add node-1
ip netns add node-2
ip l add node1-eth type veth peer node1-tap
ip l add node2-eth type veth peer node2-tap
ip l set node1-eth netns node-1
ip l set node2-eth netns node-2
brctl addif sw0 node1-tap
brctl addif sw0 node2-tap
ip l set node1-tap up
ip l set node2-tap up
ip netns exec node-1 ip l set lo up
ip netns exec node-2 ip l set lo up
ip netns exec node-1 ip l set node1-eth up
ip netns exec node-2 ip l set node2-eth up
ip netns exec node-1 ip a add 192.168.1.2/24 dev node1-eth
ip netns exec node-2 ip a add 192.168.1.3/24 dev node2-eth
ip netns exec node-1 brctl addbr br0
ip netns exec node-2 brctl addbr br0
ip netns exec node-1 ip l set br0 up
ip netns exec node-2 ip l set br0 up
ip netns exec node-1 ip a add 192.168.101.2/24 dev br0
ip netns exec node-2 ip a add 192.168.101.3/24 dev br0
# 设置单播点对点的vxlan（如果设置多播使用命令 ip l add vxlan0 type vxlan id 1111 dstport 4789 group 239.1.1.1 dev node1-eth）
ip netns exec node-1 ip l add vxlan0 type vxlan id 1111 dstport 4789 remote 192.168.1.3 local 192.168.1.2 dev node1-eth
ip netns exec node-2 ip l add vxlan0 type vxlan id 1111 dstport 4789 remote 192.168.1.2 local 192.168.1.3 dev node2-eth
ip netns exec node-1 ip l set vxlan0 up
ip netns exec node-2 ip l set vxlan0 up
ip netns exec node-1 brctl addif br0 vxlan0
ip netns exec node-2 brctl addif br0 vxlan0