ENSPLAB笔记：配置VXLAN（分布式网关，BGP EVPN方式）（Part2)
(继续上一篇的5、结果验证）
5.3、同网段，不同Leaf服务器Ping测试
执行操作：172.16.1.10 Ping 172.16.1.20，并在Leaf1连Spine1端口抓包。
在Serv1上执行 Ping 172.16.1.20
PC>ping 172.16.1.20
Ping 172.16.1.20: 32 data bytes, Press Ctrl_C to break
From 172.16.1.10: Destination host unreachable
From 172.16.1.10: Destination host unreachable
From 172.16.1.10: Destination host unreachable
From 172.16.1.10: Destination host unreachable
From 172.16.1.10: Destination host unreachable
如果Ping不通，可能是ENSP bug，此时重新配置业务接入点的bridge-domain，可以解决：
#
[Leaf2]int g1/0/1.3
[Leaf2-GE1/0/1.3]undo bri
[Leaf2-GE1/0/1.3]undo bridge-domain
[Leaf2-GE1/0/1.3]bri
[Leaf2-GE1/0/1.3]bridge-domain 300
[Leaf2-GE1/0/1.3]再次执行Ping操作，可以Ping通。
PC>ping 172.16.1.20
Ping 172.16.1.20: 32 data bytes, Press Ctrl_C to break
From 172.16.1.20: bytes=32 seq=1 ttl=128 time=78 ms
From 172.16.1.20: bytes=32 seq=2 ttl=128 time=79 ms
From 172.16.1.20: bytes=32 seq=3 ttl=128 time=62 ms
From 172.16.1.20: bytes=32 seq=4 ttl=128 time=47 ms
From 172.16.1.20: bytes=32 seq=5 ttl=128 time=62 ms
观察网络状态：
Serv1有远端地址mac信息：
PC>arp -a
Internet Address Physical Address Type
172.16.1.20 54-89-98-E3-2A-D7 dynamic
Leaf1上：
bridge-domain 100有相关mac信息：
[Leaf1]dis mac-address bridge-domain 100
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-98e3-2ad7 -/-/100 30.30.30.30 dynamic -
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-98e3-2ad7 -/-/100 30.30.30.30 dynamic -
-------------------------------------------------------------------------------
Total items: 4但BGP EVPN没有相关mac的学习信息，
通过Wireshark查看，没有BGP Update信息。
Leaf1 bgp e mac route没有变化：
[Leaf1]dis bgp e all routing-table mac-route
Local AS number : 100
BGP Local router ID is 2.2.2.2
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Mac Routes: 2
Route Distinguisher: 20:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
Route Distinguisher: 30:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:0001-0001-0001:0:0.0.0.0 30.30.30.30
EVPN-Instance 100:
Number of Mac Routes: 2
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
* i 30.30.30.30
[Leaf1] Leaf1上的bgp e prefix-route也没有变化：
[Leaf1]dis bgp e all routing-table prefix-route
Local AS number : 100
BGP Local router ID is 2.2.2.2
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Ip Prefix Routes: 8
Route Distinguisher: 200:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*> 0:172.16.1.0:24 0.0.0.0
*> 0:10.10.10.0:24 0.0.0.0
*> 0:172.16.1.1:32 0.0.0.0
*> 0:10.10.10.1:32 0.0.0.0
Route Distinguisher: 300:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:172.16.1.0:24 30.30.30.30
*>i 0:172.16.1.1:32 30.30.30.30
Route Distinguisher: 400:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*>i 0:182.1.1.1:32 40.40.40.40
EVPN-Instance __RD_1_200_1__:
Number of Ip Prefix Routes: 8
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*> 0:172.16.1.0:24 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.0:24 0.0.0.0
*>i 0:182.1.1.1:32 40.40.40.40
*> 0:172.16.1.1:32 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.1:32 0.0.0.0
[Leaf1]所以，单纯从ENSPLAB观察，同网段Ping通，完全依靠数据平面arp完成，控制平面的BGP EVPN没有触发任何动作（没有update，没有mac学习）。
因为同网段Ping，并不解析网关地址，所以没有触发bgp e控制平面学习。
5.4、不同网段，同一Leaf下服务器Ping测试
（Leaf1 GE1/0/0 wireshark抓包）
1、Ping之前检查Leaf1的e prefix-route(type 5)：
[~Leaf1]dis bgp e all routing-table prefix-route
Local AS number : 100
BGP Local router ID is 2.2.2.2
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Ip Prefix Routes: 8
Route Distinguisher: 200:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*> 0:172.16.1.0:24 0.0.0.0
*> 0:10.10.10.0:24 0.0.0.0
*> 0:172.16.1.1:32 0.0.0.0
*> 0:10.10.10.1:32 0.0.0.0
Route Distinguisher: 300:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:172.16.1.0:24 30.30.30.30
*>i 0:172.16.1.1:32 30.30.30.30
Route Distinguisher: 400:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*>i 0:182.1.1.1:32 40.40.40.40
EVPN-Instance __RD_1_200_1__:
Number of Ip Prefix Routes: 8
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*> 0:172.16.1.0:24 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.0:24 0.0.0.0
*>i 0:182.1.1.1:32 40.40.40.40
*> 0:172.16.1.1:32 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.1:32 0.0.0.0
[~Leaf1]查看e前缀路由（IP路由），由于本实验中每个Leaf/BL都分配了独立的L3VPN RD，很容易看出是哪个L3VPN产生的路由信息。
最后查看真正注入到L3VPN instance的路由（EVPN-Instance __RD_1_200_1__），这个是会影响L3VPN（L3VRF）的路由，最终通过BGP选路规则，成为best route，再根据路由器自己的路由选择规则，进入路由表。
另外，此时的/32路由是type 5前缀路由，由直连端口的ip地址产生（import direct)，需要和后面由type 2 irb 路由产生的/32路由区分开来。
2、开始ping:
PC>ping 10.10.10.10
Ping 10.10.10.10: 32 data bytes, Press Ctrl_C to break
Request timeout!
Request timeout!
Request timeout!
Request timeout!
Request timeout!
--- 10.10.10.10 ping statistics ---
5 packet(s) transmitted
0 packet(s) received
100.00% packet loss
PC>ENSP中，没有ping通。
此时可以查看wireshark信息：
第一个为ARP，因为是跨网段Ping，172.16.1.10首先查找自己网关的mac地址。
Frame 179: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface -, id 0
Ethernet II, Src: 38:7d:c8:01:01:00 (38:7d:c8:01:01:00), Dst: 38:7d:c8:03:01:01 (38:7d:c8:03:01:01)
Internet Protocol Version 4, Src: 20.20.20.20, Dst: 30.30.30.30
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Ethernet II, Src: HuaweiTe_b3:20:fb (54:89:98:b3:20:fb), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)
Hardware type: Ethernet (1)
Protocol type: IPv4 (0x0800)
Hardware size: 6
Protocol size: 4
Opcode: request (1)
Sender MAC address: HuaweiTe_b3:20:fb (54:89:98:b3:20:fb)
Sender IP address: 172.16.1.10
Target MAC address: Broadcast (ff:ff:ff:ff:ff:ff)
Target IP address: 172.16.1.1
Exablaze trailer, Device: 0, Port: 0, Timestamp: 08:00:0.000000000000
因为是广播包，所以arp也会通过vxlan发给30.30.30.30进行查找。
Leaf1直接进行了arp reply(172.16.1.1的arp reply）。
正常来说，ping包到Leaf1后，应该正常进行路由转发，但可能是ENSP软件Bug，此时没有转发（按步骤，Leaf1可以路由到10.10.10.0/24网段，但10.10.10.1没有10.10.10.10对应的mac地址，此时也应该arp请求10.10.10.10的mac地址，但这一步没有执行。）
但没关系，此时可以看到产生了一条e irb route:
wireshark抓包：Leaf1发给Spine1的e irb route:
Frame 180: 181 bytes on wire (1448 bits), 181 bytes captured (1448 bits) on interface -, id 0
Ethernet II, Src: 38:7d:c8:01:01:00 (38:7d:c8:01:01:00), Dst: 38:7d:c8:03:01:01 (38:7d:c8:03:01:01)
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
Transmission Control Protocol, Src Port: 50409, Dst Port: 179, Seq: 229, Ack: 229, Len: 127
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 127
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 104
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 32
Carried extended communities: (4 communities)
Route Target: 100:10 [Transitive 2-Octet AS-Specific]
Route Target: 200:10 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
EVPN Router MAC: Router MAC: 70:7b:e8:da:58:76 [Transitive EVPN]
Type: Transitive EVPN (0x06)
Subtype (EVPN): EVPN Router MAC (0x03)
Router MAC: HuaweiTe_da:58:76 (70:7b:e8:da:58:76)
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop: 14141414
IPv4 Address: 20.20.20.20
[Expert Info (Error/Malformed): Unknown Next Hop length (4 bytes)]
Number of Subnetwork points of attachment (SNPA): 0
Network Layer Reachability Information (NLRI)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0000001400000001 (20:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: HuaweiTe_b3:20:fb (54:89:98:b3:20:fb)
IP Address Length: 32
IPv4 address: 172.16.1.10
VNI: 8000
VNI: 100
Leaf1的bgp e路由表可以看到同时产生了对应的type 2 的irb route（RD 20：1下面）：
[Leaf1]dis bgp e all routing-table mac-route
Local AS number : 100
BGP Local router ID is 2.2.2.2
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Mac Routes: 3
Route Distinguisher: 20:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
*> 0:48:5489-98b3-20fb:32:172.16.1.10 0.0.0.0
Route Distinguisher: 30:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:0001-0001-0001:0:0.0.0.0 30.30.30.30
EVPN-Instance 100:
Number of Mac Routes: 3
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
* i 30.30.30.30
*> 0:48:5489-98b3-20fb:32:172.16.1.10 0.0.0.0
[Leaf1]可以看到是在RD20:1（即Leaf1下的e instance 100下产生的）。
此时看Leaf1的前缀路由，没有变化（Leaf1没有产生172.16.1.10的主机路由，即Leaf并不产生直连自己的服务器主机路由信息，只是通过e type 2的IRB路由，告知其他e peer）。
[Leaf1]dis bgp e all routing-table prefix-route
Local AS number : 100
BGP Local router ID is 2.2.2.2
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Ip Prefix Routes: 8
Route Distinguisher: 200:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*> 0:172.16.1.0:24 0.0.0.0
*> 0:10.10.10.0:24 0.0.0.0
*> 0:172.16.1.1:32 0.0.0.0
*> 0:10.10.10.1:32 0.0.0.0
Route Distinguisher: 300:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:172.16.1.0:24 30.30.30.30
*>i 0:172.16.1.1:32 30.30.30.30
Route Distinguisher: 400:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*>i 0:182.1.1.1:32 40.40.40.40
EVPN-Instance __RD_1_200_1__:
Number of Ip Prefix Routes: 8
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:182.1.1.0:24 40.40.40.40
*> 0:172.16.1.0:24 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.0:24 0.0.0.0
*>i 0:182.1.1.1:32 40.40.40.40
*> 0:172.16.1.1:32 0.0.0.0
* i 30.30.30.30
*> 0:10.10.10.1:32 0.0.0.0
[Leaf1]但Leaf2、BL1都产生了172.16.1.10这条主机路由：
Leaf2:
[Leaf2]dis bgp e all routing-table mac-route
Local AS number : 100
BGP Local router ID is 3.3.3.3
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Mac Routes: 3
Route Distinguisher: 20:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:0001-0001-0001:0:0.0.0.0 20.20.20.20
*>i 0:48:5489-98b3-20fb:32:172.16.1.10 20.20.20.20
Route Distinguisher: 30:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
EVPN-Instance 300:
Number of Mac Routes: 3
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
* i 20.20.20.20
*>i 0:48:5489-98b3-20fb:32:172.16.1.10 20.20.20.20
EVPN-Instance __RD_1_300_1__:
Number of Mac Routes: 1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:5489-98b3-20fb:32:172.16.1.10 20.20.20.20
[Leaf2]Leaf2上，可以看到172.16.1.10这条irb路由已经传递过来（RD20：1），并已经分别注入到本地的e instance(mac vrf， EVPN-Instance 300)和 instance(ip vrf， __RD_1_300_1__)。
可以查看irb具体明细信息：
[Leaf2]dis bgp e all routing-table mac-route 0:48:5489-98b3-20fb:32:172.16.1.10
BGP local router ID : 3.3.3.3
Local AS number : 100
Total routes of Route Distinguisher(20:1): 1
BGP routing table entry information of 0:48:5489-98b3-20fb:32:172.16.1.10:
Label information (Received/Applied): 8000 100/NULL
From: 1.1.1.1 (1.1.1.1)
Route Duration: 0d00h23m47s
Relay IP Nexthop: 192.168.13.1
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.20.20.20
Qos information : 0x0
Ext-Community: RT <100 : 10>, RT <200 : 10>, Tunnel Type , Router's MAC <707b-e8da-5876>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255, IGP cost 2
Originator: 2.2.2.2
Cluster list: 1.1.1.1
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-98b3-20fb/48, IP Address/Len: 172.16.1.10/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
EVPN-Instance 300:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-98b3-20fb:32:172.16.1.10:
Route Distinguisher: 20:1
Remote-Cross route
Label information (Received/Applied): 8000 100/NULL
From: 1.1.1.1 (1.1.1.1)
Route Duration: 0d00h23m47s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.20.20.20
Qos information : 0x0
Ext-Community: RT <100 : 10>, RT <200 : 10>, Tunnel Type , Router's MAC <707b-e8da-5876>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 2.2.2.2
Cluster list: 1.1.1.1
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-98b3-20fb/48, IP Address/Len: 172.16.1.10/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
EVPN-Instance __RD_1_300_1__:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-98b3-20fb:32:172.16.1.10:
Route Distinguisher: 20:1
Remote-Cross route
Label information (Received/Applied): 8000 100/NULL
From: 1.1.1.1 (1.1.1.1)
Route Duration: 0d00h23m48s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.20.20.20
Qos information : 0x0
Ext-Community: RT <100 : 10>, RT <200 : 10>, Tunnel Type , Router's MAC <707b-e8da-5876>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 2.2.2.2
Cluster list: 1.1.1.1
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-98b3-20fb/48, IP Address/Len: 172.16.1.10/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
[Leaf2] 查看Leaf2的路由表，可以看到172.16.1.10主机路由：
[Leaf2]dis ip routing-table -instance 1
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to -instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : 1
Destinations : 9 Routes : 9
Destination/Mask Proto Pre Cost Flags NextHop Interface
10.10.10.0/24 IBGP 255 0 RD 20.20.20.20 VXLAN
10.10.10.1/32 IBGP 255 0 RD 20.20.20.20 VXLAN
172.16.1.0/24 Direct 0 0 D 172.16.1.1 Vbdif300
172.16.1.1/32 Direct 0 0 D 127.0.0.1 Vbdif300
172.16.1.10/32 IBGP 255 0 RD 20.20.20.20 VXLAN
172.16.1.255/32 Direct 0 0 D 127.0.0.1 Vbdif300
182.1.1.0/24 IBGP 255 0 RD 40.40.40.40 VXLAN
182.1.1.1/32 IBGP 255 0 RD 40.40.40.40 VXLAN
255.255.255.255/32 Direct 0 0 D 127.0.0.1 InLoopBack0
[Leaf2]路由172.16.1.10具体明细：
[Leaf2]dis ip routing-table -instance 1 172.16.1.10 verbose
Route Flags: R - relay, D - download to fib, T - to -instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : 1
Summary Count : 1
Destination: 172.16.1.10/32
Protocol: IBGP Process ID: 0
Preference: 255 Cost: 0
NextHop: 20.20.20.20 Neighbour: 1.1.1.1
State: Active Adv Relied Age: 00h27m02s
Tag: 0 Priority: low
Label: NULL QoSInfo: 0x0
IndirectID: 0x1000074 Instance:
RelayNextHop: 0.0.0.0 Interface: VXLAN
TunnelID: 0x0000000027f0000001 Flags: RD
[Leaf2]查看二层mac信息：
[Leaf2]dis mac-address bridge-domain 300
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-98b3-20fb -/-/300 20.20.20.20 dynamic -
5489-98b3-20fb -/-/300 20.20.20.20 dynamic -
-------------------------------------------------------------------------------
Total items: 2
[Leaf2]在Leaf1上，可以查看对应的arp信息
[Leaf1]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.12.2 387d-c801-0100 I GE1/0/0
192.168.12.1 387d-c803-0101 12 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif100 1
172.16.1.10 5489-98b3-20fb 20 D/BD100 GE1/0/1.1 1
10.10.10.1 707b-e8da-5876 I Vbdif200 1
----------------------------------------------------------------------------------------
Total:5 Dynamic:2 Static:0 Interface:3 OpenFlow:0
Redirect:0
[Leaf1]Leaf2的arp表，没有对应信息。
[Leaf2]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.13.3 387d-c802-0100 I GE1/0/0
192.168.13.1 387d-c803-0102 11 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif300 1
----------------------------------------------------------------------------------------
Total:3 Dynamic:1 Static:0 Interface:2 OpenFlow:0
Redirect:0
[Leaf2]可以登录到BL1，进行类似的检查。
[BL1]dis bgp e all routing-table mac-route
Local AS number : 100
BGP Local router ID is 4.4.4.4
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Mac Routes: 3
Route Distinguisher: 20:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:0001-0001-0001:0:0.0.0.0 20.20.20.20
*>i 0:48:5489-98b3-20fb:32:172.16.1.10 20.20.20.20
Route Distinguisher: 30:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:0001-0001-0001:0:0.0.0.0 30.30.30.30
EVPN-Instance __RD_1_400_1__:
Number of Mac Routes: 1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*>i 0:48:5489-98b3-20fb:32:172.16.1.10 20.20.20.2020:1下有172.16.1.10这条irb route。
已经注入到L3VPN（EVPN-Instance __RD_1_400_1__）。
查看L3VPN（L3VRF）的路由表，存在172.16.1.10/32主机路由。
[BL1]dis ip routing-table -instance 1
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to -instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : 1
Destinations : 9 Routes : 9
Destination/Mask Proto Pre Cost Flags NextHop Interface
10.10.10.0/24 IBGP 255 0 RD 20.20.20.20 VXLAN
10.10.10.1/32 IBGP 255 0 RD 20.20.20.20 VXLAN
172.16.1.0/24 IBGP 255 0 RD 20.20.20.20 VXLAN
172.16.1.1/32 IBGP 255 0 RD 20.20.20.20 VXLAN
172.16.1.10/32 IBGP 255 0 RD 20.20.20.20 VXLAN
182.1.1.0/24 Direct 0 0 D 182.1.1.1 GE1/0/1
182.1.1.1/32 Direct 0 0 D 127.0.0.1 GE1/0/1
182.1.1.255/32 Direct 0 0 D 127.0.0.1 GE1/0/1
255.255.255.255/32 Direct 0 0 D 127.0.0.1 InLoopBack0
[BL1]以上是IRB路由的检查过程。
irb路由会携带mac/ip/l2vni/l3vni/router mac等信息，远端可以学习mac/l2vni注入对应的mac vrf(l2 vni)；学习ip/l3vni注入对应的ip vrf(l3)。
继续ping通测试，由于ensp bug，在Leaf1的二层端口int ge1/0/1.2下执行undo bridge-domain/bridge-domain操作，类似上面的操作。
然后172.16.1.10可以正常ping通10.10.10.10。
PC>ping 10.10.10.10
Ping 10.10.10.10: 32 data bytes, Press Ctrl_C to break
From 10.10.10.10: bytes=32 seq=1 ttl=127 time=62 ms
From 10.10.10.10: bytes=32 seq=2 ttl=127 time=94 ms
From 10.10.10.10: bytes=32 seq=3 ttl=127 time=78 ms
From 10.10.10.10: bytes=32 seq=4 ttl=127 time=78 ms
From 10.10.10.10: bytes=32 seq=5 ttl=127 time=63 ms
正常ping的情况下，可以看到Leaf1由完整的mac地址信息（不需要进行arp查找）：
[Leaf1]dis mac-address
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-9896-70c5 -/-/200 GE1/0/1.2 dynamic -
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-9896-70c5 -/-/200 GE1/0/1.2 dynamic -
-------------------------------------------------------------------------------
Total items: 4
[Leaf1]对端口执行shutdown/undo shutdown，
[Leaf1]int g1/0/1
[Leaf1-GE1/0/1]shud
[Leaf1-GE1/0/1]shutd
[Leaf1-GE1/0/1]shutdown
[Leaf1-GE1/0/1]undo shut
[Leaf1-GE1/0/1]undo shutdown 此时Leaf1上的mac信息已经清楚，之前学习的主机路由也撤回，再次执行ping 操作。
此时ENSP可以正常工作了！！
PC>ping 10.10.10.10
Ping 10.10.10.10: 32 data bytes, Press Ctrl_C to break
Request timeout!
From 10.10.10.10: bytes=32 seq=2 ttl=127 time=78 ms
From 10.10.10.10: bytes=32 seq=3 ttl=127 time=78 ms
From 10.10.10.10: bytes=32 seq=4 ttl=127 time=78 ms
From 10.10.10.10: bytes=32 seq=5 ttl=127 time=78 ms
对Leaf1 GE1/0/1端口（连接二层交换机端口）抓包：
172.16.1.10 ping 10.10.10.10，
172.16.1.10首先arp，请求172.16.1.1的mac。
然后发出第一个ping包（23）.
Leaf1收到第一个ping包，希望通过网关10.10.10.1转发，10.10.10.1请求10.10.10.10的mac。
正常后，可以正常Ping通了。
同时抓包Leaf1--Spine1端口，和之前一样，第一个是arp(172.16.1.10发出的广播包），后面是Leaf1发出的bgp e type 2 update(172.16.1.10的irb路由）。
Frame 1291: 181 bytes on wire (1448 bits), 181 bytes captured (1448 bits) on interface -, id 0
Ethernet II, Src: 38:7d:c8:01:01:00 (38:7d:c8:01:01:00), Dst: 38:7d:c8:03:01:01 (38:7d:c8:03:01:01)
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
Transmission Control Protocol, Src Port: 50409, Dst Port: 179, Seq: 2769, Ack: 2858, Len: 127
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 127
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 104
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop: 14141414
Number of Subnetwork points of attachment (SNPA): 0
Network Layer Reachability Information (NLRI)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0000001400000001 (20:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: HuaweiTe_b3:20:fb (54:89:98:b3:20:fb)
IP Address Length: 32
IPv4 address: 172.16.1.10
VNI: 8000
VNI: 100
和之前是一样的。
------------
重新实验：
172.16.1.10 ping 10.10.10.10
不通。
查看mac/arp:
[Leaf1]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.12.2 387d-c801-0100 I GE1/0/0
192.168.12.1 387d-c803-0101 4 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif100 1
172.16.1.10 5489-98b3-20fb 17 D/BD100 GE1/0/1.1 1
10.10.10.1 707b-e8da-5876 I Vbdif200 1
10.10.10.10 5489-9896-70c5 4 D/BD200 GE1/0/1.2 1
----------------------------------------------------------------------------------------
Total:6 Dynamic:3 Static:0 Interface:3 OpenFlow:0
Redirect:0
[Leaf1]dis mac
[Leaf1]dis mac-ad
[Leaf1]dis mac-address
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
-------------------------------------------------------------------------------
Total items: 2
[Leaf1]10.10.10.10有arp信息，但对应的mac，在mac表中没有信息，所以无法转发出去。此时arp和二层信息是不同步的。
shutdown ge1/0/1端口（连接二层交换机端口），
dis mac/dis arp，该端口侧的信息已经清除：
[Leaf1]dis mac
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Total items: 0
[Leaf1]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.12.2 387d-c801-0100 I GE1/0/0
192.168.12.1 387d-c803-0101 19 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif100 1
10.10.10.1 707b-e8da-5876 I Vbdif200 1
----------------------------------------------------------------------------------------
Total:4 Dynamic:1 Static:0 Interface:3 OpenFlow:0
Redirect:0
[Leaf1]undo shutdown ge1/0/1后，能正常ping通了。
此时查看mac-add表格和arp表格：
[Leaf1-GE1/0/1]dis mac
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-9896-70c5 -/-/200 GE1/0/1.2 dynamic -
5489-98b3-20fb -/-/100 GE1/0/1.1 dynamic -
5489-9896-70c5 -/-/200 GE1/0/1.2 dynamic -
-------------------------------------------------------------------------------
Total items: 4
[Leaf1-GE1/0/1]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.12.2 387d-c801-0100 I GE1/0/0
192.168.12.1 387d-c803-0101 14 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif100 1
172.16.1.10 5489-98b3-20fb 18 D/BD100 GE1/0/1.1 1
10.10.10.1 707b-e8da-5876 I Vbdif200 1
10.10.10.10 5489-9896-70c5 18 D/BD200 GE1/0/1.2 1
----------------------------------------------------------------------------------------
Total:6 Dynamic:3 Static:0 Interface:3 OpenFlow:0
Redirect:0
[Leaf1-GE1/0/1]arp表项显示18分钟后超时（缺省20分钟timeout），mac表项没显示。。。可以查看mac地址aging time为5分钟。。
[Leaf1]arp timeout ?
INTEGER<60-86400> ARP timeout value(in seconds, default is 1200)[Leaf1-GE1/0/1]dis mac-address aging-time
Aging time: 300 second(s)
[Leaf1-GE1/0/1]过了4分钟后，mac信息没有了，arp还有：
[Leaf1-GE1/0/1]dis mac
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Total items: 0
[Leaf1-GE1/0/1]dis arp
ARP Entry Types: D - Dynamic, S - Static, I - Interface, O - OpenFlow, RD - Redirect
EXP: Expire-time VLAN:VLAN or Bridge Domain
IP ADDRESS MAC ADDRESS EXP(M) TYPE/VLAN INTERFACE VPN-INSTANCE
----------------------------------------------------------------------------------------
192.168.12.2 387d-c801-0100 I GE1/0/0
192.168.12.1 387d-c803-0101 9 D GE1/0/0
172.16.1.1 0001-0001-0001 I Vbdif100 1
172.16.1.10 5489-98b3-20fb 14 D/BD100 GE1/0/1.1 1
10.10.10.1 707b-e8da-5876 I Vbdif200 1
10.10.10.10 5489-9896-70c5 14 D/BD200 GE1/0/1.2 1
----------------------------------------------------------------------------------------
Total:6 Dynamic:3 Static:0 Interface:3 OpenFlow:0
Redirect:0
[Leaf1-GE1/0/1]172.16.1.10ping 10.10.10.10，ping不通，此时可以理解为由于存在arp信息，10.10.10.1不发arp，直接根据arp表信息进行封装，试图从端口发出，但没有mac信息，无法发出icmp ping包。。。（因为vbdif端口类似irb端口，从vbdif送出的包，还要依赖bridge-domain的mac地址表进行正常的转发）
此时，需要在10.10.10.10上，主动ping 10.10.10.1，让leaf1上mac表有相应信息后，172.16.1.10才能正常ping 通10.10.10.10。
可以考虑调整arp timeout时间为5分钟，和mac地址timeout时间一致(比如设置为5分钟），这样mac表和arp的时间始终一致。（arp timeout时，会detect是否存活，此时mac表也可以得到更新）。