PLX PCIe Switch PEX8724/PEX8764使用
参考
Exploring the Complexities of PCIe Connectivity and Peer-to-Peer Communication
The mystery of peer-to-peer transfer
pcie-peer-to-peer-communication
PCIe Switch高级功能及应用
SOFTWARE DEVELOPMENT KITS
PCI EXPRESS SWITCHES
P2P
As you told Memory Write transactions requires “valid” Addr of recipient and Data and Memory read transactions requires “Vaild” Addr and data "size " required to creates write or read requests.
For the requests from Downstream port, switch takes care of routing to any Upstream (Root) or Downstream (peer to peer), by comparing with its “Base and Limit” Registers.
Switch Routing:
First check the address on its own bars, if it matched, it will consume. Two bars are available in each Switches.
If not, check its IO/P-MMIO/NP-MMIO Base and Limit Register pairs based on the request type.
If a TLP travels to Upstream port and if it matches to its Base and Limit Registers it will be handled as “Unsupported Request” on secondary interface. ( again it will pass to the downstream port, other than one that it received, since it may be peer-to-peer communication).
If not matches at any interface, it will forwarded to its primary interface as it not matches for the bridge and any function beneath this bridge.
软件架构
PCIe总线架构，
Port Numbering端口号和PCIe设备号的对应关系，
PEX8724有2个Station，最多6+1个Port（Port 8是一个软件虚拟Port），PEX8764有4个Station，最多16个Port，对比PEX8724和PEX8764系统，对于系统上的软件拓扑和实际硬件的对应关系，首先确定上行端口在哪个Station，如果在Station 0，就很简单，比如下面的PEX8764系统，15个下行Port全预留，可以看到1，2下没有EP，对应Station 0 Port 1/2，没有6，7，因为4，5配置成x8。
root@t2080rdb:~# lspci -tv
-[0000:00]---00.0-[01-0f]----00.0-[02-0f]--+-01.0-[03]--
+-02.0-[04]--
+-03.0-[05]----00.0 Samsung Electronics Co Ltd Device a804
+-04.0-[06]----00.0 Device 0731:8000
+-05.0-[07]----00.0 Device 0731:8000
+-08.0-[08]----00.0 Samsung Electronics Co Ltd Device a804
+-09.0-[09]----00.0 Samsung Electronics Co Ltd Device a804
+-0a.0-[0a]----00.0 Samsung Electronics Co Ltd Device a804
+-0b.0-[0b]--
+-0c.0-[0c]----00.0 Samsung Electronics Co Ltd Device a804
+-0d.0-[0d]----00.0 Samsung Electronics Co Ltd Device a804
+-0e.0-[0e]----00.0 Samsung Electronics Co Ltd Device a804
\-0f.0-[0f]----00.0 Samsung Electronics Co Ltd Device a804
下面的PEX8724系统，上行端口接在Station 1 Port 9，x8模式，Station 1还有Port 8，Station 0有4个x4 Port，所以总线号还是0，1，2，3，8这几个值，但Port 8和上行端口在一个Station，它先被识别，占用0，还剩1，2，3，8四个号分给Station 0的4个Port，所以当上行端口不在Station 0时，一一对应的关系就不那么明显了，具体情况具体分析，
root@zynqmp:~# lspci -tv
-[0000:00]---00.0-[01-07]----00.0-[02-07]--+-00.0-[03]----00.0 PLX Technology, Inc. Device 87b1
+-01.0-[04]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
+-02.0-[05]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
+-03.0-[06]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
\-08.0-[07]--
PCI内存映射配置空间，
读取映射空间，
root@zynqmp:~# modprobe pcie_ep pcie_ep_vid=0x10b5 pcie_ep_did=0x8724
root@zynqmp:~# lspci -k -nn
00:00.0 PCI bridge [0604]: Xilinx Corporation Device [10ee:9128]
01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
02:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
02:02.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
02:03.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
Kernel driver in use: pcie_ep
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
Kernel driver in use: nvme
Kernel modules: nvme
04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
Kernel driver in use: nvme
Kernel modules: nvme
05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
Kernel driver in use: nvme
Kernel modules: nvme
06:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
Subsystem: Samsung Electronics Co Ltd Device [144d:a801]
Kernel driver in use: nvme
Kernel modules: nvme
root@zynqmp:~# lspci -v
...
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 64
Memory at a0f00000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=01, secondary=02, subordinate=07, sec-latency=0
I/O behind bridge: None
Memory behind bridge: a0700000-a0efffff [size=8M]
Prefetchable memory behind bridge: 00000000a0000000-00000000a05fffff [size=6M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA
Capabilities: [100] Device Serial Number ca-87-00-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [b00] Latency Tolerance Reporting
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcie_ep
...
根据手册，我们选择上行端口256k的bar来读取，也只有这个01:00.0位置的有bar，其他桥片端口没有bar空间，这里Port 0不是上行端口，读取Port 0的配置空间，
root@zynqmp:~# memtool -32 0xa0f00000 0x40
Reading 0x40 count starting at address 0xA0F00000
0xA0F00000: 872410B5 00100406 060400CA 00010000
0xA0F00010: 00000000 00000000 00030302 000001F1
0xA0F00020: A090A070 A011A001 00000000 00000000
0xA0F00030: 00000000 00000040 00000000 0000012C
0xA0F00040: C8034801 00000008 01876805 6CD11000
0xA0F00050: 00000000 00000011 000000FE 00000000
0xA0F00060: 00000000 00000000 0162A410 00008002
0xA0F00070: 00010810 00796843 20410000 00000CDF
0xA0F00080: 016007C0 00000000 00000000 00040860
0xA0F00090: 00000000 0000010E 00270003 00000000
0xA0F000A0: 00000000 0000000D 872410B5 00000000
0xA0F000B0: 00000000 00000000 00000000 00000000
0xA0F000C0: 00000000 00000000 00000000 00000000
0xA0F000D0: 00000000 00000000 00000000 00000000
0xA0F000E0: 00000000 00000000 00000000 00000000
0xA0F000F0: 00000000 00000000 00000000 00000000
T2080，无需加载驱动，默认取得pcieport可直接读取，
01:00.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 21
Memory at c41000000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=01, secondary=02, subordinate=0f, sec-latency=0
I/O behind bridge: 00000000-00003fff
Memory behind bridge: e1100000-e63fffff
Prefetchable memory behind bridge: 00000000e0000000-00000000e07fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8764
Capabilities: [100] Device Serial Number ab-87-00-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [b00] Latency Tolerance Reporting
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport
root@t2080rdb:~# memtool md -l 0xc41000000
c41000000: b5106487 07041000 ab000406 08000100 ..d.............
c41000010: 000000e1 00000000 01020f00 01310000 .............1..
c41000020: 10e130e6 01e071e0 00000000 00000000 ..0...q.........
c41000030: 00000000 40000000 00000000 00010000 ....@...........
c41000040: 014803c8 08000000 05688701 484104ff .H.......h..HA..
c41000050: 00000000 01000000 ff000000 00000000 ................
c41000060: 00000000 00000000 10a45200 02800000 ..........R.....
c41000070: 1f080900 83584100 00004200 00000000 .....XA...B.....
c41000080: 00000000 00000000 00000000 40080400 ............@...
c41000090: 00000000 0e010000 03000000 00000000 ................
c410000a0: 00000000 0d000000 b5106487 00000000 ..........d.....
c410000b0: 00000000 00000000 00000000 00000000 ................
c410000c0: 00000000 00000000 00000000 00000000 ................
c410000d0: 00000000 00000000 00000000 00000000 ................
c410000e0: 00000000 00000000 00000000 00000000 ................
c410000f0: 00000000 00000000 00000000 00000000 ................
硬件架构
4.4节Hardware Architecture，PEX8724包含两个Station，Station 0对应Port 0 ~ Port 3，Station 1对应Port 8 ~ Port 10，其中Port 8是软件虚拟的，
端口配置
芯片复位
I2C
寄存器访问方法，
写命令格式，
读命令格式稍有不同，Command字段为100b，写数据流程，
start > addr > cmd > data > stop
读数据流程为，
start > addr > cmd > stop > start > addr > data > stop
Read Pacing
Read Pacing是协调资源分配，在大流量设备工作时，防止小流量设备得不到仲裁被饿死，Read Pacing的控制在Port 0和Port 8上，对应Station 0和Station 1，地址1D0，读命令为0x4，0x0，0x3c，0xd0，
root@zynqmp:~# memtool -32 0xa0f001d0 4
Reading 0x4 count starting at address 0xA0F001D0
0xA0F001D0: 1000000F 06000800 03000400 00000002
root@zynqmp:~# memtool -32 0xa0f081d0 4
Reading 0x4 count starting at address 0xA0F081D0
0xA0F081D0: 1000000F 06000800 03000400 00000802