Xilinx ZYNQ Ultrascale+ 性能测试之 PL/PS PCIe Root Port NVMe

Xilinx MPSOC

Xilinx MPSoc 为 XCZU4EV-SFVC784AAZ
8G DDR4 * 4
PL 端通过 M.2 形式导出 x4 PCIe
PS 端通过 M.2 形式导出 x2 PCIe

NVMe M.2 盘

我们手上一共有3个，都支持 PCIe Gen3 x4
A. SAMSUNG 250GB 970 EVO Plus
B. Western Digital 500GB Blue SN550
C. 爱国者 128GB P2000
全部格式化为 EXT4 格式

DD测试，写入速度大概135MByte/s，读取速度350MByte/s
写入的瓶颈主要是因为占满了一个单核，top显示CPU占用率 25%
time dd if=/dev/zero of=./write_tmp bs=2M count=5000
time dd if=/dev/nvme0n1p1 of=/dev/null bs=2M count=5000

PL 8G PCIe x4

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
浮动值比较大，写入速度在 420~500 MByte/s 之间

B. Western Digital 500GB Blue SN550
比较稳定，写入速度在 450MByte/s 左右

C. 爱国者 128GB P2000
写入速度稳定在 100MByte/s 左右

lspci 显示信息分别如下

00:00.0 PCI bridge: Xilinx Corporation Device 9134 (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
LnkSta: Speed 8GT/s (ok), Width x4 (ok)

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 8GT/s (ok), Width x4 (ok)

01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 8GT/s (ok), Width x4 (ok)

01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 8GT/s (ok), Width x4 (ok)

PS 5G PCIe x2

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
写入速度稳定在 420MByte/s 左右

B. Western Digital 500GB Blue SN550
PS 端不支持这个磁盘，会反复提示
nvme nvme0: I/O 8 QID 0 timeout, completion polled
nvme nvme0: I/O 9 QID 0 timeout, completion polled

C. 爱国者 128GB P2000
写入速度稳定在100MByte/s 左右

lspci 显示信息分别如下

0001:00:00.0 PCI bridge: Xilinx Corporation Device d024 (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM not supported
LnkSta: Speed 5GT/s (ok), Width x2 (ok)

0001:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 5GT/s (downgraded), Width x2 (downgraded)

0001:01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 5GT/s (downgraded), Width x2 (downgraded)

PL 8G PCIe x1

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
写入速度稳定在 430MByte/s 左右

B. Western Digital 500GB Blue SN550
写入速度稳定在 435MByte/s 左右

C. 爱国者 128GB P2000
写入速度稳定在 60MByte/s 左右

lspci 显示信息分别如下

00:00.0 PCI bridge: Xilinx Corporation Device 9131 (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM not supported
LnkSta: Speed 8GT/s (ok), Width x1 (ok)

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 8GT/s (ok), Width x1 (downgraded)

01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 8GT/s (ok), Width x1 (downgraded)

01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)

PL 2.5G PCIe x1

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
写入速度稳定在 148MByte/s 左右

这个时候用DD测试读取速度
time dd if=/dev/nvme0n1p1 of=/dev/null bs=2M count=5000
测出来读取数据为200MByte/s，这个 200MByte/s 应该就是 2.5G x1 PCIe总线的瓶颈了。

B. Western Digital 500GB Blue SN550
写入速度稳定在 160MByte/s 左右

C. 爱国者 128GB P2000
写入速度稳定在 60MByte/s 左右

lspci 显示信息分别如下

0000:00:00.0 PCI bridge: Xilinx Corporation Device 9111 (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)

0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)

0000:01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)

0000:01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)

PS 2.5G PCIe x1

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
写入速度稳定在 166MByte/s 左右

B. Western Digital 500GB Blue SN550
不支持

C. 爱国者 128GB P2000
写入速度在 60MByte/s 左右

PS 5G PCIe x1

测试结果如下：
A. SAMSUNG 250GB 970 EVO Plus
写入速度稳定在 295MByte/s 左右

这个时候用DD测试，写入速度仍然为135MByte/s 左右

B. Western Digital 500GB Blue SN550
不支持

C. 爱国者 128GB P2000
写入速度在 80MByte/s 左右，DD测试结果差不多，为80MByte/s 左右

lspci 显示信息分别如下

0001:00:00.0 PCI bridge: Xilinx Corporation Device d024 (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
LnkSta: Speed 5GT/s (ok), Width x1 (ok)

0001:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)

0001:01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)

测试代码如下

void * g_buffer0 = 0;
void * g_buffer1 = 0;
void * g_buffer2 = 0;

const int mem_size = 1024 * 1024;
const int write_count = 1024;

void write_file(void * dat, int d){
	char buffer[128];
	snprintf(buffer, sizeof(buffer), "file%d.dat", dat);

	int fd = open(buffer, O_WRONLY | O_CREAT | O_TRUNC | O_DIRECT, 0777);
	if(fd == -1)return;

	for(int i = 0; i < write_count; i++) {
		ssize_t written = write(fd, dat, mem_size);
	}

	close(fd);
}

int main(){
	g_buffer0 = aligned_alloc(4096, mem_size); // 1M
	g_buffer1 = aligned_alloc(4096, mem_size); // 1M
	g_buffer2 = aligned_alloc(4096, mem_size); // 1M

	memset(g_buffer0, mem_size, 0);
	memset(g_buffer1, mem_size, 1);
	memset(g_buffer2, mem_size, 2);

	time_t b = time(0);

	for(int i = 0; i < 10; ++i){
		write_file(g_buffer0, 0);
		write_file(g_buffer1, 1);
		write_file(g_buffer2, 2);
	}

	system("sync");

	time_t e = time(0);

	time_t d = e - b;

	int speed = 30 * write_count / d;
	printf("Speed:%d MByte/s\n", speed);

	free(g_buffer0);
	free(g_buffer1);
	free(g_buffer2);

	return 0;
}