存储性能测试

概述
存储性能测试主要包括DPU硬件模拟virtio-blk引擎的性能测试，引擎+SPDK bDev走NVMe-o-RoCE到对端NULL bDev的性能测试。

影响测试的因素：1.宿主机的内存读写能力； 2.PCIe带宽； 3.DPU的virtio-blk引擎的能力。

硬件模拟virtio-blk引擎的性能测试
启动SNAP，并创建NULL bDev：

alias spdk_rpc.py=‘spdk_rpc.py -s /run/spdk.sock’
alias snap_rpc.py=‘snap_rpc.py -s /run/spdk.sock’

spdk_rpc.py bdev_null_create null0 2048000 4096
vuid=$(snap_rpc.py virtio_blk_emulation_device_attach --num_queues 8 --queue_depth 128 --num_msix 9 | grep vuid | awk -F ': ’ ‘{print $2}’) | awk -F ‘"’ ‘{print $2}’
echo ${vuid}
snap_rpc.py virtio_blk_controller_create --num_queues 8 --queue_size 128 --seg_max 16 --size_max 4096 --vuid ${vuid} --dbg_bdev_type spdk --bdev null0

spdk_rpc.py bdev_malloc_create 128 512
vuid=$(snap_rpc.py virtio_blk_emulation_device_attach --num_queues 8 --queue_depth 128 --num_msix 9 | grep vuid | awk -F ': ’ ‘{print $2}’) | awk -F ‘"’ ‘{print $2}’
echo ${vuid}
snap_rpc.py virtio_blk_controller_create --num_queues 8 --queue_size 128 --seg_max 16 --size_max 4096 --vuid ${vuid} --dbg_bdev_type spdk --bdev Malloc0

宿主机上跑fio测试性能：

root@ubuntu:~/fio-3.35# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=write --group_reporting --numjobs=32 --filename=/dev/vdb --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=write, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.35
Starting 32 processes
Jobs: 32 (f=32): [W(32)][0.0%][w=9931MiB/s][w=2542k IOPS][eta 06d:22h:39m:44s]

root@ubuntu:~/fio-3.35# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=read --group_reporting --numjobs=32 --filename=/dev/vdb --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=read, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.35
Starting 32 processes
Jobs: 32 (f=32): [R(32)][0.6%][r=10.9GiB/s][r=2868k IOPS][eta 06d:21h:44m:49s]
可以测试两种情况：write / read。

目前影响存储性能的主要瓶颈在于宿主机的访存速度，主要是4K小数据的访存速度，相当于网络测试里面的pps能力不够。有待更换到8458P机型之后进一步测试。

8458P机器测试结果（NULL bDev）:

root@ubuntu:~# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=randwrite --group_reporting --numjobs=32 --filename=/dev/vdc --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=randwrite, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.28
Starting 32 processes
Jobs: 32 (f=32): [w(32)][0.0%][w=5878MiB/s][w=1505k IOPS][eta 06d:22h:39m:41s]

root@ubuntu:~# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=write --group_reporting --numjobs=32 --filename=/dev/vdc --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=write, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.28
Starting 32 processes
Jobs: 32 (f=32): [W(32)][0.0%][w=14.4GiB/s][w=3782k IOPS][eta 06d:22h:39m:43s]

8458P机器测试结果（ESSD bDev，4个后端bs）:

root@ubuntu:~# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=write --group_reporting --numjobs=32 --filename=/dev/vdb --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=write, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.28
Starting 32 processes
Jobs: 32 (f=32): [W(32)][0.0%][w=1753MiB/s][w=449k IOPS][eta 06d:22h:35m:40s]

root@ubuntu:~# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=randwrite --group_reporting --numjobs=32 --filename=/dev/vdb --iomem=shmhuge
sequential_write_bandwidth_test: (g=0): rw=randwrite, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.28
Starting 32 processes
Jobs: 32 (f=32): [w(32)][0.0%][w=546MiB/s][w=140k IOPS][eta 06d:22h:37m:57s]

引擎+SPDK bDev走NVMe-o-RoCE到对端NULL bDev的性能测试
为了模拟实际的ESSD，采用了SPDK bDev走NVMe-o-RoCE的方式，对端用NULL bDEv，用来测试virtio-blk引擎+RoCE的性能。

板卡上编译SPDK：

apt install -y autoconf libtool python3-pyelftools libaio-dev libncurses-dev
mkdir -p /opt/my/spdk
tar -xf spdk-22.05-all.tar.gz
cd spdk/
vim ./dpdk/config/arm/meson.build
Delete Unsupported part number
error('Unsupported part number @0@ of implementer @1@. ’
.format(part_number, implementer_id) +
'Please add support for it or use the generic ’ +
‘(-Dplatform=generic) build.’)

./configure --with-rdma --without-isal --disable-tests --disable-unit-tests --prefix=/opt/my/spdk --with-shared

make install
cp -r dpdk/build/include/* /opt/my/spdk/include/
cp -r dpdk/build/lib/* /opt/my/spdk/lib/
cp -r python /opt/my/spdk
cp scripts/rpc.py /opt/my/spdk/bin/

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/my/spdk/lib ./build/bin/nvmf_tgt -h

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/my/spdk/lib ./build/bin/nvmf_tgt -m 0xFFC0

板卡上配置NVMMe-o-RoCE：

./scripts/rpc.py bdev_get_bdevs
./scripts/rpc.py nvmf_get_subsystems
./scripts/rpc.py bdev_null_create nvme1n1 1024000 4096
./scripts/rpc.py nvmf_create_transport -t “RDMA”
./scripts/rpc.py nvmf_create_subsystem nqn.2020-04.io.spdk:cnode1 -s SPDK001 -a -m 128
./scripts/rpc.py nvmf_subsystem_add_ns nqn.2020-04.io.spdk:cnode1 nvme1n1
./scripts/rpc.py nvmf_subsystem_add_listener nqn.2020-04.io.spdk:cnode1 -t rdma -f ipv4 -s 4420 -a 10.240.24.216
./scripts/rpc.py nvmf_get_subsystems

root@qyyc01-tron-bf3-240024216:/home/aaa/spdk# ./scripts/rpc.py nvmf_get_subsystems
[
{
“nqn”: “nqn.2014-08.org.nvmexpress.discovery”,
“subtype”: “Discovery”,
“listen_addresses”: [],
“allow_any_host”: true,
“hosts”: []
},
{
“nqn”: “nqn.2020-04.io.spdk:cnode1”,
“subtype”: “NVMe”,
“listen_addresses”: [
{
“transport”: “RDMA”,
“trtype”: “RDMA”,
“adrfam”: “IPv4”,
“traddr”: “10.240.24.216”,
“trsvcid”: “4420”
}
],
“allow_any_host”: true,
“hosts”: [],
“serial_number”: “SPDK001”,
“model_number”: “SPDK bdev Controller”,
“max_namespaces”: 128,
“min_cntlid”: 1,
“max_cntlid”: 65519,
“namespaces”: [
{
“nsid”: 1,
“bdev_name”: “nvme1n1”,
“name”: “nvme1n1”,
“nguid”: “94570EBB32D849B5AB6F2E65A3779A80”,
“uuid”: “94570ebb-32d8-49b5-ab6f-2e65a3779a80”
}
]
}
]

对端板卡上可以先用内核nvme rdma验证一下配置是否正确：

modprobe nvme_rdma
nvme discover -t rdma -a 10.240.24.216 -s 4420
nvme connect -t rdma -n nqn.2020-04.io.spdk:cnode1 -a 10.240.24.216 -s 4420
[root@qyyc01-tron-bf3-240024220 ~]# nvme list
Node SN Model Namespace Usage Format FW Rev

---

/dev/nvme0n1 536204E1NSJ4 KBG40ZPZ128G TOSHIBA MEMORY 1 128.04 GB / 128.04 GB 512 B + 0 B AEGA0103
/dev/nvme1n1 SPDK001 SPDK bdev Controller 1 10.74 GB / 10.74 GB 4 KiB + 0 B 22.05.1
rmmod nvme_rdma

对端板卡针对SNAP进行相关配置：

docker cp 1:/usr/sbin/spdk_rpc.py /usr/sbin
docker cp 1:/usr/bin/spdk_rpc.py /usr/bin

alias spdk_rpc.py=‘spdk_rpc.py -s /run/spdk.sock’
alias snap_rpc.py=‘snap_rpc.py -s /run/spdk.sock’

snap_rpc.py emulation_function_list
snap_rpc.py virtio_blk_controller_list

spdk_rpc.py bdev_get_bdevs

spdk_rpc.py bdev_nvme_attach_controller -b nvme0 -t rdma -f ipv4 -a 10.240.24.216 -s 4420 -n nqn.2020-04.io.spdk:cnode1
snap_rpc.py virtio_blk_emulation_device_attach --num_queues 8 --queue_depth 128 --num_msix 9
nbdf=snap_rpc.py emulation_function_list | grep pci_bdf | tail -n 1 | awk '{print $2}' | awk -F'"' '{print $2}'
echo “${nbdf}”
snap_rpc.py virtio_blk_controller_create --num_queues 8 --queue_size 128 --seg_max 16 --size_max 4096 --pci_bdf ${nbdf} --dbg_bdev_type spdk --bdev nvme0n1

对端板卡所在宿主机进行fio测试：

root@ubuntu:~# fio --name=sequential_write_bandwidth_test --size=900G --time_based --ramp_time=1s --runtime=10000m --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 --bs=4K --iodepth=128 --rw=randwrite --group_reporting --numjobs=32 --filename=/dev/vdb
sequential_write_bandwidth_test: (g=0): rw=randwrite, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
…
fio-3.28
Starting 32 processes
Jobs: 32 (f=32): [w(32)][0.0%][w=8797MiB/s][w=2252k IOPS][eta 06d:22h:39m:40s]