OpenHarmony 3.2使用fio进行I/O性能测试
一、环境配置
- 源码环境：OpenHarmony 3.2 Release
- 平台硬件： RK3568
- 平台软件：OpenHarmony 3.2 Release 64bit
- fio版本：3.35
二、fio简介
fio是一个I/O测试工具，可以运行在Linux、Windows等多种系统之上，可以用来测试本地磁盘、网络存储等的I/O性能。
fio官网：https://fio.readthedocs.io/en/latest/fio_doc.html
fio源码下载：https://brick.kernel.dk/snaps/
三、交叉编译fio源码
参考资料：Openharmony 交叉编译haveged
将下载的fio源码fio-3.35.tar.gz放入OpenHarmony项目源码third_party/目录下进行解压
$ cd third_party
$ tar -zxvf fio-3.35.tar.gz
3.1、配置交叉编译环境
在third_party/fio-3.35/目录下创建交叉编译环境的配置脚本fio_env.sh
#!/bin/bash
#aarch64-linux-ohos
SDK_PATH=../../out/sdk/ohos-sdk/linux
CUR_PATH=$(pwd)
cd $SDK_PATH
export OHOS_SDK=$(pwd)
cd $CUR_PATH
export AS=${OHOS_SDK}/native/llvm/bin/llvm-as
export CC="${OHOS_SDK}/native/llvm/bin/clang --target=aarch64-linux-ohos"
export CXX="${OHOS_SDK}/native/llvm/bin/clang++ --target=aarch64-linux-ohos"
export LD="${OHOS_SDK}/native/llvm/bin/lld --target=aarch64-linux-ohos"
export STRIP=${OHOS_SDK}/native/llvm/bin/llvm-strip
export RANLIB=${OHOS_SDK}/native/llvm/bin/llvm-ranlib
export OBJDUMP=${OHOS_SDK}/native/llvm/bin/llvm-objdump
export OBJCOPY=${OHOS_SDK}/native/llvm/bin/llvm-objcopy
export NM=${OHOS_SDK}/native/llvm/bin/llvm-nm
export AR=${OHOS_SDK}/native/llvm/bin/llvm-ar
export CFLAGS="-fPIC -D__MUSL__=1"
export CXXFLAGS="-fPIC -D__MUSL__=1"
注意:交叉编译环境与设备CPU架构和软件系统架构有关，需根据实际情况配置和修改。
- RK3568 cpu架构为aarch64，查看方法如下：
$ hdc shell uname -a
Linux localhost 5.10.97 #1 SMP Thu Aug 31 13:59:32 CST 2023 aarch64
- 设备系统软件架构为64bit,查看方法如下：
$ hdc shell getconf LONG_BIT
64
在third_party/fio-3.35/目录下运行fio_env.sh
$ cd third_party/fio-3.35/
$ source fio_env.sh
3.2、编译fio源码
接着上面步骤，在third_party/fio-3.35/目录下执行下面的命令
#配置fio编译环境，cpu架构为aarch64; 安装文件生成路径为./install
$ ./configure --cpu=aarch64 --prefix=$PWD/install
...
Lib-based ioengines dynamic no
TCMalloc support no
seed_buckets 4
#执行编译
$ make && make install
...
install -m 755 -d /home/xxx/third_party/fio-3.35/install/bin
install fio t/fio-genzipf t/fio-btrace2fio t/fio-verify-state ./tools/fio_generate_plots ./tools/plot/fio2gnuplot ./tools/genfio ./tools/fiologparser.py ./tools/hist/fiologparser_hist.py ./tools/hist/fio-histo-log-pctiles.py ./tools/fio_jsonplus_clat2csv /home/xxx/third_party/fio-3.35/install/bin
install -m 755 -d /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./fio.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/fio_generate_plots.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/plot/fio2gnuplot.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 644 ./tools/hist/fiologparser_hist.py.1 /home/xxx/third_party/fio-3.35/install/man/man1
install -m 755 -d /home/xxx/third_party/fio-3.35/install/share/fio
install -m 644 ./tools/plot/*gpm /home/xxx/third_party/fio-3.35/install/share/fio/
编译通过后在third_party/fio-3.35/install/bin目录下生成fio可执行文件
fio可执行文件网盘下载
third_party/fio-3.35$ tree install/bin
install/bin
├── fio
├── fio2gnuplot
├── fio-btrace2fio
├── fio_generate_plots
├── fio-genzipf
├── fio-histo-log-pctiles.py
├── fio_jsonplus_clat2csv
├── fiologparser_hist.py
├── fiologparser.py
├── fio-verify-state
└── genfio
使用file检测fio文件属性
third_party/fio-3.35$ file install/bin/fio
install/bin/fio: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, with debug_info, not stripped
3.3、验证fio可执行文件
将上述生成的fio 放入开发板/system/bin下
$ hdc shell "mount -o remount,rw /";hdc file send xxx\third_party\fio-3.35\install\bin\fio /system/bin
#修改/system/bin/fio 文件权限和用户组，与同目录文件保持属性一致。
$ hdc shell
# cd system/bin
# chmod 755 fio && chown shell:root fio
# 输出fio版本信息，检测fio运行正常与否
# fio -v
fio-3/35
四、测试RK3568 I/O性能
在进行I/O测试前，有必要先了解下fio 常用参数
fio的简单介绍及部分参数翻译
fio 命令行参数详解
| 参数 | 举例 | 说明 |
|---|---|---|
| filename | filename=/data/fio_read_test | 测试文件名，通常选择data分区 |
| direct | direct=1 | 1表示不使用I/O缓存，反之则使用。为了使测试结果更真实，通常设为1。 |
| rw | rw=read / write / randread / randwrite / randrw | I/O类型，可设置为顺序读 、顺序写、随机读、随机写、混合随机读写 |
| bs | bs=4k | 单次I/O的块文件大小，通常为2k、4k、6k、8k等 |
| bsrange | bsrange=512-2048 | 设定单次I/O块文件大小范围 |
| size | size=5G | 设定本次测试文件大小为5GB，每次以bs=4k大小进行I/O测试 |
| numjobs | numjobs=30 | 测试线程数为30 |
| runtime | runtime=120 | 测试时长为120秒，未设置默认写满size=5GB大小 |
| ioengine | ioegine=psync | I/O引擎类型，包括libaio、sync、psync等 |
| rwmixwrite | rwmixwrite=30 | 混合读写模式下，写占比30% |
| group_reporting | 按组展示结果 | |
| output | output=/data/fio_test_result | 输出测试结果至/data/fio_test_result文件中 |
| output-format | output-format=json | 设置fio输出结果的格式 |
| lockmem | lockmem=1g | 只使用1g内存进行测试 |
| iodepth | iodepth 1 | 文件上I/O模块的数量(大于1的iodepth对同步io无意义) |
| thread | 使用pthread_create创建线程代替使用fork创建进程,可在一定程度上节省系统开销 |
4.1、测试实例
- 顺序写
fio -filename=/data/fio_test_ordwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
# fio -filename=/data/fio_test_ordwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads #启用5个线程，对应“numjobs=5”
mytest: Laying out IO file (1 file / 1024MiB)
Jobs: 5 (f=5): [W(5)][98.7%][w=65.0MiB/s][w=65 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1688: Mon Aug 7 18:46:21 2017
write: IOPS=66, BW=66.1MiB/s (69.3MB/s)(5120MiB/77482msec); 0 zone resets
clat (msec): min=8, max=285, avg=75.35, stdev=56.61
lat (msec): min=9, max=285, avg=75.61, stdev=56.57
clat percentiles (msec):
| 1.00th=[ 37], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 45],
| 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 53], 60.00th=[ 54],
| 70.00th=[ 55], 80.00th=[ 85], 90.00th=[ 186], 95.00th=[ 220],
| 99.00th=[ 253], 99.50th=[ 255], 99.90th=[ 266], 99.95th=[ 271],
| 99.99th=[ 288]
bw ( KiB/s): min=20462, max=114818, per=99.81%, avg=67534.76, stdev=7052.44, samples=770
iops : min= 18, max= 111, avg=64.64, stdev= 6.72, samples=770
lat (msec) : 10=0.04%, 20=0.02%, 50=44.75%, 100=39.92%, 250=13.12%
lat (msec) : 500=2.15%
cpu : usr=0.42%, sys=2.60%, ctx=10327, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): #顺序写速率为66.1MiB/s ，共I/O操作5120MiB数据，运行77.482秒。
WRITE: bw=66.1MiB/s (69.3MB/s), 66.1MiB/s-66.1MiB/s (69.3MB/s-69.3MB/s), io=5120MiB (5369MB), run=77482-77482msec
Disk stats (read/write):
mmcblk0: ios=0/10232, merge=0/4, ticks=0/106644, in_queue=106769, util=100.00%
- 顺序读
fio -filename=/data/fio_test_ordread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
# fio -filename=/data/fio_test_ordread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 1024MiB)
Jobs: 5 (f=5): [R(5)][100.0%][r=167MiB/s][r=167 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=5): err= 0: pid=1718: Mon Aug 7 18:53:10 2017
read: IOPS=167, BW=167MiB/s (175MB/s)(5120MiB/30634msec)
clat (usec): min=19427, max=38153, avg=29892.50, stdev=4558.43
lat (usec): min=19430, max=38156, avg=29893.75, stdev=4558.32
clat percentiles (usec):
| 1.00th=[23725], 5.00th=[23987], 10.00th=[23987], 20.00th=[23987],
| 30.00th=[24773], 40.00th=[29754], 50.00th=[29754], 60.00th=[30016],
| 70.00th=[33817], 80.00th=[35914], 90.00th=[35914], 95.00th=[35914],
| 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914],
| 99.99th=[38011]
bw ( KiB/s): min=163510, max=174080, per=100.00%, avg=171331.97, stdev=902.72, samples=305
iops : min= 155, max= 170, avg=166.48, stdev= 0.97, samples=305
lat (msec) : 20=0.02%, 50=99.98%
cpu : usr=0.09%, sys=0.76%, ctx=5127, majf=0, minf=1280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): #顺序读速率为167MiB/s ，共I/O操作5120MiB数据，运行30.634秒。
READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5120MiB (5369MB), run=30634-30634msec
Disk stats (read/write):
mmcblk0: ios=10227/0, merge=0/0, ticks=257561/0, in_queue=257561, util=99.83%
- 随机写
fio -filename=/data/fio_test_randw -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest
# fio -filename=/data/fio_test_randw -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads #启用5个线程，对应“numjobs=5”
mytest: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [w(5)][99.8%][w=23.0MiB/s][w=23 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1651: Mon Aug 7 18:34:55 2017
write: IOPS=49, BW=49.4MiB/s (51.8MB/s)(25.0GiB/517785msec); 0 zone resets
clat (msec): min=7, max=336, avg=100.86, stdev=71.81
lat (msec): min=7, max=337, avg=101.09, stdev=71.77
clat percentiles (msec):
| 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 46],
| 30.00th=[ 52], 40.00th=[ 57], 50.00th=[ 82], 60.00th=[ 83],
| 70.00th=[ 90], 80.00th=[ 184], 90.00th=[ 239], 95.00th=[ 255],
| 99.00th=[ 266], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 288],
| 99.99th=[ 338]
bw ( KiB/s): min=10215, max=118734, per=100.00%, avg=50645.50, stdev=6021.63, samples=5167
iops : min= 5, max= 115, avg=47.89, stdev= 5.89, samples=5167
lat (msec) : 10=0.03%, 20=0.02%, 50=28.41%, 100=45.87%, 250=19.45%
lat (msec) : 500=6.23%
cpu : usr=0.30%, sys=1.63%, ctx=51655, majf=0, minf=0 #cpu使用情况
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% #I/O深度情况
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): #随机写 速率为49.4MiB/s，共I/O操作25GiB数据，运行517.785秒。
WRITE: bw=49.4MiB/s (51.8MB/s), 49.4MiB/s-49.4MiB/s (51.8MB/s-51.8MB/s), io=25.0GiB (26.8GB), run=517785-517785msec
Disk stats (read/write):
mmcblk0: ios=0/51248, merge=0/5, ticks=0/743625, in_queue=743778, util=100.00%
- 随机读
# fio -filename=/data/fio_test_randr -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1M -size=5G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [r(2),_(1),r(2)][99.4%][r=167MiB/s][r=167 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1736: Mon Aug 7 19:00:02 2017
read: IOPS=166, BW=167MiB/s (175MB/s)(25.0GiB/153732msec)
clat (usec): min=6391, max=47451, avg=29965.34, stdev=5073.05
lat (usec): min=6394, max=47454, avg=29966.54, stdev=5073.05
clat percentiles (usec):
| 1.00th=[17957], 5.00th=[17957], 10.00th=[23987], 20.00th=[29754],
| 30.00th=[30016], 40.00th=[30016], 50.00th=[30016], 60.00th=[30016],
| 70.00th=[30016], 80.00th=[35914], 90.00th=[35914], 95.00th=[35914],
| 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206],
| 99.99th=[42206]
bw ( KiB/s): min=153414, max=207179, per=100.00%, avg=170873.53, stdev=1759.68, samples=1531
iops : min= 147, max= 202, avg=166.46, stdev= 1.73, samples=1531
lat (msec) : 10=0.01%, 20=6.49%, 50=93.50%
cpu : usr=0.08%, sys=0.77%, ctx=25649, majf=0, minf=1280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): #随机读 速率为167MiB/s，共I/O操作25GiB数据，运行153.732秒。
READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=25.0GiB (26.8GB), run=153732-153732msec
Disk stats (read/write):
mmcblk0: ios=51145/0, merge=0/0, ticks=1445831/0, in_queue=1445831, util=100.00%
- 随机读&写 50%
fio -filename=/data/fio_test_randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=50 -ioengine=psync -bs=1M -size=2G -numjobs=5 -group_reporting -name=mytest
# fio -filename=/data/fio_test_randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=50 -ioengine=psync -bs=1M -size=2G -numjobs=5 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.35
Starting 5 threads
mytest: Laying out IO file (1 file / 2048MiB)
Jobs: 4 (f=4): [m(4),_(1)][98.6%][r=67.1MiB/s,w=74.1MiB/s][r=67,w=74 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=5): err= 0: pid=1756: Mon Aug 7 19:06:00 2017
read: IOPS=68, BW=68.5MiB/s (71.8MB/s)(5014MiB/73211msec)
clat (usec): min=6113, max=47540, avg=12604.83, stdev=5299.80
lat (usec): min=6114, max=47541, avg=12606.44, stdev=5299.79
clat percentiles (usec):
| 1.00th=[ 6521], 5.00th=[ 6587], 10.00th=[ 6587], 20.00th=[ 6718],
| 30.00th=[ 7242], 40.00th=[12125], 50.00th=[12649], 60.00th=[12911],
| 70.00th=[13042], 80.00th=[18220], 90.00th=[19006], 95.00th=[23987],
| 99.00th=[30278], 99.50th=[30540], 99.90th=[31065], 99.95th=[31065],
| 99.99th=[47449]
bw ( KiB/s): min=10240, max=157749, per=100.00%, avg=70549.60, stdev=5970.51, samples=721
iops : min= 10, max= 154, avg=67.92, stdev= 5.87, samples=721
write: IOPS=71, BW=71.4MiB/s (74.8MB/s)(5226MiB/73211msec); 0 zone resets
clat (msec): min=7, max=114, avg=57.44, stdev=13.11
lat (msec): min=7, max=114, avg=57.60, stdev=13.11
clat percentiles (msec):
| 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 47],
| 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60],
| 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 82],
| 99.00th=[ 90], 99.50th=[ 95], 99.90th=[ 108], 99.95th=[ 112],
| 99.99th=[ 115]
bw ( KiB/s): min=49131, max=92552, per=100.00%, avg=73276.53, stdev=1939.47, samples=724
iops : min= 47, max= 90, avg=70.59, stdev= 1.88, samples=724
lat (msec) : 10=15.42%, 20=31.24%, 50=17.71%, 100=35.50%, 250=0.13%
cpu : usr=0.32%, sys=1.14%, ctx=16027, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5014,5226,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
#随机读 速率为68.5MiB/s，共I/O操作5014MiB数据，运行73.211秒。
READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=5014MiB (5258MB), run=73211-73211msec
#随机写 速率为71.4MiB/s，共I/O操作5226MiB数据，运行73.211秒。
WRITE: bw=71.4MiB/s (74.8MB/s), 71.4MiB/s-71.4MiB/s (74.8MB/s-74.8MB/s), io=5226MiB (5480MB), run=73211-73211msec
Disk stats (read/write):
mmcblk0: ios=10071/10464, merge=0/0, ticks=109367/124484, in_queue=233859, util=100.00%