【dpdk】qat-getting-started-guide-with-dpdk-v2.0

参考文档

632506-qat-getting-started-guide-v2.0.pdf

21. Intel® QuickAssist (QAT) Crypto Poll Mode Driver — Data Plane Development Kit 23.11.0 documentation (dpdk.org)

1.QAT driver installation

This section provides details on building and installing the software.

1.1. Installation using kernel.org driver

The examples below are based on the C62x device, if you have a different device use the corresponding values in the above table.

In BIOS ensure that SRIOV is enabled and either:

• Disable VT-d or
• Enable VT-d and set "intel_iommu=on iommu=pt" in the grub file.

Check that the QAT driver is loaded on your system, by executing:

lsmod | grep qa

You should see the kernel module for your device listed, e.g.:

qat_c62x               5626  0
intel_qat              82336  1 qat_c62x

Next, you need to expose the Virtual Functions (VFs) using the sysfs file system.

First find the BDFs (Bus-Device-Function) of the physical functions (PFs) of your device, e.g.:

lspci -d:37c8

You should see output similar to:

1a:00.0 Co-processor: Intel Corporation Device 37c8
3d:00.0 Co-processor: Intel Corporation Device 37c8
3f:00.0 Co-processor: Intel Corporation Device 37c8

Enable the VFs for each PF by echoing the number of VFs per PF to the pci driver:

echo 16 > /sys/bus/pci/drivers/c6xx/0000:1a:00.0/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/0000:3d:00.0/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/0000:3f:00.0/sriov_numvfs

Check that the VFs are available for use. For example lspci -d:37c9 should list 48 VF devices available for a C62x device.

To complete the installation follow the instructions in Binding the available VFs to the vfio-pci driver.

---

#链接start

1.2. Binding the available VFs to the vfio-pci driver

Note:

• Please note that due to security issues, the usage of older DPDK igb_uio driver is not recommended. This document shows how to use the more secure vfio-pci driver.
• If QAT fails to bind to vfio-pci on Linux kernel 5.9+, please see the QATE-39220 and QATE-7495 issues in IDZ doc which details the constraint about trusted guests and add disable_denylist=1 to the vfio-pci params to use QAT. See also this patch description.

Unbind the VFs from the stock driver so they can be bound to the vfio-pci driver.

1.2.1. For an Intel® QuickAssist Technology DH895xCC device

The unbind command below assumes BDFs of 03:01.00-03:04.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 4); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:03:0${device}.${fn}; \
    done; \
done

1.2.2. For an Intel® QuickAssist Technology C62x device

The unbind command below assumes BDFs of 1a:01.00-1a:02.07, 3d:01.00-3d:02.07 and 3f:01.00-3f:02.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 2); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:1a:0${device}.${fn}; \
        usertools/dpdk-devbind.py -u 0000:3d:0${device}.${fn}; \
        usertools/dpdk-devbind.py -u 0000:3f:0${device}.${fn}; \
    done; \
done

1.2.3. For Intel® QuickAssist Technology C3xxx or 200xx or D15xx device

The unbind command below assumes BDFs of 01:01.00-01:02.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 2); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:01:0${device}.${fn}; \
    done; \
done

1.2.4. Bind to the vfio-pci driver

Load the vfio-pci driver, bind the VF PCI Device id to it using the dpdk-devbind.py script then use the --status option to confirm the VF devices are now in use by vfio-pci kernel driver, e.g. for the C62x device:

cd to the top-level DPDK directory
modprobe vfio-pci
usertools/dpdk-devbind.py -b vfio-pci 0000:03:01.1
usertools/dpdk-devbind.py --status

Use modprobe vfio-pci disable_denylist=1 from kernel 5.9 onwards. See note in the section Binding the available VFs to the vfio-pci driver above.

1.2.5. Testing

QAT SYM crypto PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>cryptodev_qat_autotest

QAT ASYM crypto PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>cryptodev_qat_asym_autotest

QAT compression PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>compressdev_autotest

1.2.6. Debugging

There are 2 sets of trace available via the dynamic logging feature:

• pmd.qat.dp exposes trace on the data-path.
• pmd.qat.general exposes all other trace.

pmd.qat exposes both sets of traces. They can be enabled using the log-level option (where 8=maximum log level) on the process cmdline, e.g. using any of the following:

--log-level="pmd.qat.general,8"
--log-level="pmd.qat.dp,8"
--log-level="pmd.qat,8"

Note

The global RTE_LOG_DP_LEVEL overrides data-path trace so must be set to RTE_LOG_DEBUG to see all the trace. This variable is in config/rte_config.h for meson build. Also the dynamic global log level overrides both sets of trace, so e.g. no QAT trace would display in this case:

--log-level="7" --log-level="pmd.qat.general,8"

#链接end

---

Note

If the QAT kernel modules are not loaded and you see an error like Failed to load MMP firmware qat_895xcc_mmp.bin in kernel logs, this may be as a result of not using a distribution, but just updating the kernel directly.

Download firmware from the kernel firmware repo.

Copy qat binaries to /lib/firmware:

cp qat_895xcc.bin /lib/firmware
cp qat_895xcc_mmp.bin /lib/firmware

Change to your linux source root directory and start the qat kernel modules:

insmod ./drivers/crypto/qat/qat_common/intel_qat.ko
insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko

Note

If you see the following warning in /var/log/messages it can be ignored: IOMMU should be enabled for SR-IOV to work correctly.

1.3. Installation using IDZ QAT driver

浏览器输入https://01.org/ 进入Projects->QuickAssist Technology

或：

Download the latest QuickAssist Technology Driver from Intel Developer Zone. Consult the Quick Start Guide at the same URL for further information.

The steps below assume you are:

• Building on a platform with one C62x device.
• Using package qat1.7.l.4.2.0-000xx.tar.gz.
• On Fedora26 kernel 4.11.11-300.fc26.x86_64.

In the BIOS ensure that SRIOV is enabled and VT-d is disabled.

Uninstall any existing QAT driver, for example by running:

• ./installer.sh uninstall in the directory where originally installed.

Build and install the SRIOV-enabled QAT driver:

mkdir /QAT
cd /QAT

# Copy the package to this location and unpack
tar zxof qat1.7.l.4.2.0-000xx.tar.gz

./configure --enable-icp-sriov=host
make install

You can use cat /sys/kernel/debug/qat/version/fw to confirm the driver is correctly installed and is using firmware version 4.2.0. You can use lspci -d:37c9 to confirm the presence of the 16 VF devices available per C62x PF.

Confirm the driver is correctly installed and is using firmware version 4.2.0:

cat /sys/kernel/debug/qat/version/fw

Confirm the presence of 48 VF devices - 16 per PF:

lspci -d:37c9

To complete the installation - follow instructions in 1.2.Binding the available VFs to the vfio-pci driver

Note

If using a later kernel and the build fails with an error relating to strict_stroul not being available apply the following patch:

/QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h
+ #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5)
+ #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
+ #else
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38)
#define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
#else
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25)
#define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));}
#else
#define STR_TO_64(str, base, num, endPtr)                                 \
     do {                                                               \
           if (str[0] == '-')                                           \
           {                                                            \
                *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \
           }else {                                                      \
                *(num) = simple_strtoull((str), &(endPtr), (base));      \
           }                                                            \
     } while(0)
+ #endif
#endif
#endif

Note

If the build fails due to missing header files you may need to do following:

sudo yum install zlib-devel
sudo yum install openssl-devel
sudo yum install libudev-devel

Note

If the build or install fails due to mismatching kernel sources you may need to do the following:

sudo yum install kernel-headers-`uname -r`
sudo yum install kernel-src-`uname -r`
sudo yum install kernel-devel-`uname -r`

Note

If the build fails on newer GCC versions (such as GCC 12) with an error relating to -lc not being found, apply the following patch:

/QAT/quickassist/lookaside/access_layer/src/Makefile
cd $(ICP_FINAL_OUTPUT_DIR);\
cmd="$(LINKER) $(LIB_SHARED_FLAGS) -o \
  $(LIB_SHARED) $(ADDITIONAL_OBJECTS) $(ADDITIONAL_LIBS) *.o -lpthread -ludev \
- -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
- -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
- -losal -Bdynamic -lc"; \
+ -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
+ -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
+ -losal -Bdynamic -L/lib/x86_64-linux-gnu/ -lc"; \
echo "$$cmd"; \
$$cmd

Followed by this patch:

/QAT/quickassist/build_system/build_files/OS/linux_common_user_space_rules.mk
@echo 'Creating shared library ${LIB_SHARED}'; \
cd $($(PROG_ACY)_FINAL_OUTPUT_DIR);\
-  echo $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) -lc;\
-  $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) -lc ;
+  echo $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) \
+  -L/lib/x86_64-linux-gnu/ -lc;\
+  $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) \
+  -L/lib/x86_64-linux-gnu/ -lc ;

2.upstreamed acceleration software installation

​ 2.1 QATzip @github仓库

​ 2.2 QATlib @github仓库

3.实验过程（using kernel.org driver）

 cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt intel_iommu=on quiet splash"
GRUB_CMDLINE_LINUX=""

grub-mkconfig -o /boot/grub/grub.cfg
reboot

lsmod |grep qat
qat_4xxx               20480  0
intel_qat             270336  1 qat_4xxx
crc8                   16384  1 intel_qat
authenc                16384  1 intel_qat

lspci |grep Co-processor
76:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
78:00.0 Co-processor: Intel Corporation Device 2710
f3:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
f5:00.0 Co-processor: Intel Corporation Device 2710

cat /sys/bus/pci/drivers/4xxx/0000\:
0000:76:00.0/ 0000:f3:00.0/     #两个 

echo 16 > /sys/bus/pci/drivers/4xxx/0000\:76\:00.0/sriov_numvfs
cat /sys/bus/pci/drivers/4xxx/0000\:76\:00.0/sriov_numvfs
16
echo 16 > /sys/bus/pci/drivers/4xxx//0000\:f3\:00.0/sriov_numvfs
cat  /sys/bus/pci/drivers/4xxx//0000\:f3\:00.0/sriov_numvfs
16

lspci -d:4943
76:00.1 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.2 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.3 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.4 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.5 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.6 Co-processor: Intel Corporation Device 4943 (rev 40)
76:00.7 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.0 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.1 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.2 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.3 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.4 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.5 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.6 Co-processor: Intel Corporation Device 4943 (rev 40)
76:01.7 Co-processor: Intel Corporation Device 4943 (rev 40)
76:02.0 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.1 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.2 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.3 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.4 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.5 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.6 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:00.7 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.0 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.1 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.2 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.3 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.4 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.5 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.6 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:01.7 Co-processor: Intel Corporation Device 4943 (rev 40)
f3:02.0 Co-processor: Intel Corporation Device 4943 (rev 40)

modprobe vfio-pci
lsmod |grep vfio
vfio_pci               16384  0
vfio_pci_core          94208  1 vfio_pci
vfio_iommu_type1       49152  0
vfio                   61440  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                73728  1 vfio
irqbypass              16384  2 vfio_pci_core,kvm

# unbind
for device in $(seq 0 1); do \
    for fn in $(seq 1 7); do \
        usertools/dpdk-devbind.py -u 0000:76:0${device}.${fn}; \
    done; \
done
usertools/dpdk-devbind.py -u 0000:76:01.0
usertools/dpdk-devbind.py -u 0000:76:02.0

for device in $(seq 0 1); do \
    for fn in $(seq 1 7); do \
        usertools/dpdk-devbind.py -u 0000:f3:0${device}.${fn}; \
    done; \
done
usertools/dpdk-devbind.py -u 0000:f3:01.0
usertools/dpdk-devbind.py -u 0000:f3:02.0
# bind
for device in $(seq 0 1); do \
    for fn in $(seq 1 7); do \
        usertools/dpdk-devbind.py -b vfio-pci 0000:76:0${device}.${fn}; \
    done; \
done
usertools/dpdk-devbind.py -b vfio-pci 0000:76:01.0
usertools/dpdk-devbind.py -b vfio-pci 0000:76:02.0

for device in $(seq 0 1); do \
    for fn in $(seq 1 7); do \
        usertools/dpdk-devbind.py -b vfio-pci 0000:f3:0${device}.${fn}; \
    done; \
done
usertools/dpdk-devbind.py -b vfio-pci 0000:f3:01.0
usertools/dpdk-devbind.py -b vfio-pci 0000:f3:02.0



示例：

./usertools/dpdk-devbind.py -b vfio-pci 0000:76:00.1

spdk-23.09/dpdk# ./usertools/dpdk-devbind.py --status

Network devices using kernel driver
===================================
0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens38f0 drv=i40e unused=vfio-pci *Active*
0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=ens38f1 drv=i40e unused=vfio-pci
0000:2a:00.0 'MT28908 Family [ConnectX-6] 101b' if=ens3f0np0 drv=mlx5_core unused=vfio-pci
0000:2a:00.1 'MT28908 Family [ConnectX-6] 101b' if=ens3f1np1 drv=mlx5_core unused=vfio-pci

No 'Baseband' devices detected
==============================

Crypto devices using DPDK-compatible driver
===========================================
0000:76:00.1 'Device 4943' drv=vfio-pci unused=
0000:76:00.2 'Device 4943' drv=vfio-pci unused=
0000:76:00.3 'Device 4943' drv=vfio-pci unused=
0000:76:00.4 'Device 4943' drv=vfio-pci unused=
0000:76:00.5 'Device 4943' drv=vfio-pci unused=
0000:76:00.6 'Device 4943' drv=vfio-pci unused=
0000:76:00.7 'Device 4943' drv=vfio-pci unused=
0000:76:01.0 'Device 4943' drv=vfio-pci unused=
0000:76:01.1 'Device 4943' drv=vfio-pci unused=
0000:76:01.2 'Device 4943' drv=vfio-pci unused=
0000:76:01.3 'Device 4943' drv=vfio-pci unused=
0000:76:01.4 'Device 4943' drv=vfio-pci unused=
0000:76:01.5 'Device 4943' drv=vfio-pci unused=
0000:76:01.6 'Device 4943' drv=vfio-pci unused=
0000:76:01.7 'Device 4943' drv=vfio-pci unused=
0000:76:02.0 'Device 4943' drv=vfio-pci unused=
0000:f3:00.1 'Device 4943' drv=vfio-pci unused=
0000:f3:00.2 'Device 4943' drv=vfio-pci unused=
0000:f3:00.3 'Device 4943' drv=vfio-pci unused=
0000:f3:00.4 'Device 4943' drv=vfio-pci unused=
0000:f3:00.5 'Device 4943' drv=vfio-pci unused=
0000:f3:00.6 'Device 4943' drv=vfio-pci unused=
0000:f3:00.7 'Device 4943' drv=vfio-pci unused=
0000:f3:01.0 'Device 4943' drv=vfio-pci unused=
0000:f3:01.1 'Device 4943' drv=vfio-pci unused=
0000:f3:01.2 'Device 4943' drv=vfio-pci unused=
0000:f3:01.3 'Device 4943' drv=vfio-pci unused=
0000:f3:01.4 'Device 4943' drv=vfio-pci unused=
0000:f3:01.5 'Device 4943' drv=vfio-pci unused=
0000:f3:01.6 'Device 4943' drv=vfio-pci unused=
0000:f3:01.7 'Device 4943' drv=vfio-pci unused=
0000:f3:02.0 'Device 4943' drv=vfio-pci unused=

Crypto devices using kernel driver
==================================
0000:76:00.0 'Device 4942' drv=4xxx unused=qat_4xxx,vfio-pci
0000:f3:00.0 'Device 4942' drv=4xxx unused=qat_4xxx,vfio-pci

Other Crypto devices
====================
0000:78:00.0 'Device 2710' unused=vfio-pci
0000:f5:00.0 'Device 2710' unused=vfio-pci

DMA devices using kernel driver
===============================
0000:75:01.0 'Device 0b25' drv=idxd unused=vfio-pci
0000:f2:01.0 'Device 0b25' drv=idxd unused=vfio-pci

Other Eventdev devices
======================
0000:78:00.0 'Device 2710' unused=vfio-pci
0000:f5:00.0 'Device 2710' unused=vfio-pci

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

No 'ML' devices detected
========================



# 分配hugepage
cd dpdk
./usertools/dpdk-hugepages.py --setup 2G


执行./build/app/test/dpdk-test -l1 -n1 -a 0000:76:00.1
**************************************************cryptodev_qat_autotest***********************
cd dpdk
./build/app/test/dpdk-test -l1 -n1 -a 0000:76:00.1
EAL: Detected CPU lcores: 80
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: qat (8086:4943) device: 0000:76:00.1 (socket 0)
CRYPTODEV: Creating cryptodev 0000:76:00.1_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:76:00.1_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:76:00.1_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:76:00.1_qat_asym,socket id: 0, max queue pairs: 0
TELEMETRY: No legacy callbacks, legacy socket not created
APP: HPET is not enabled, using TSC as default timer
RTE>>cryptodev_qat_autotest

...
+ DOCSIS Proto Unit Test Suite : 52/52 passed, 0/52 skipped, 0/52 failed, 0/52 unsupported
 + ------------------------------------------------------- +
 + Sub Testsuites Total :     29
 + Sub Testsuites Skipped :   13
 + Sub Testsuites Passed :    16
 + Sub Testsuites Failed :     0
 + ------------------------------------------------------- +
 + Tests Total :       792
 + Tests Skipped :     496
 + Tests Executed :    494
 + Tests Unsupported:   0
 + Tests Passed :      296
 + Tests Failed :       0
 + ------------------------------------------------------- +
Test OK

***************test_compressdev negative test, errors are expected***********************
 + TestCase [25] : test_compressdev_deflate_im_buffers_SGL_over_2ops_first failed
qat_comp_qp_setup(): qp_id 0 invalid for this device
USER1: Queue pair setup failed
rte_compressdev_stop(): Device with dev_id=0 already stopped
 + TestCase [26] : test_compressdev_deflate_im_buffers_SGL_over_2ops_second failed
 + ------------------------------------------------------- +
 + Test Suite Summary : compressdev unit test suite
 + ------------------------------------------------------- +
 + Tests Total :       27
 + Tests Skipped :      0
 + Tests Executed :    27
 + Tests Unsupported:   0
 + Tests Passed :       1
 + Tests Failed :      26
 + ------------------------------------------------------- +
Test Failed
RTE>>quit


更加详细的日志
spdk-23.09/dpdk# ./build/app/test/dpdk-test -l1 -n1 -a 76:00.1 --log-level=qat:8
EAL: Detected CPU lcores: 80
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: qat (8086:4943) device: 0000:76:00.1 (socket 0)
CRYPTODEV: Creating cryptodev 0000:76:00.1_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:76:00.1_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:76:00.1_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:76:00.1_qat_asym,socket id: 0, max queue pairs: 0
TELEMETRY: No legacy callbacks, legacy socket not created
APP: HPET is not enabled, using TSC as default timer
RTE>>compressdev_autotest
 + ------------------------------------------------------- +
 + Test Suite : compressdev unit test suite
USER1: Running tests on device 0000:76:00.1_qat_comp
 + ------------------------------------------------------- +
USER1: This is a negative test, errors are expected
rte_compressdev_queue_pairs_config(): invalid param: dev 0x5599e5f163c0, nb_queues 0
rte_compressdev_configure(): dev0 rte_comp_dev_queue_pairs_config = -22
rte_compressdev_queue_pair_setup(): Invalid queue_pair_id=0
 + TestCase [ 0] : test_compressdev_invalid_configuration succeeded
qat_comp_qp_setup(): qp_id 0 invalid for this device
USER1: Queue pair setup failed
rte_compressdev_stop(): Device with dev_id=0 already stopped
 + TestCase [ 1] : test_compressdev_deflate_stateless_fixed failed
qat_comp_qp_setup(): qp_id 0 invalid for this device
USER1: Queue pair setup failed
rte_compressdev_stop(): Device with dev_id=0 already stopped
 + TestCase [ 2] : test_compressdev_deflate_stateless_dynamic failed
qat_comp_qp_setup(): qp_id 0 invalid for this device
USER1: Queue pair setup failed
rte_compressdev_stop(): Device with dev_id=0 already stopped
 + TestCase [ 3] : test_compressdev_deflate_stateless_dynamic_big failed
qat_comp_qp_setup(): qp_id 0 invalid for this device
USER1: Queue pair setup failed
rte_compressdev_stop(): Device with dev_id=0 already stopped
 + TestCase [ 4] : test_compressdev_deflate_stateless_multi_op failed
qat_comp_qp_setup(): qp_id 0 invalid for this device

4.实验过程（using IDZ QAT driver）

说明，实际并未使用该种方式。

4.1.Installing Intel QAT (IDZ 方式)

• 下载安装driver

  https://github.com/intel/qatlib

./configure; make install
# Sample programs can be installed by using the “make samples-install”. The sample program “cpa_sample_code runTests=32” can be used to test compression performance.

• 检查安装状态

root@hik-Super-Server:/home/lhx/spdk-23.09# lsmod | grep qat
qat_4xxx               20480  0
intel_qat             270336  1 qat_4xxx
crc8                   16384  1 intel_qat
authenc                16384  1 intel_qat

• 检查 service 状态

# service qat_service start||stop||status||restart||shutdown
# For a system with multiple Intel® QAT Endpoints, you can start, stop or restart each individual device by passing the Intel® QAT Endpoint to be restarted or stopped as a parameter qat_dev, for example:
service qat_service stop qat_dev0
service qat_service stop qat_dev1
# The shutdown qualifier enables the user to bring down all Intel® QAT Endpoints and unload driver modules from the kernel. This contrasts with the stop qualifier, which brings down one or more Intel® QAT Endpoints, but does not unload kernel modules, so other Intel® QAT Endpoints can still run.
# service qat_service status

Checking status of all devices.

There is 3 QAT acceleration device(s) in the system:

qat_dev0 - type: c6xx, inst_id: 0, node_id: 1, bsf: 0000:cc:00.0, #accel: 5 #engines: 10 state: up

qat_dev1 - type: c6xx, inst_id: 1, node_id: 1, bsf: 0000:ce:00.0, #accel: 5 #engines: 10 state: up

qat_dev2 - type: c6xx, inst_id: 2, node_id: 1, bsf: 0000:d0:00.0, #accel: 5 #engines: 10 state: up

4.2.Intel QAT Settings

# 位置 quickassist/utilities/adf_ctl/conf_files
以QAT20.L.1.0.50-00003\quickassist\utilities\adf_ctl\conf_files\4xxx_template.conf为例
并参考programmer guide 进行配置


[GENERAL]
ServicesEnabled = asym;dc

ConfigVersion = 2

#Default values for number of concurrent requests*/
CyNumConcurrentSymRequests = 512
CyNumConcurrentAsymRequests = 64

#Statistics, valid values: 1,0
statsGeneral = 1
statsDh = 1
statsDrbg = 1
statsDsa = 1
statsEcc = 1
statsKeyGen = 1
statsDc = 1
statsLn = 1
statsPrime = 1
statsRsa = 1
statsSym = 1

# Default heartbeat timer is 1s
HeartbeatTimer = 1000

# This flag is to enable SSF features
StorageEnabled = 0

# Disable public key crypto and prime number
# services by specifying a value of 1 (default is 0)
PkeServiceDisabled = 0

# This flag is to enable device auto reset on heartbeat error
AutoResetOnError = 0

# Default value for power management idle interrrupt delay
PmIdleInterruptDelay = 512

# This flag is to enable power management idle support
PmIdleSupport = 1

# This flag is to enable key protection technology
KptEnabled = 1

# Define the maximum SWK count per function can have
# Default value is 1, the maximum value is 128
KptMaxSWKPerFn = 1

# Define the maximum SWK count per pasid can have
# Default value is 1, the maximum value is 128
KptMaxSWKPerPASID = 1

# Define the maximum SWK lifetime in second
# Default value is 0 (eternal of life)
# The maximum value is 31536000 (one year)
KptMaxSWKLifetime = 31536000

# Flag to define whether to allow SWK to be shared among processes
# Default value is 0 (shared mode is off)
KptSWKShared = 0

# Disable AT
ATEnabled = 0
##############################################
# Kernel Instances Section
##############################################
[KERNEL]
NumberCyInstances = 0
NumberDcInstances = 0

##############################################
# ADI Section for Scalable IOV
##############################################
[SIOV]
NumberAdis = 0

##############################################
# User Process Instance Section
##############################################
[SSL]
NumberCyInstances = 2
NumberDcInstances = 2
NumProcesses = 1
LimitDevAccess = 0

# Crypto - User instance #0
Cy0Name = "SSL0"
Cy0IsPolled = 1
# List of core affinities
Cy0CoreAffinity = 0

# Crypto - User instance #1
Cy1Name = "SSL1"
Cy1IsPolled = 1
# List of core affinities
Cy1CoreAffinity = 1

# Data Compression - User instance #0
Dc0Name = "Dc0"
Dc0IsPolled = 1
# List of core affinities
Dc0CoreAffinity = 2

# Data Compression - User instance #1
Dc1Name = "Dc1"
Dc1IsPolled = 1
# List of core affinities
Dc1CoreAffinity = 3