Solarflare网卡开发：openonload 安装与调试

Solarflare网卡开发：openonload 安装与调试
固件信息
可以通过命令查看当前的网卡连接情况

ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 4c:ed:fb:c2:25:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.115/24 brd 192.168.1.255 scope global noprefixroute dynamic enp5s0
       valid_lft 1496sec preferred_lft 1496sec
    inet6 fe80::6804:b02d:fb9f:19b0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0f:53:5d:d3:90 brd ff:ff:ff:ff:ff:ff
4: enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0f:53:5d:d3:91 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.112/24 brd 192.168.1.255 scope global noprefixroute dynamic enp1s0f1
       valid_lft 1544sec preferred_lft 1544sec
    inet6 fe80::20f:53ff:fe5d:d391/64 scope link 
       valid_lft forever preferred_lft forever
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c5:0a:77 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c5:0a:77 brd ff:ff:ff:ff:ff:ff

厂商的MAC地址前四位是00:0F，以此判断哪个网卡为Solarflare的网卡。因此，我们可以看出以下两个网卡为 Solarflare

enp1s0f0
enp1s0f1
我们还可以通过命令查看具体的网卡信息

需要先安装 openonload

ethtool -i enp1s0f1

driver: sfc
version: 4.15.0.1012
firmware-version: 4.7.1.1001 rx1 tx1
expansion-rom-version: 
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

下载驱动
OpenOnload 原先是作为一个独立的开发项目存在，在官网可以获取相关的资讯，或者直接进入下载网页获取指定版本的驱动。

https://www.xilinx.com/support/download/nic-software-and-drivers.html#open
https://support-nic.xilinx.com/wp/onload
https://support-nic.xilinx.com/wp/onload

# M卡下载驱动
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
https://docs.mellanox.com/display/MLNXOFEDv541030/Programming

安装

## 进入相应的文件夹
cd openonload-201811/

## 源代码存放
cd scripts/
ll
total 240K
drwxr-xr-x. 6 fl fl 4.0K Dec  3  2018 .
drwxr-xr-x. 7 fl fl  243 Dec  9 03:39 ..
-rwxr-xr-x. 1 fl fl  154 Apr 12  2018 check_binary_presence
-rwxr-xr-x. 1 fl fl  329 Apr 12  2018 check_header_presence
-rwxr-xr-x. 1 fl fl  358 Apr 12  2018 check_library_presence
-rwxr-xr-x. 1 fl fl  573 Apr 12  2018 check_prototype
-rwxr-xr-x. 1 fl fl 2.2K Nov  8  2018 libc_compat.sh
-rwxr-xr-x. 1 fl fl 1.4K Apr 12  2018 mmake
-rwxr-xr-x. 1 fl fl  16K Nov  8  2018 mmakebuildtree
-rwxr-xr-x. 1 fl fl  15K Nov  8  2018 mmaketool
-rwxr-xr-x. 1 fl fl  13K Nov  9  2018 onload
drwxr-xr-x. 2 fl fl   75 Oct 26  2018 onload_apps
-rwxr-xr-x. 1 fl fl 6.0K Nov  8  2018 onload_build
-rwxr-xr-x. 1 fl fl  27K Dec  3  2018 onload_install
-rwxr-xr-x. 1 fl fl  19K Jul 16  2018 onload_iptables
-rwxr-xr-x. 1 fl fl 8.2K Nov  8  2018 onload_metrics
drwxr-xr-x. 2 fl fl 4.0K Dec  3  2018 onload_misc
drwxr-xr-x. 2 fl fl 4.0K Nov  9  2018 onload_profiles
-rwxr-xr-x. 1 fl fl 2.2K Jul 16  2018 onload_rpmtest
-rwxr-xr-x. 1 fl fl 3.3K Nov  8  2018 onload_tcpdump
-rwxr-xr-x. 1 fl fl 8.0K Nov  9  2018 onload_tool
-rwxr-xr-x. 1 fl fl  748 Apr 12  2018 sfcaffinity
-rwxr-xr-x. 1 fl fl  30K Apr 12  2018 sfcaffinity_config
-rw-r--r--. 1 fl fl  15K Apr 12  2018 sfcaffinity.py
-rwxr-xr-x. 1 fl fl 6.5K Apr 12  2018 sfcirqaffinity
-rw-r--r--. 1 fl fl 4.6K Apr 12  2018 sfcmask.py
drwxr-xr-x. 2 fl fl  108 Dec  3  2018 sh
-rwxr-xr-x. 1 fl fl 1.1K Apr 12  2018 zf_debug

## 搭建环境
locate crti.o
LIBRARY_PATH=/usr/lib64:$LIBRARY_PATH
export LIBRARY_PATH

./onload_build
## 执行安装
./onload_install
#加载 onload
onload_tool reload

## 开启 rx1 tx1 模式
## Ref： https://support.solarflare.com/docs/SF-103837-CD
## Adapters Software Support
sfboot --adapter=enp1s0f1 firmware-variant=ultra-low-latency
## 需要重启
sudo reboot now

## 查看即可显示
ethtool -i enp1s0f1
driver: sfc
version: 4.15.0.1012
firmware-version: 6.2.7.1001 rx1 tx1
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

卸载

#unload onload
onload_tool unload
#remove sfc
modprobe –r sfc
#if found any sfc rpm, then remove them
rpm -qa | grep sfc
rpm -e *sfc*.rpm
#unintall onload
./onload_misc/onload_uninstall

使用
OpenOnload有两种模式。

Spinning模式（busy-wait）中，加速进程的每个线程会占用一整个CPU core，始终处于waiting状态。通过htop可以看到该CPU的使用率为100%。spinning模式速度更快，但是要注意CPU core的数量。
Interrupt模式中，线程不会占用满一个CPU core，但可以将中断Interrupt放在一个CPU core。interrupt模式也有加速效果，理论上比spinning略差一些。当服务器上总线程数大于CPU core的数量时，interrupt可能优于spinning，需要测试来论证。
使用 spinning 模式加速应用
查看CPU core序号（注意，一般服务器是从0开始排序，也有不正常的版本，请查证。否则OpenOnload选择的CPU core可能不存在）

egrep "(id|processo).*:|^ *$" /proc/cpuinfo
onload -p latency taskset -c 3 ./[Application]
onload --profile=latency taskset -c 2 ./[Application]

其中，-c 参数选择 CPU core 的号码，也可以选择多个core：-c 2,3。选择core的数量与进程的线程数有关。

使用 internal 模式加速应用

onload ./[Application]