dpdk 内核模块 Unknown symbol in module 问题
@[dpdk 内核模块 Unknown symbol in module 问题]
问题1:
igb_uio.ko, rte_kni.ko 在编译的时候出现问题 undefined symbol：
- 编译时出现的故障现象：
cd ${RTE_SDK}
chmod a+x scripts/ tools/ -vRf
make -j 16 install T=${RTE_TARGET}
> == Build lib/librte_eal/linuxapp/igb_uio
> (cat /dev/null; echo kernel//dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko;) > /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/modules.order
> Building modules, stage 2.
> MODPOST 1 modules
> WARNING: "dev_notice" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> WARNING: "pci_enable_msix_range" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> WARNING: "pci_intx" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> WARNING: "pci_cfg_access_unlock" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> WARNING: "pci_cfg_access_lock" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> WARNING: "pci_disable_msix" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko] undefined!
> == Build lib/librte_eal/linuxapp/kni
> (cat /dev/null; echo kernel//dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko;) > /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/modules.order
> Building modules, stage 2.
> MODPOST 1 modules
> WARNING: "netdev_info" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "napi_disable" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "napi_schedule_prep" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dev_mc_add_excl" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dev_uc_add_excl" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "pci_disable_msix" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dql_reset" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_carrier_on" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_carrier_off" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__dev_kfree_skb_any" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_napi_del" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "kthread_bind" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__netdev_alloc_skb" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "csum_ipv6_magic" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netdev_set_tc_queue" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_schedule_queue" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_rx_ni" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_tx_wake_queue" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_tx_stop_all_queues" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "ethtool_op_get_link" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__cpu_online_mask" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dql_completed" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "nla_find" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "free_netdev" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "register_netdev" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dev_close" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_set_real_num_rx_queues" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_set_real_num_tx_queues" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netif_napi_add" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__sw_hweight64" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "dev_open" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "synchronize_irq" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "napi_gro_receive" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__napi_schedule" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "alloc_netdev_mqs" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "napi_complete_done" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "eth_type_trans" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netdev_err" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "ether_setup" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "msleep_interruptible" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "netdev_warn" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "eth_validate_addr" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__netlink_kernel_create" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "schedule_timeout_interruptible" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "skb_add_rx_frag" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "pci_get_device" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "unregister_netdev" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "ndo_dflt_bridge_getlink" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "pci_dev_put" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "vzalloc_node" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "pci_enable_device_mem" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "skb_tstamp_tx" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "hwmon_device_register" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "__skb_pad" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
> WARNING: "alloc_etherdev_mqs" [/dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko] undefined!
- 插入模块出现的故障现象：
insmod /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
> insmod: ERROR: could not insert module /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko: Unknown symbol in module
dmesg
> [ 2868.006572] igb_uio: Unknown symbol pci_cfg_access_unlock (err 0)
> [ 2868.006603] igb_uio: Unknown symbol pci_cfg_access_lock (err 0)
> [ 2868.006614] igb_uio: Unknown symbol pci_intx (err 0)
> [ 2868.006621] igb_uio: Unknown symbol pci_enable_msix_range (err 0)
> [ 2868.006632] igb_uio: Unknown symbol dev_notice (err 0)
> [ 2868.006655] igb_uio: Unknown symbol pci_disable_msix (err 0)
insmod /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko
> insmod: ERROR: could not insert module /dpdk/arm64-armv8a-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/rte_kni.ko: Unknown symbol in module
dmesg
> [ 3371.685528] rte_kni: Unknown symbol alloc_etherdev_mqs (err 0)
> [ 3371.685547] rte_kni: Unknown symbol __skb_pad (err 0)
> [ 3371.685561] rte_kni: Unknown symbol hwmon_device_register (err 0)
> [ 3371.685575] rte_kni: Unknown symbol skb_tstamp_tx (err 0)
> [ 3371.685583] rte_kni: Unknown symbol pci_enable_device_mem (err 0)
> [ 3371.685593] rte_kni: Unknown symbol vzalloc_node (err 0)
> [ 3371.685602] rte_kni: Unknown symbol pci_dev_put (err 0)
> [ 3371.685611] rte_kni: Unknown symbol ndo_dflt_bridge_getlink (err 0)
> [ 3371.685618] rte_kni: Unknown symbol unregister_netdev (err 0)
> [ 3371.685627] rte_kni: Unknown symbol pci_get_device (err 0)
> [ 3371.685635] rte_kni: Unknown symbol skb_add_rx_frag (err 0)
> [ 3371.685649] rte_kni: Unknown symbol schedule_timeout_interruptible (err 0)
> [ 3371.685657] rte_kni: Unknown symbol __netlink_kernel_create (err 0)
> [ 3371.685664] rte_kni: Unknown symbol eth_validate_addr (err 0)
> [ 3371.685672] rte_kni: Unknown symbol netdev_warn (err 0)
> [ 3371.685683] rte_kni: Unknown symbol msleep_interruptible (err 0)
> [ 3371.685689] rte_kni: Unknown symbol ether_setup (err 0)
> [ 3371.685696] rte_kni: Unknown symbol netdev_err (err 0)
> [ 3371.685708] rte_kni: Unknown symbol eth_type_trans (err 0)
> [ 3371.685715] rte_kni: Unknown symbol napi_complete_done (err 0)
> [ 3371.685722] rte_kni: Unknown symbol alloc_netdev_mqs (err 0)
> [ 3371.685732] rte_kni: Unknown symbol __napi_schedule (err 0)
> [ 3371.685745] rte_kni: Unknown symbol napi_gro_receive (err 0)
> [ 3371.685756] rte_kni: Unknown symbol synchronize_irq (err 0)
> [ 3371.685762] rte_kni: Unknown symbol dev_open (err 0)
> [ 3371.685769] rte_kni: Unknown symbol __sw_hweight64 (err 0)
> [ 3371.685784] rte_kni: Unknown symbol netif_napi_add (err 0)
> [ 3371.685791] rte_kni: Unknown symbol netif_set_real_num_tx_queues (err 0)
> [ 3371.685799] rte_kni: Unknown symbol netif_set_real_num_rx_queues (err 0)
> [ 3371.685805] rte_kni: Unknown symbol dev_close (err 0)
> [ 3371.685816] rte_kni: Unknown symbol register_netdev (err 0)
> [ 3371.685823] rte_kni: Unknown symbol free_netdev (err 0)
> [ 3371.685831] rte_kni: Unknown symbol nla_find (err 0)
> [ 3371.685837] rte_kni: Unknown symbol dql_completed (err 0)
> [ 3371.685844] rte_kni: Unknown symbol __cpu_online_mask (err 0)
> [ 3371.685853] rte_kni: Unknown symbol ethtool_op_get_link (err 0)
> [ 3371.685862] rte_kni: Unknown symbol netif_tx_stop_all_queues (err 0)
> [ 3371.685868] rte_kni: Unknown symbol netif_tx_wake_queue (err 0)
> [ 3371.685877] rte_kni: Unknown symbol netif_rx_ni (err 0)
> [ 3371.685887] rte_kni: Unknown symbol netif_schedule_queue (err 0)
> [ 3371.685894] rte_kni: Unknown symbol netdev_set_tc_queue (err 0)
> [ 3371.685904] rte_kni: Unknown symbol csum_ipv6_magic (err 0)
> [ 3371.685910] rte_kni: Unknown symbol __netdev_alloc_skb (err 0)
> [ 3371.685917] rte_kni: Unknown symbol kthread_bind (err 0)
> [ 3371.685926] rte_kni: Unknown symbol netif_napi_del (err 0)
> [ 3371.685939] rte_kni: Unknown symbol __dev_kfree_skb_any (err 0)
> [ 3371.685946] rte_kni: Unknown symbol netif_carrier_off (err 0)
> [ 3371.685955] rte_kni: Unknown symbol netif_carrier_on (err 0)
> [ 3371.685962] rte_kni: Unknown symbol dql_reset (err 0)
> [ 3371.685969] rte_kni: Unknown symbol pci_disable_msix (err 0)
> [ 3371.685976] rte_kni: Unknown symbol dev_uc_add_excl (err 0)
> [ 3371.685983] rte_kni: Unknown symbol dev_mc_add_excl (err 0)
> [ 3371.685991] rte_kni: Unknown symbol napi_schedule_prep (err 0)
> [ 3371.685999] rte_kni: Unknown symbol napi_disable (err 0)
> [ 3371.686014] rte_kni: Unknown symbol netdev_info (err 0)
原因
内核的 symbol 没有导出
cd /lib/modules/$(uname -r)/build/
grep pci_disable_msix System.map
> ffff0000083b46a8 T pci_disable_msix
grep pci_disable_msix Module.symvers
>
grep -i symbol /lib/modules/$(uname -r)/build/.config
> CONFIG_UNUSED_SYMBOLS=n
修正动作
将 /lib/modules/$(uname -r)/build/.config
的 CONFIG_UNUSED_SYMBOLS 设置为 y
grep -i symbol /lib/modules/$(uname -r)/build/.config
> CONFIG_UNUSED_SYMBOLS=y
验证结果
# 重编内核
cd /lib/modules/$(uname -r)/build/
make -j 16
make -j 16 modules
make -j 16 modules_install
make -j 16 install
# 检查 symbol 是否导出
grep pci_disable_msix System.map
> ffff0000083b5fa8 T pci_disable_msix
> ffff0000089334a8 r __ksymtab_pci_disable_msix
> ^^^^^^^^^^^^^^^^^^^^^^^^^^ export symbol
> ffff000008957049 r __kstrtab_pci_disable_msix
> ^^^^^^^^^^^^^^^^^^^^^^^^^^ export symbol
grep pci_disable_msix Module.symvers
> 0x00000000 pci_disable_msix vmlinux EXPORT_SYMBOL
> ^^^^^^^ module name where the symbol belongs
问题2
如果只是在插入的时候出现 Unknown symbol in module 问题。
原因
需要查看 模块间的依赖关系：
modinfo igb_uio.ko
> filename: /dpdk/arm64-armv8a-linuxapp-gcc/kmod/igb_uio.ko
> author: Intel Corporation
> license: GPL
> description: UIO driver for Intel IGB PCI cards
> depends: uio
> ^^^ igb_uio.ko is depends on uio.ko
> name: igb_uio
> vermagic: 4.14.76-ngfw.el7.aarch64 SMP mod_unload aarch64
> parm: intr_mode:igb_uio interrupt mode (default=msix):
> msix Use MSIX interrupt
> legacy Use Legacy interrupt
>
> (charp)
修正动作
先加载依赖的模块。
modprobe uio
insmod igb_uio.ko