MTK T750平台:CCCI驱动调试

MTK T750平台:CCCI驱动调试

  • 1. T750简介
  • 2. CCCI驱动调试
    • 2.1 调试环境
    • 2.2 Release Note
    • 2.3 make
    • 2.4 error: ‘struct dev_pm_info’ has no member named ‘driver_flags’
    • 2.5 error: inlining failed in call to always_inline
    • 2.6 error: ‘struct pci_dev’ has no member named ‘priv_flags’
    • 2.7 error: missing binary operator before token "("
    • 2.8 error: ‘struct net_device’ has no member named ‘max_mtu’
    • 2.9 error: ‘atomic_t {aka struct }’ has no member named ‘refs’
    • 2.10 Unknown symbol in module
  • 3. 验证
    • 3.1 加载成功
    • 3.2 端口、网卡枚举情况
    • 3.3 端口、网卡枚举情况 拨号测试
  • 4. 附件
  • 5. 总结

1. T750简介

MediaTek T750 是一款面向新一代5G CPE无线产品,可应用于5G固定无线接入(FWA)和移动热点(MiFi)等设备,为家庭、企业和移动用户带来高速5G连接,芯片平台采用 7nm 制程工艺,高度集成 5G NR FR1 调制解调器,4 核 Arm Cortex-A55 CPU 可提供完整的功能和配置,支持 5G NR Sub-6GHz 下双载波聚合(2CC CA)200MHz 频率,不仅拥有更大的信号覆盖范围,同时也让 5G 的下行速度大幅提升。

在独立组网(SA)中,T750 芯片平台最大下行峰值可达到 4.76Gbps ,上行峰值可达到 1.25Gbps ,可支持两个 2.5Gbps SGMII 接口、多种 LAN 端口配置、多种 Wi-Fi 配置包括单颗 5G 4×4 ,单颗 2.4G 4×4 和 5G 2×2 + 2.4G 2×2 双频 Wi-Fi 6 芯片,在保证网络高速连接的同时,也确保了 Wi-Fi 的稳定性。

2. CCCI驱动调试

2.1 调试环境

主控:NVIDIA XAVIER (ARM)
系统:Ubuntu 16.04
内核:Linux 4.9.0
Modem:MTK T750
驱动:CCCI (Cross Core Communication Interface)

2.2 Release Note

CCCI驱动Release_note:

The host CCCI driver provides standard linux device nodes, which can be find in directory /dev/. User can use this device nodes with linux standard API--open/close and read/write.
You can execute command "make" in driver top directory, then will get driver object files(*.ko) in folder /out, and the directory structure is shown as following.
(Note kernel version: 4.15.0-29-generic, gcc version: 5.4.0 20160609)

从Release_note可以看出MTK仅适配了4.15.0内核版本。为了更快的完成调试工作,我们需要快速的发现和解决问题。

2.3 make

拷贝CCCI驱动到NVIDIA XAVIER (ARM)后,解压直接执行make进行编译时报错:

make[1]: Entering directory '/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9'
  CC [M]  /home/sdk/Linux_PCIe_Driver_v1.0.11/./PCIE/core/mtk-pcie.o
gcc: error: unrecognized argument in option ‘-mcmodel=kernel’
gcc: note: valid arguments to ‘-mcmodel=’ are: large small tiny
gcc: error: unrecognized command line option ‘-mno-sse’; did you mean ‘-fno-dse’?
gcc: error: unrecognized command line option ‘-mno-mmx’
gcc: error: unrecognized command line option ‘-mno-sse2’; did you mean ‘-fno-dse’?
gcc: error: unrecognized command line option ‘-mno-3dnow’; did you mean ‘-fno-doc’?
gcc: error: unrecognized command line option ‘-m64’
gcc: error: unrecognized command line option ‘-mno-red-zone’; did you mean ‘-fno-regmove’?
scripts/Makefile.build:335: recipe for target '/home/sdk/Linux_PCIe_Driver_v1.0.11/./PCIE/core/mtk-pcie.o' failed

分析与解决

  1. 通过分析,我们发现在顶级目录Makefile中目标架构写成了固定的x86_64,这就导致在编译源码时,编译器无法寻找到合适的交叉编译链,导致编译失败。
COMPILER = "gcc"
LINKER = "ld"
TARGET_ARCH := x86_64
DATE = "BUILD_TIME="
LABEL = "LABEL="
  1. 针对该问题有如下解决方案:修改TARGET_ARCH 为 arm64,注意,这里只能写成arm64,非aarch64。
COMPILER = "gcc"
LINKER = "ld"
TARGET_ARCH := arm64
DATE = "BUILD_TIME="
LABEL = "LABEL="

2.4 error: ‘struct dev_pm_info’ has no member named ‘driver_flags’

通过上述方法的修改,可以开始编译驱动。出现了诸多“has no member named”错误:

./PCIE/core/mtk-pci.c:374:9: error: implicit declaration of function ‘dev_pm_set_driver_flags’; did you mean ‘devm_get_free_pages’? [-Werror=implicit-function-declaration]
./PCIE/core/mtk-pci.c:374:74: error: ‘DPM_FLAG_NEVER_SKIP’ undeclared (first use in this function); did you mean ‘PF_FREEZER_SKIP’?
./PCIE/core/mtk-pci.c:374:60: error:struct dev_pm_info’ has no member named ‘driver_flags’
./PCIE/core/mtk-pci.c:1391:3: error:const struct pci_error_handlers’ has no member named ‘reset_prepare’
./PCIE/core/mtk-pci.c:1391:19: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
./PCIE/core/mtk-pci.c:1392:3: error:const struct pci_error_handlers’ has no member named ‘reset_done’; did you mean ‘reset_notify’?
./PCIE/core/mtk-pci.c:1478:12: error:struct pci_dev’ has no member named ‘ltr_path’
…

由于NVIDIA XAVIER (ARM)环境的Linux内核版本是4.9.0,而驱动中申明支持4.15.0,怀疑为内核版本差异导致。对比4.9.0和4.15.0版本相关文件:

  1. struct dev_pm_info结构体:
    MTK T750平台:CCCI驱动调试_第1张图片
    上图是struct dev_pm_info结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。
static int mtk_pci_pm_init(struct mtk_pci_dev *mtk_dev)
{device_init_wakeup(&pdev->dev, true);
    #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
    if(!suspend_direct_complete)
        dev_pm_set_driver_flags(&pdev->dev, pdev->dev.power.driver_flags|DPM_FLAG_NEVER_SKIP);
    #endif}
  1. pci_error_handlers结构体
    MTK T750平台:CCCI驱动调试_第2张图片
    上图是pci_error_handlers结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以看到在4.15.0中是将4.0.9中的“void (*reset_notify)(struct pci_dev *dev, bool prepare);”函数拆分为两个“void (*reset_prepare)(struct pci_dev *dev);void (*reset_done)(struct pci_dev *dev);”,作用实质上是一致的。进一步阅读CCCI源码对这一块的处理:
static const struct pci_error_handlers mtk_pci_err_handler = {
	.error_detected = mtk_pci_error_detected,
	.slot_reset = mtk_pci_slot_reset,
	.reset_prepare = mtk_pci_reset_prepare,
	.reset_done = mtk_pci_reset_done,
	.mmio_enabled = mtk_pcie_mmio_enabled,
	.resume = mtk_pci_resume,
};

static void mtk_pci_reset_prepare(struct pci_dev *pdev)
{
	MTK_PCI_ERS_FUNC_ENTER();
}

#define MTK_PCI_ERS_FUNC_ENTER() \
	MTK_INFO_LOG("ERS", "%s Enter\n", __func__)

通过阅读CCCI代码,我们发现,实际处理仅仅是一个log的打印,因此毫无意义,无需再修改CCCI对这块的处理,直接进行版本的区分即可。

static const struct pci_error_handlers mtk_pci_err_handler = {
	.error_detected = mtk_pci_error_detected,
	.slot_reset = mtk_pci_slot_reset,
    #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	.reset_prepare = mtk_pci_reset_prepare,
	.reset_done = mtk_pci_reset_done,
    #endif
	.mmio_enabled = mtk_pcie_mmio_enabled,
	.resume = mtk_pci_resume,
};
  1. struct pci_dev
    MTK T750平台:CCCI驱动调试_第3张图片
    上图是struct pci_dev结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。
static void mtk_pci_bridge_configure(struct mtk_pci_dev *mtk_dev, struct pci_dev *dev)
{//re-configure bridge ltr to satisfy remove-rescan case
#ifdef CONFIG_PCIEASPM
    #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	if (bridge->ltr_path == 1) {
		pcie_capability_read_word(bridge, PCI_EXP_DEVCTL2, &cap);
		if(!(cap&PCI_EXP_DEVCTL2_LTR_EN))
			pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
	}
	MTK_INFO_LOG(TAG, "Bridge LTR_Path %x/%x cap %x flag %x\n", bridge->ltr_path, dev->ltr_path, cap, cap&PCI_EXP_DEVCTL2_LTR_EN);
    #endif
#endif
}

static void mtk_pci_configure_ltr(struct pci_dev *dev)
{
	//re-configure device ltr to satisfy remove-rescan case
#ifdef CONFIG_PCIEASPM
   #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	if (dev->ltr_path == 1) {
		pcie_capability_clear_word(dev, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_LTR_EN);
	}
   #endif
#endif
}

2.5 error: inlining failed in call to always_inline

通过修改上述问题,再次进行编译出现如下错误:

/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-pci.c: In function ‘mtk_pcie_interrupt_reinit’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/platform/M8X/75/mtk-pcie-mac.h:109:13: error: inlining failed in call to always_inline ‘mtk_pcie_intr_enable’: function body not available
 inline void mtk_pcie_intr_enable(PPCIE_MAC_IREG_REGS pbase, bool enable);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-pci.c:958:3: note: called from here
   mtk_pcie_intr_enable(mtk_dev->base_addr.pcie_mac_ireg_base, TRUE);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-isr.c: In function ‘mtk_pcie_msix_irq’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/platform/M8X/75/mtk-pcie-mac.h:132:21: error: inlining failed in call to always_inline ‘mtk_pcie_msix_status_get’: function body not available
 inline unsigned int mtk_pcie_msix_status_get(PPCIE_MAC_IREG_REGS pbase);
/home/sdk/Linux_PCIe_Driver_v1.0.26/./PCIE/core/mtk-isr.c:100:14: note: called from here
  msix_status = mtk_pcie_msix_status_get(mtk_dev->base_addr.pcie_mac_ireg_base);

开始我们以为inline函数的作用域只是当前C文件,而不能被别人C文件调用。但深入学习C语言的inline关键字http://hi.baidu.com/serial_story/blog/item/d9f4c2fa9abf11d4b58f31bf.html后,发现上面的inline,是可以被外部所引用的,也就是说,上面的代码,是可以正常编译的。因此我们尝试通过添加CFLAGS进行规避问题:

COMPILE_FLAGS := -g
COMPILE_FLAGS += -O3 \
                 -U_FORTIFY_SOURCE \
                 -fno-stack-protector
COMPILE_FLAGS += -Wall
COMPILE_FLAGS += $(INCLUDE_DIRS)
COMPILE_FLAGS += $(MICRO_DEF) 
COMPILE_FLAGS += -Wno-date-time \
                 -Wno-implicit-function-declaration \
                 -Wno-misleading-indentation \
                 -Wno-incompatible-pointer-types

-O3是恢复默认优化所必需的,这是避免许多故障所必需的;-U_FORTIFY_SOURCE需要避免gcc …/sysdeps/unix/sysv/linux/syslog.c失败(在函数’vsyslog_chk’中:[…]内联调用’syslog’失败);-fno-stack-protector需要避免gcc […] elf / dl-allobjs.os libc_pic.a((init-first.os).data 0x0)的失败:’ libc_multiple_libcs’的多重定义)。
通过上述修改,问题仍然存在,通过再次深入阅读代码,发现inline在此可直接省去,非必要,直接删除问题解决。

2.6 error: ‘struct pci_dev’ has no member named ‘priv_flags’

通过修改上述问题,再次进行编译出现如下错误:

/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c: In function ‘t_pcie_lp_show_lp_info’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c:562:31: error:struct pci_dev’ has no member named ‘priv_flags’; did you mean ‘dev_flags’?
   test_bit(0, &mtk_dev->pdev->priv_flags));

对比4.9.0和4.15.0版本相关文件:
MTK T750平台:CCCI驱动调试_第4张图片
上图是struct pci_dev结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。

int t_pcie_lp_show_lp_info(struct mtk_pci_dev *mtk_dev, int argc, char **argv)
{MTK_INFO_LOG(TAG, "runtime usage_count=%d\n",
		atomic_read(&(mtk_dev->pdev->dev.power.usage_count)));
    #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	MTK_INFO_LOG(TAG, "dev disconnected=%d\n",
		test_bit(0, &mtk_dev->pdev->priv_flags));
    #endif}

2.7 error: missing binary operator before token “(”

通过修改上述问题,再次进行编译出现如下错误:

/home/sdk/Linux_PCIe_Driver_v1.0.26/./pcie_drv/mtk-pcie-drv-test.c:561:48: error: missing binary operator before token "("
         #if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))

分析发现是未识别LINUX_VERSION_CODE和KERNEL_VERSION宏,为确实头文件导致。

#include 

2.8 error: ‘struct net_device’ has no member named ‘max_mtu’

通过修改上述问题,再次进行编译出现如下错误:

/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c: In function ‘ccmni_dev_init’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:695:5: error:struct net_device’ has no member named ‘max_mtu’ dev->max_mtu = CCMNI_MTU_MAX;
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:728:5: error:struct net_device’ has no member named ‘needs_free_netdev’ dev->needs_free_netdev = true;
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.c:731:7: error:struct net_device’ has no member named ‘priv_destructor’; did you mean ‘destructor’?dev->priv_destructor = NULL;
scripts/Makefile.build:335: recipe for target '/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/ccmni/ccmni_m80.o' failed

对比4.9.0和4.15.0版本相关文件:
MTK T750平台:CCCI驱动调试_第5张图片
上图是struct net_device结构体部分截图,左侧为4.9.0,右侧为4.15.0,可以很明显的看到4.15.0中新增了几个成员用于更好的系统性能,而在4.9.0中是没有的,两者间存在差异,由于是新增功能,未能在4.9.0中找到替代,所以修改CCCI驱动源码,将该结构体相关新增功能进行版本的区分。另外4.15.0中修改destructor函数名为priv_destructor,按照内核版本区分即可。

static inline void ccmni_dev_init(int md_id,struct ccmni_ctl_block *ctlb, struct net_device *dev)
{#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	dev->max_mtu = CCMNI_MTU_MAX;
	#endif#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	/* use kernel default free_netdev function */
	dev->needs_free_netdev = true;
	/* don't need to free again, because last step has called free_netdev */
	dev->priv_destructor = NULL;
	#else
	dev->destructor = NULL;
	#endif}

2.9 error: ‘atomic_t {aka struct }’ has no member named ‘refs’

通过修改上述问题,再次进行编译出现如下错误:

/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/eccci/port/port_char.c: In function ‘port_char_uninit’:
/home/sdk/Linux_PCIe_Driver_v1.0.26/./ext_drv/ccci/eccci/port/port_char.c:111:50: error:atomic_t {aka struct <anonymous>}’ has no member named ‘refs’
       atomic_read(&port->cdev->kobj.kref.refcount.refs));

对比4.9.0和4.15.0版本相关文件:
MTK T750平台:CCCI驱动调试_第6张图片
上图是struct kref结构体截图,左侧为4.9.0,右侧为4.15.0,可以看到4.15.0中使用的refcount_t结构体作为自己的成员,再进一步使用atomic_t结构体,而在4.9.0中是直接将atomic_t结构体作为自己的成员,两者间存在差异,所以修改CCCI驱动源码,将该结构体相关按照内核版本区分即可。

static void port_char_uninit(struct port_t *port)
{#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
			MTK_DEBUG_LOG(CHAR, "%s,kobject cnt %d\n",port->name,
						atomic_read(&port->cdev->kobj.kref.refcount.refs));
			#else
			MTK_DEBUG_LOG(CHAR, "%s,kobject cnt %d\n",port->name,
						atomic_read(&port->cdev->kobj.kref.refcount));
			#endif}

经过以上的修改,所有由于内核差异导致的编译问题全部解决,可编译成功,进一步加载驱动进行测试。

2.10 Unknown symbol in module

通过对内核版本的对比和适配,完成了驱动在客户环境上的编译,现进行加载验证:

root@:/home/sdk/Linux_PCIe_Driver_v1.0.26/out/script# ./setup.sh 
insmod: ERROR: could not insert module mtk_pcie_wwan_m80.ko: Unknown symbol in module
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
cat: '/sys/kernel/mtk_wwan_*_pcie/build_time': No such file or directory
cat: '/sys/kernel/mtk_wwan_*_pcie/label': No such file or directory

加载时报如下错误,通过dmesg log我们可以看到是timer_setup函数存在问题:

[24380.191354] mtk_pcie_wwan_m80: Unknown symbol timer_setup (err 0)

进一步对比4.9.0和4.15.0内核之间timer的差异:
MTK T750平台:CCCI驱动调试_第7张图片
上图是timer.h部分截图,左侧为4.9.0,右侧为4.15.0,可以看到4.15.0中修改了timer中函数名称和实现,两者间存在差异,所以修改CCCI驱动源码,将该结构体相关按照内核版本区分即可。

	#if(LINUX_VERSION_CODE > KERNEL_VERSION(4,15,0))
	//timer_setup(&dpmaif_ctrl->traffic_monitor,dpmaif_traffic_monitor_func, 0);
	#else
	__setup_timer(&dpmaif_ctrl->traffic_monitor,
		dpmaif_traffic_monitor_func,&dpmaif_ctrl->traffic_monitor, 0);
	#endif

3. 验证

3.1 加载成功

通过上述修改,再次编译进行加载,成功。
在这里插入图片描述

3.2 端口、网卡枚举情况

查看设备是否正常枚举端口,并发送AT进行测试,均ok:
在这里插入图片描述
在这里插入图片描述
查看网卡枚举是否正常,结果ok:

会枚举出20个网卡,这个可由加载脚本nic_interface_num参数控制:

#!/bin/sh

# install pcie driver and enable all pcie logs
insmod mtk_pcie_wwan_*.ko dynamic_log_level=6 nic_interface_num=21
# set max log level
echo 7 > /proc/sys/kernel/printk

# set system max network buffer
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216

cat /sys/kernel/mtk_wwan_*_pcie/build_time
cat /sys/kernel/mtk_wwan_*_pcie/label

3.3 端口、网卡枚举情况 拨号测试

MTK T750平台:CCCI驱动调试_第8张图片
拨号成功后,配置IP、路由,并进行ping包测试:
在这里插入图片描述
MTK T750平台:CCCI驱动调试_第9张图片
在这里插入图片描述
Ping ip和域名均ok,功能正常,调试完成。

4. 附件

在安装PCIE驱动时,可以用shell命令dmesg –w查看PCIE驱动安装过程日志,在日志的末尾看到有“md_state change from 3 to 4”和“Create ccmni netdev successfully”字样,则说明PCIE驱动的安装状态正常,并且ccmni网络设备成功生成。

[13098.660843] [MTK_PCIE_WWAN][CCCI_FSM][fsm_append_command][958]:command 3 is appended 0 from fsm_main_thread [mtk_pcie_wwan_m80]
[13098.660851] [MTK_PCIE_WWAN][CCCI_FSM][fsm_finish_command][981]:command 2 is completed 1 by fsm_main_thread [mtk_pcie_wwan_m80]
[13098.660852] [MTK_PCIE_WWAN][HIF_CLDMA][cldma_gpd_tx_collect][743]:txq:0,16,21c,80011001
[13098.660855] [MTK_PCIE_WWAN][CCCI_FSM][fsm_main_thread][881]:command 3 process
[13098.660856] [MTK_PCIE_WWAN][CCCI_FSM][fsm_broadcast_state][232]:md_state change from 3 to 4
[13098.661250] [MTK_PCIE_WWAN][DATA][ccmni_netdev_create][904]:ccmni_netdev_create--nic_num: 21
[13098.663100] [MTK_PCIE_WWAN][DATA][ccmni_md_state_callback][1001]:Create ccmni netdev successfully
[13098.663122] [MTK_PCIE_WWAN][CCCI_FSM][fsm_finish_command][981]:command 3 is completed 1 by fsm_main_thread [mtk_pcie_wwan_m80]
[13098.641689] [MTK_PCIE_WWAN][PCIE][mtk_pci_bridge_configure][1405]:Parent [Bus:0 Devfn:e8 Vendor:8086 Device:9d18]

5. 总结

驱动在新环境的适配,除了架构的差异,更多的是内核上的差异,所以我们需要去按照编译报错信息对比内核相关文件,按照差异去找是否有替换方案,如果没有,为新增内容,最好的方式便是用内核版本进行控制,便于管理和维护。

你可能感兴趣的:(高通\展锐\MTK等平台调试,linux)