Xilinx zynq万兆网使用与优化
方法
使用vivado2015.2.1和petalinux2015.2.1，采用axi-10g-ethernet IP核，这个IP核感觉现在xilinx已经不在维护了。
bug
这个版本的linux内核感觉有bug，napi_schedule后并没有调用napi的poll函数，导致无法收包，排除了FPGA的程序问题，看来需要升级内核了。两个万兆网不能同时up，否则第二个网口不能工作，这肯定是内核有问题。
static irqreturn_t axienet_rx_irq(int irq, void *_ndev)
{
u32 cr;
unsigned int status;
struct net_device *ndev = _ndev;
struct axienet_local *lp = netdev_priv(ndev);
status = axienet_dma_in32(lp, XAXIDMA_RX_SR_OFFSET);
if (status & (XAXIDMA_IRQ_IOC_MASK | XAXIDMA_IRQ_DELAY_MASK)) {
cr = axienet_dma_in32(lp, XAXIDMA_RX_CR_OFFSET);
cr &= ~(XAXIDMA_IRQ_IOC_MASK | XAXIDMA_IRQ_DELAY_MASK);
axienet_dma_out32(lp, XAXIDMA_RX_CR_OFFSET, cr);/*delay here*/
napi_schedule(&lp->napi);
}
...
}
该问题偶然间定位了，
/* Enable NAPI scheduling before enabling Axi DMA Rx IRQ, or you
* might run into a race condition; the RX ISR disables IRQ processing
* before scheduling the NAPI function to complete the processing.
* If NAPI scheduling is (still) disabled at that time, no more RX IRQs
* will be processed as only the NAPI function re-enables them!
*/
napi_enable(&lp->napi);
/* Enable interrupts for Axi DMA Tx */
ret = request_irq(lp->tx_irq, axienet_tx_irq, 0, ndev->name, ndev);
if (ret)
goto err_tx_irq;
/* Enable interrupts for Axi DMA Rx */
ret = request_irq(lp->rx_irq, axienet_rx_irq, 0, ndev->name, ndev);
if (ret)
goto err_rx_irq;
测试
线程绑核之后的速度是91MB/s，未绑核的速度是77MB/s，zynq上PS端的千兆网我测试过是88MB/s，已经达到了CPU的处理极限，加窗等优化也是没有意义的。所以在zynq上实现万兆网只是解决了有无问题，可用性不行，下一步可以考虑用FPGA加速TCP/IP协议栈。
root@zynq:~# iperf3 -c 192.168.10.6 -t 1000
Connecting to host 192.168.10.6, port 5201
[ 4] local 192.168.10.8 port 36747 connected to 192.168.10.6 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.01 sec 92.3 MBytes 768 Mbits/sec 0 185 KBytes
[ 4] 1.01-2.01 sec 91.2 MBytes 766 Mbits/sec 0 185 KBytes
[ 4] 2.01-3.00 sec 91.2 MBytes 769 Mbits/sec 0 185 KBytes
[ 4] 3.00-4.01 sec 92.5 MBytes 769 Mbits/sec 0 185 KBytes
[ 4] 4.01-5.01 sec 91.2 MBytes 768 Mbits/sec 0 185 KBytes
[ 4] 5.01-6.00 sec 91.2 MBytes 769 Mbits/sec 0 185 KBytes
[ 4] 6.00-7.00 sec 91.2 MBytes 766 Mbits/sec 0 185 KBytes
[ 4] 7.00-8.00 sec 91.2 MBytes 768 Mbits/sec 0 185 KBytes
[ 4] 8.00-9.01 sec 92.5 MBytes 769 Mbits/sec 0 185 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-20.97 sec 1.88 GBytes 768 Mbits/sec 0 sender
[ 4] 0.00-20.97 sec 0.00 Bytes 0.00 bits/sec receiver
root@zynq:~# iperf3 -c 192.168.10.6 -t 1000
Connecting to host 192.168.10.6, port 5201
[ 4] local 192.168.10.8 port 36749 connected to 192.168.10.6 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 76.8 MBytes 643 Mbits/sec 0 188 KBytes
[ 4] 1.00-2.00 sec 77.5 MBytes 648 Mbits/sec 0 188 KBytes
[ 4] 2.00-3.01 sec 77.5 MBytes 647 Mbits/sec 0 188 KBytes
[ 4] 3.01-4.02 sec 77.5 MBytes 646 Mbits/sec 0 188 KBytes
[ 4] 4.02-5.01 sec 76.2 MBytes 645 Mbits/sec 0 188 KBytes
[ 4] 5.01-6.02 sec 77.3 MBytes 639 Mbits/sec 0 212 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-6.54 sec 502 MBytes 644 Mbits/sec 0 sender
[ 4] 0.00-6.54 sec 0.00 Bytes 0.00 bits/sec receiver
我们在zynq上实现了两个10g以太网，测试结果，两个网卡的发送接收中断都位于CPU0，这样一个iperf 33MB/s，第二路iperf 68MB/s，两个CPU利用率均为45%，中断和应用绑定之后，两路均为52MB左右，两个CPU利用率均为49%。