FreeNas 11.1 U6 - re0:watchdog timeout 问题解决
csdn脱库以来就没登录过了，今天找回密码下东东，顺便发个博客过来赚赚积分吧。
家里一直在用FreeNas做存储，从FreeNas 8开始搭的，现在升到11.1 U6。
一直以来就有一个问题，网络高负载时，控制台会报 re0:watchdog timeout ，然后断网。
正常读写文件都没问题，当从某数据集读蓝光盘的同时，再往另一数据集写入加了音轨的ts时，这个错误100%出现。
中文环境下FreeNas可参考的资料太少了。本人年纪也大了，很久没大段地啃英文的技术文档，用源码编译驱动也有十多年没做过了。
这次算折腾的比较久，记录下来，方便后来人吧。
一、先说说环境：
1、软件
# uname -a
FreeBSD xx-FreeNas.local 11.1-STABLE FreeBSD 11.1-STABLE #0 r321665+9902d126c39(freenas/11.1-stable): Tue Aug 21 12:24:37 EDT 2018
[email protected]:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
2、硬件
# dmesg | grep -C 8 E-450
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-STABLE #0 r321665+9902d126c39(freenas/11.1-stable): Tue Aug 21 12:24:37 EDT 2018
[email protected]:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn)
CPU: AMD E-450 APU with Radeon(tm) HD Graphics (1650.02-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x500f20 Family=0x14 Model=0x2 Stepping=0
Features=0x178bfbff
Features2=0x802209
AMD Features=0x2e400800
AMD Features2=0x35ff
SVM: (disabled in BIOS) NP,NRIP,NAsids=8
TSC: P-state invariant, performance statistics
real memory = 9126805504 (8704 MB)
3、网卡
# dmesg | grep re0
re0:
re0: Using 1 MSI-X message
re0: Chip rev. 0x48000000
re0: MAC rev. 0x00000000
4、ifconfig
# ifconfig
re0: flags=8843
options=8209b
ether 30:85:a9:ec:dd:ed
hwaddr 30:85:a9:ec:dd:ed
inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9
media: Ethernet autoselect (1000baseT
status: active
5、pciconf
# pciconf -lv | grep -C 4 re0
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Family 12h/14h Processor Function 7'
class = bridge
subclass = HOST-PCI
re0@pci0:3:0:0: class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet
二、问题分析
这是一块华硕 e45m1-m pro主板，集成CPU，集成大螃蟹8111网卡，问题就出在这个集成网卡上。
我没有去翻FreeNas的支持硬件列表，我想一定没有这款网卡，网上的大多建议是，禁用集成网卡，买一块Intel网卡装上，反馈是100%解决了问题。不想继续折腾的同学，看到这儿就可以去刷某宝了。二手的82580，i340都不错。
也有建议自行编译RealTek 8111驱动，“据说”可以解决问题，但应者寥寥，我也好多年没搞过编译驱动源码了，虽然也头大，但还是打算试试，与往机器里塞新设备这种事比起来，还是塞新驱动更容易接受些。
这些集成在主板上的网络芯片是廉价的，在我看来，螃蟹公司没花过多精力写好所有平台的默认驱动，在FreeBSD系统中，某些特定的网络环境下，这个驱动的bug就触发了。
三、解决过程
既然是驱动问题，就去找驱动，无奈没有编译好的驱动，在Realtek官网，下载了驱动的源代码，决定自行编译。
下载回来的驱动有3个文件，if_re.c 、 if_rereg.h 、 Makefile
执行下面一串命令，运气好的话，就编译成功了。
cp /tmp/rtl_bsd_drv_v195.00/if_re* /usr/src/sys/dev/re/
cp /tmp/rtl_bsd_drv_v195.00/Makefile /usr/src/sys/modules/re/
cd /usr/src/sys/modules/re/
make
但是，做为基于FreeBSD开发的FreeNas，是没有FreeBSD源码，没有编译环境，上面的几句话，处处碰壁。要一一解决：
1、FreeNas默认是没有/usr/src/下面的子目录，即使自己手工创建，make的时候还会报下面的错误：
make: "/usr/share/mk/bsd.kmod.mk" line 12: Unable to locate the kernel source tree. Set SYSDIR to override.
所以要把FreeBSD的源码给拽回来，大约不到2G的数据。拽回来之前先
cd /usr/src/
rm -rf *
拽源码命令如下：
git clone --single-branch -b releng/11.1 https://github.com/freebsd/freebsd /usr/src
下载之后，
# vi /usr/src/sys/i386/conf/GENERIC dd掉 re 那一行
# vi /usr/src/sys/modules/Makefile dd掉 re 那一行
虽然我觉得应该没用，但ReadME里有写，我也照做了。
2、这时，再make:
# make
sh: cc: not found
make: "/usr/src/share/mk/bsd.compiler.mk" line 145: Unable to determine compiler type for CC=cc. Consider setting COMPILER_TYPE.
vi 2个新文件 /etc/src.conf 和 /etc/make.conf ，记得以前是用gcc编译，后来查到，FreeBSD 10.2 以后开始使用 clang 做为新的标准编译器，于是新建文件加入内容如下：
WITH_CLANG_EXTRAS=YES
CC=/usr/local/flang/bin/clang
CXX=/usr/local/flang/bin/clang++
CPP=/usr/local/flang/bin/clang-cpp
3、加好之后，make自然是报错的：
# make
sh: /usr/local/flang/bin/clang: not found
Warning: Object directory not changed from original /usr/src/sys/modules/re
/usr/local/flang/bin/clang -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I. -I/usr/src/sys -fno-common
-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.if_re.o -MTif_re.o -mcmodel=kernel -mno-red-zone -mno-mmx -m
no-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-exter
ns -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -W
missing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-er
ror-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -mno-aes -mno-avx -std=iso9899:1999 -c /usr/src/sys/d
ev/re/if_re.c -o if_re.o
/bin/sh: /usr/local/flang/bin/clang: not found
*** Error code 127
Stop.
make: stopped in /usr/src/sys/modules/re
4、安装clang,直接装不了
#pkg install flang-clang
Updating local repository catalogue...
pkg: file:///usr/ports/packages/meta.txz: No such file or directory
repository local has no meta file, using default settings
pkg: file:///usr/ports/packages/packagesite.txz: No such file or directory
Unable to update repository local
Error updating repositories!
原因在于FreeNas的关于本地缓存的配置与FreeBSD不同，FreeNas是关闭的，FreeBSD是打开的。
需要这样做：
把/usr/local/etc/pkg/repos/local.conf中的yes改no
把/usr/local/etc/pkg/repos/FreeBSD.conf中的no改yes
这时，就能安装成功了。一路Yes，顺便把pkg升了个级。共计下载200多M数据，安装好后占用1G多点空间。
# pkg install flang-clang
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 6 MiB 3.4MB/s 00:02
Processing entries: 100%
FreeBSD repository update completed. 33377 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
pkg: 1.10.1 -> 1.10.5_5
Number of packages to be upgraded: 1
The operation will free 22 MiB.
3 MiB to be downloaded.
Proceed with this action? [y/N]: y
[1/1] Fetching pkg-1.10.5_5.txz: 100% 3 MiB 3.1MB/s 00:01
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.10.1 to 1.10.5_5...
Extracting pkg-1.10.5_5: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
flang-clang: 6.0.g20180904_2 [FreeBSD]
llvm60: 6.0.1_3 [FreeBSD]
Number of packages to be installed: 2
The process will require 1 GiB more space.
247 MiB to be downloaded.
Proceed with this action? [y/N]: y
[1/2] Fetching flang-clang-6.0.g20180904_2.txz: 100% 89 MiB 2.5MB/s 00:37
[2/2] Fetching llvm60-6.0.1_3.txz: 100% 158 MiB 2.5MB/s 01:06
Checking integrity... done (0 conflicting)
[1/2] Installing llvm60-6.0.1_3...
[1/2] Extracting llvm60-6.0.1_3: 100%
[2/2] Installing flang-clang-6.0.g20180904_2...
[2/2] Extracting flang-clang-6.0.g20180904_2: 100%
5、这回再make，还有错：
# make
Warning: Object directory not changed from original /usr/src/sys/modules/re
/usr/local/flang/bin/clang -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I. -I/usr/src/sys -fno-common
-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.if_re.o -MTif_re.o -mcmodel=kernel -mno-red-zone -mno-mmx -m
no-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-exter
ns -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_
kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-
body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error
-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 -c /usr/src/sys/dev/re/if_re.c -o if_re.o
ld -d -warn-common -r -d -o if_re.ko if_re.o
make: exec(ld) failed (No such file or directory)
*** Error code 1
Stop.
make: stopped in /usr/src/sys/modules/re
这次缺ld，安装ld，20M数据，占用195M空间：
# pkg install binutils
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
binutils: 2.30_5,1 [FreeBSD]
Number of packages to be installed: 1
The process will require 195 MiB more space.
20 MiB to be downloaded.
Proceed with this action? [y/N]: y
[1/1] Fetching binutils-2.30_5,1.txz: 100% 20 MiB 2.1MB/s 00:10
Checking integrity... done (0 conflicting)
[1/1] Installing binutils-2.30_5,1...
[1/1] Extracting binutils-2.30_5,1: 100%
6、最终，终于编译好了，if_re.ko就是了：
# make
Warning: Object directory not changed from original /usr/src/sys/modules/re
ld -d -warn-common -r -d -o if_re.ko if_re.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk if_re.ko export_syms | xargs -J% objcopy % if_re.ko
objcopy --strip-debug if_re.ko
7、给驱动搬个家：
cp /usr/src/sys/modules/re/if_re.ko /boot/kernel/
在FreeNAS GUI 界面 系统/高级/启用autotune。添加一条微调记录 变量： if_re_load, 值： YES, 类型： loader, 可用： 打勾，注释： Realtek driver
这时你的 /boot/loader.conf.local 就会是这样的 : if_re_load="YES" # Realtek driver
8、重启之后，确认一下驱动是否正确调用：
# grep re /boot/loader.conf.local
if_re_load="YES" # Realtek driver
# ls -l /boot/kernel/if_re.ko
-rw-r--r-- 1 root wheel 508856 Nov 1 08:58 /boot/kernel/if_re.ko
# kldstat|grep _re
2 1 0xffffffff82661000 7bf06 if_re.ko
# dmesg | grep re0
re0:
re0: Using Memory Mapping!
re0: Using 1 MSI-X message
re0: version:1.95.00
re0: Ethernet address: 30:85:a9:ec:dd:ed
re0: Ethernet address: 30:85:a9:ec:dd:ed
re0: link state changed to UP
对比编译之前的结果，看到新的驱动re0: version:1.95.00已经生效。
最后放上Realtek的驱动源码和我编译好的ko文件，环境差不多的同学可以直接用ko（不确定100%能用哈），也可以自己用源码编译。
https://download.csdn.net/download/xinew4712/10916374