目录
Sometimes it's hard to capture a kernel panic: You don't know how to reproduce the kernel panic and/or once you have rebooted your system, there are no clues in the logfiles. This is where netconsole might help out. Netconsole is a kernel module that logs kernel printk messages over UDP allowing debugging of problems where disk logging fails. This is just a techno-way of saying that the kernel messages will get send over the network in certain packets (UDP packets).
There are a couple of disadvantages though:
This document will guide you through the steps to setup netconsole for Ubuntu {*}.
There are 5 steps to set it all up! Good luck!
We need to know a mac address where the UDP packets will get sent to (also known as the "receiver"). This "receiver" can or can't be in the same subnet:
In this example I would like to send the UDP packets to 192.168.1.103 (which is in the same subnet as the sender is).
pet@sender:~$ ping -c 1 192.168.1.103 > /dev/null pet@sender:~$ arp -n 192.168.1.103 Address HWtype HWaddress Flags Mask Iface 192.168.1.103 ether 08:00:46:d4:1d:82 C eth0
In this example 08:00:46:d4:1d:82 is the mac address we need.
Okay, when that's the case, you need to determine the default gateway first:
pet@sender:~$ netstat -rn | grep ^0.0.0.0 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
In this case, that's 192.168.1.1.
Now, we need to figure out which mac address our default gateway has:
pet@sender:~$ ping -c 1 192.168.1.1 > /dev/null pet@sender:~$ arp -n 192.168.1.1 Address HWtype HWaddress Flags Mask Iface 192.168.1.1 ether 00:0f:66:5b:2a:3c C eth0
In this example 00:0f:66:5b:2a:3c is the mac address we need.
Okay, let's assume you use grub as your bootloader. In this case grub will boot the kernel with (at least) the "quiet splash" options by default. We don't want that.
Just to be sure, create a backup of /boot/grub/menu.lst first:
pet@sender:~$ sudo cp /boot/grub/menu.lst /root/menu.lst.backup
Now, open your favorite editor to edit /boot/grub/menu.lst.
sudo vi /boot/grub/menu.lst
gksudo gedit /boot/grub/menu.lst
Locate the line that starts with "# defoptions=quiet splash" (don't get mislead by the fact the line starts with a "#") and replace this line with "# defoptions=debug ignore_loglevel".
Tell grub to update accordingly:
pet@sender:~$ sudo update-grub
Just to be sure, create a backup of /etc/default/grub first:
pet@sender:~$ sudo cp /etc/default/grub /etc/default/grub.backup
Now, open your favorite editor to edit /etc/default/grub.
sudo vi /etc/default/grub
gksudo gedit /etc/default/grub
Locate the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" and replace this line withGRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel".
Tell grub to update accordingly:
pet@sender:~$ sudo update-grub
Okay, that went well. Three more steps to go!
Now we need to know the:
ifconfig -a
ip address and mac address of the "receiver" (these were already gathered in #Step 1: Determine remote mac address)
In the following example, the part that starts with "netconsole=" is setup like this:
netconsole=<LOCAL_PORT>@<SENDER_IP_ADDRESS>/<SENDER_INTERFACE>,<REMOTE_PORT>@<RECEIVER_IP_ADDRESS>/<STEP_1_MAC_ADDRESS>
In this example, I use 6666 for both <LOCAL_PORT> and <REMOTE_PORT>.
$ sudo sh -c 'echo netconsole >> /etc/modules' $ sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf'
There are several ways to retrieve the netconsole packets:
Easiest way probably is to boot the receiver with an Ubuntu LiveCD. Setup networking and gain shell access.
pet@receiver:~$ sudo apt-get install syslog-ng
Edit /etc/syslog-ng/syslog-ng.conf with your favorite editor and append these lines at the very end of the file:
source net { udp(ip("0.0.0.0") port(6666)); }; destination netconsole { file("/var/log/$HOST-netconsole.log"); }; log { source(net); destination(netconsole); };
Now restart syslog-ng:
pet@receiver:~$ sudo /etc/init.d/syslog-ng restart
Continue with #Step 5: Check if it works. Once you are done, the logging will be in /var/log/<SENDER_IP_ADDRESS>-netconsole.log.
If you have the luxury of having a Linux machine as "receiver", you just might want to try this:
In this example 192.168.1.103 is the same as "<RECEIVER_IP_ADDRESS>" (see #Step 3: Initialize netconsole at boot time)
pet@receiver:~$ netcat -l -u 192.168.1.103 6666 | tee ~/netconsole.log
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in ~/netconsole.log.
If this doesn't work, also try without the IP address specifically listed:
pet@receiver:~$ netcat -l -u 6666 | tee ~/netconsole.log
Some versions of Linux, such as Ubuntu Hardy (8.04), use a different version of Netcat that uses a different syntax. If you use a Linux receiver that has a different version of Netcat, it will print the following error message when you try the commands above:
UDP listen needs -p arg
In this case, you need to issue the following command:
pet@receiver:~$ netcat -l -p 6666 -u | tee ~/netconsole.log
Unsurprisingly, as OS-X is a BSD UNIX derivative, Netcat is available out of the box on this platform with the minor difference from Linux that the command is actually called nc rather than netcat.
In this example 192.168.1.103 is the same as "<RECEIVER_IP_ADDRESS>" (see #Step 3: Initialize netconsole at boot time)
receiver:~ pet$ nc -l -u 192.168.1.103 6666 | tee ~/netconsole.log
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in ~/netconsole.log.
If this doesn't work, also try without the IP address specifically listed:
receiver:~ pet$ nc -l -u 6666 | tee ~/netconsole.log
There's a netcat for Windows available here or here.
Unpack it somewhere (i.e. C:\Users\Pet\Desktop\nc\)
Open a dosprompt (Start -> Run -> cmd). In this example 192.168.1.103 is the same as <RECEIVER_IP_ADDRESS>.
cd C:\Users\Pet\Desktop\nc\ nc -u -l -p 6666 192.168.1.103 > netconsole.txt
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in netconsole.txt.
There's an open source syslog tool here. In the version currently available (2009, Jan), there is no way to change the port syslog listens on. You'll have to change the <REMOTE_PORT> to port 514 (which is the default syslog port).
Reboot the "sender". Once rebooted execute the command below.
$ dmesg | grep netcon [ 21.048406] netconsole: local port 6666 [ 21.048410] netconsole: local IP 192.168.1.102 [ 21.048411] netconsole: interface eth0 [ 21.048413] netconsole: remote port 6666 [ 21.048415] netconsole: remote IP 192.168.1.103 [ 21.048418] netconsole: remote ethernet address 08:00:46:d4:1d:82 [ 21.048421] netconsole: device eth0 not up yet, forcing it [ 22.908106] console [netcon0] enabled [ 22.911536] netconsole: network logging started
Seems to be working! Now check your "receiver" to see whether the kernel messages were received! If that's the case, continue to use the "sender" until it crashes, hopefully the "receiver" receives some useful information about the crash.
You don't actually need to reboot the sender if you don't want to. You can manually load netconsole via:
$ sudo modprobe netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82
One way to test whether logging is working correctly or not is to insert some removable media like a CD, DVD or usb stick.
Once you retrieved the information you needed, we need to clean things up again.
pet@sender:~$ sudo mv /root/menu.lst.backup /boot/grub/menu.lst pet@sender:~$ sudo update-grub
pet@sender:~$ sudo mv /etc/default/grub.backup /etc/default/grub pet@sender:~$ sudo update-grub
pet@sender:~$ sudo sed -i '/^netconsole/d' /etc/modules pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf
Do you have comments on the content above? Something incomplete or incorrect? Please don't hesitate to drop me a few lines.
I would like to say thanks to Peter Veerman and Leann Ogasawara for reviewing this document.
A big thanks to Bruno Girin for adding the grub-2 instructions
https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Problems%20in%20capturing%20information
Another netconsole tutorial
Netconsole documentation
Netconsole tutorial for SuSe
KiBi's netconsole tutorial