Windows Debuggers: Part 1: A WinDbg Tutorial

 

Table of contents

• Introduction
  • Overview of Debuggers
  • Comparison of Debuggers
  • WinDbg
  • PDB files
• Debugging Scenarios
  • Remote Debugging
  • Just-in-time Debugging
  • 64-bit Debugging
  • Managed Debugging
  • Debugging Services
  • Debugging Exceptions
• WinDbg Features
  • Debugger Extension DLLs
  • Dump Files
  • Crash Dump Analysis
• WinDbg Settings
  • Symbol Files and Directories
  • Source Code Directories
  • Breakpoints, Tracing
• Commands
  • Basic Commands
  • More Commands
  • Handy Extension Commands
• Example
  • Suggested Exercises
• Epilogue
  • Points to Note
  • Q & A
• References

Introduction

In my professional career, I have seen most of us use Visual Studio for debugging but not many of the other debuggers that come for free. You may want such a debugger for many reasons, for example, on your home PC which you do not use for development but on which a certain program crashes from time to time. From the stack dump, you can figure out if IE crashed because of a third party plug-in.

I did not find any good quick starters for WinDbg. This article discusses WinDbg with examples. I assume you know the basic concepts of debugging – stepping in, stepping out, breakpoints and what it means to do remote debugging.

Note that this is meant to be a Getting Started document, which you can read and start using WinDbg. To know more about specific commands, consult the WinDbg documentation. You can use the commands presented in this document with any debugger provided by Microsoft, e.g. from the Command window of Visual Studio .NET.

This article is based on WinDbg 6.3.

This is the first of a series of articles on debugging. In my next article, I shall explain how to write debugger extension DLLs.

Overview of Debuggers

A brief overview of the Windows debuggers that you can download for free from here:

• KD – Kernel debugger. You want to use this to remote debug OS problems like blue screens. You want it if you develop device drivers.
• CDB – Command-line debugger. This is a console application.
• NTSD – NT debugger. This is a user-mode debugger that you can use to debug your user-mode applications. Effectively, this is Windows-style UI added to CDB.
• Windbg – wraps KD and NTSD with a decent UI. WinDbg can function both as a kernel-mode and user-mode debugger.
• Visual Studio, Visual Studio .NET – use the same debugging engine as KD and NTSD and offer richer UI than WinDbg for debugging purposes.

Comparison of Debuggers

Feature	KD	NTSD	WinDbg	Visual Studio .NET
Kernel-mode debugging	Y	N	Y	N
User-mode debugging		Y	Y	Y
Unmanaged debugging	Y	Y	Y	Y
Managed debugging		Y	Y	Y
Remote debugging	Y	Y	Y	Y
Attach to process	Y	Y	Y	Y
Detach from process in Win2K and XP	Y	Y	Y	Y
SQL debugging	N	N	N	Y

WinDbg

WinDbg is a debugger that wraps NTSD and KD with a better UI. It provides command-line options like starting minimized (-m), attach to a process by pid (-p) and auto-open crash files (-z). It supports three types of commands:

• regular commands (e.g.: k). The regular commands are to debug processes.
• dot commands (e.g.: .sympath). The dot commands are to control the debugger.
• extension commands (e.g.: !handle) – these are custom commands that you can add to WinDbg; they are implemented as exported functions in extension DLLs.

PDB files

PDB files are program database files generated by the linker. Private PDB files contain information about private and public symbols, source lines, types, locals and globals. Public PDB files do not contain types, local and source line information.

Debugging Scenarios

Remote Debugging

Doing remote debugging using WinDbg is easy and can be done in one of a number of ways. In the following, ‘debugging server’ is the debugger running on the machine where you’d like to debug; ‘debugging client’ is the debugger controlling the session.

• Using the debugger: You need CDB, NTSD or WinDbg on the server. A WinDbg client can connect to any of CDB, NTSD and WinDbg, and vice versa. The server and client have choices of TCP and named pipes for communication protocol.
  • To start a server:
    • WinDbg –server npipe:pipe=pipename (note: multiple clients can connect), or
    • from within WinDbg: .server npipe:pipe=pipename (note: single client can connect)

    You can start multiple server sessions using multiple protocols. You can password-protect a session.

  • To connect from a client:
    • WinDbg -remote npipe:server=Server, pipe=PipeName[,password=Password]
    • from within WinDbg: File->Connect to Remote Session: for connection string, enter npipe:server=Server, pipe=PipeName [,password=Password]
• Using remote.exe: remote.exe uses named pipes for communicating. If you use a console-based application like KD, CDB or NTSD, you could use remote.exe to do remote debugging. Note: use at q (not q) to quit the client without quitting the server dot
  • To start a server:
    • Remote.exe /s “cdb –p ” test1
  • To connect from a client:
    • Remote.exe /c test1

  test1 above is the arbitrary named pipe name we chose.

Server will display who all are connected from which servers and commands executed. You can quit the server by issuing ‘qq’; or quit the client using File->Exit. You’d need to belong to the “Debugger Users” user group and the server has to allow remote connectivity if you want to remote-debug.

Just-in-time Debugging

The section “Enabling Postmortem Debugging” in the WinDbg documentation discusses this well. In short, you can set WinDbg as the default JIT debugger by running Windbg –I. This sets the registry key HKLMSoftwareMicrosoftWindows NTCurrentVersionAeDebug to WinDbg. To set WinDbg as the default managed debugger, you’d need to set these registry keys explicitly:

• HKLMSoftwareMicrosoft.NETFrameworkDbgJITDebugLaunchSetting to 2
• HKLMSoftwareMicrosoft.NETFrameworkDbgManagedDebugger to Windbg.

With the JIT setting, WinDbg will be launched if an application throws an exception while not being debugged and does not handle the exception itself.

64-bit Debugging

All these debuggers support 64-bit debugging on AMD64 and IA64.

Managed Debugging

WinDbg 6.3+ supports managed debugging, with the Whidbey .NET CLR. There is a good discussion on managed debugging in the documentation. Remember that there are no PDBs with managed code since managed code is compiled to ILASM; the debugger talks to the CLR to query extra information.

Points to note:

You can set a breakpoint at a managed code function only after it has been invoked at least once; because that is when it is JIT-compiled to ASM code. Keep in mind:

• Complications with function addresses and hence breakpoints:
  • The CLR can discard compiled code, so function addresses may change.
  • The same code may be multiply compiled if multiple app domains do not share the code. If you set a breakpoint, it gets set for the app domain of the current thread.
  • Specialization of generics can cause multiple addresses for the same function.
• Complications with data layout and hence data inspection:
  • The CLR may change data layout arbitrarily at runtime, so field offsets in a structure may change over time.
  • Type information is loaded only on first use, so you may not be able to inspect a data field if it has not been used yet.
• Complications with debugger commands:
  • When tracing through managed code, you would pass through chunks of runtime code like the JIT compiler code because you stepped into a function for the first time, or, when transitioning from managed to unmanaged code.

Debugging Services

You can debug a service just as any other application using WinDbg, both after starting the service by attaching to the service process, and, by using WinDbg as a JIT debugger and programmatically calling DbgBreakPoint or DebugBreak, or an ASM int 3 on x86.

Debugging Exceptions

A debugger gets notified of each exception twice – it is notified the first time before the application gets a chance to handle the exception (‘first chance exception’); if the application does not handle the exception, the debugger is given a chance to handle the exception ( ‘second-chance exception’). If the debugger does not handle a second-chance exception, the application quits.

.lastevent, or, !analyze –v will show you the exception record and stack trace of the function where the exception occurred.

You can also use the .exr, .cxr and .ecxr commands to display the exception and context records. Note also that you can change the first-chance handling option for an exception using the sxe, sxd, sxn and sxi commands.

WinDbg Features

Debugger Extension DLLs

Debugger extensions are DLLs that you can hook up with a debugger to execute custom commands from within the debugger. There are certain functions that a DLL needs to implement and some requirements that a DLL needs to meet in order to qualify as an extension DLL. In the next article, we shall learn how to write an extension DLL yourself. The bang (!) commands are commands executed from your extension DLLs. Note that extension DLLs are loaded in the process space of the debugger.

Dump Files

You can take snapshot information of a process using the dump facility. A mini-dump is usually small, unless you take a full-memory minidump (.dump /mf). It is useful to dump handle information also, as .dump/mfh. A mini-dump contains information about all threads including their stacks and list of loaded modules. A full dump contains more information, like that of the process heap.

Crash Dump Analysis

If your Windows OS crashes, it dumps the physical memory contents and all process information to a dump file, configured through System->Control Panel->Advanced->’Startup and Recovery’. It is also possible to take dumps of any live process by breaking into it. You can also take a dump of any process (.dump) that terminates abnormally by configuring WinDbg as a JIT debugger. Note that figuring out bugs in the code from a crash dump could be an involved process.

To analyze a dump, follow these steps:

Step 1: In WinDbg, File->’Open Crash Dump’, and point to the dump file

Step 2: WinDbg will show you the instruction your app was executing when it crashed.

Step 3: Set your symbol path and source path properly. If you cannot match symbols, you could have a hard time figuring out control flow. If you can match the symbols to source code of the appropriate version, it should be easy to figure out the bug at this point. Note that private symbol files have line number information and will blindly show the line in your source code without further checks; if your source is not version-matched properly, you’d not see the correct source code matching the assembly code. If you have public PDB files, you’ll see the last public function (on the call stack) that was invoked.

Note that debugging drivers or managed code is much different. Refer to [2] for debugging techniques for device drivers.

WinDbg Settings

Symbol Files and Directories

You need symbols in order to be able to do effective debugging. Symbol files could be in an older COFF format or the PDB format. PDBs are program database files and contain public symbols. These debuggers allow you to mention a list of URIs where they would look for symbols for loaded binaries.

OS symbols are usually installed in the %SYSTEMDIR%Symbols directory. Driver symbols (.DBG or .PDB files) are usually in the same folder as the driver (.sys file). Private symbol files contain information about functions, local and global variables, and line information to correlate assembly code to source code; symbol files that are usually made available to customers are public symbol files – these files contain information about public members only.

You can set symbol directories through File->Symbol File Path, or using .sympath from the WinDbg command window. To add reference to a symbol server on the web, add:

SRV*downstream_store*

to your .sympath, thus:

.sympath+ SRV*c:tmp*

Where c:tmp is the download_store where necessary symbols will be downloaded and stored. Note that this particular symbol server exposes public symbols only.

The debugger matches information like filename, timestamp and checksum when matching a PDB with a binary (DLL or exe). If you have symbol information, you’d be able to see function names and their arguments in your call stack. If the binaries and PDBs are from your application, you’d additionally have information about private functions, local variables and type information.

The sympath can consist of multiple URIs. Sympath is initialized from the _NT_SYMBOL_PATH system environment variable.

Source Code Directories

You can set source code directories through File->Source File Path, or using .srcpath from the WinDbg command window. If you set source code directories, the debugger will pull up matching source code based on line number information from the PDB files during debugging.

Breakpoints, Tracing

• Set soft breakpoints using the bp commands or using the toolbar breakpoint icon.
• Set hard breakpoints using code like DbgBreakPoint() or KdBreakPoint().
• Use tracing routines DbgPrint, KdPrint, OutputDebugString to print out to the WinDbg output window, from debugger extension DLLs.

Commands

Basic Commands

The help file that comes with the WinDbg installation documents commands well, but the following basic commands should get you started:

Feature	Command	What Does it Do	Example / Comments	See Also Related Commands
Stack trace	K, KB x	Displays stack trace of current thread (x frames). Kb causes the display to include the first three parameters passed to each function.		KP, Kp, or KV
Frame	.frame X
Register watch	R	Displays register set. reax – displays the eax register.
Step	t	Trace = Step into (F11)
	p	Step over (F10)
	Step out	Shift + F11
Disassemble	u	Unassemble next few instructions
	u <start_address>	Unassemble instructions at start_address
	u <start_address> <end_address>	Unassemble instructions from start_address till end_address
Breakpoints	Bl	List breakpoints.
	be, bd, bc	Enable / disable / clear breakpoint.
	bp	Set a breakpoint.
	bu	Set unresolved breakpoint. Breakpoint is resolved by symbolic name, not absolute address. Use this to set breakpoint at a function whose containing module has not yet been loaded.	bu foo
Comment	*	Ignores the command	* Hello World
Continue	G <address_X / symbol>	Go. Resumes execution until address_X
	GH	Go, exception handled
	GN	Go, exception not handled
Quit	Q
Dumping data	dv	Display local variables.	You need private symbols.
	Dd <address>	Display dword values at specified address.	To see value of an int, DD <addr> L1
	Ds, da (ASCII), du (Unicode)	Dump string
	Dt [dt module!typedef adr]	Dump type. Will dump the contents of the memory using typedef as a template.
Change / Edit Values	Eb (byte), ed (dword), ea (ASCII), eu (Unicode)	Edit value of a variable
List modules	lm	List loaded modules		Lmi, lml, !dlls
Threads	~	Lists all threads
Command on thread n	~n<command>	Switch to a specific thread by thread-id and execute a command on the thread.	~2kb (second thread’s stack)
Search for a symbol in a module	X module!		X blah!*foo*
Dump	.dump
Source line display	.lines	Turns on source code display
	ln adr	Will show the symbol nearest to that location.

Note:

1. There is no “step out” (Shift+F11). You have to find the return address on the stack manually and use “g adr”. You can find this address by using “k”. If you know the function uses ebp frames you can use “g poi(ebp+4)” to step out.
2. To inspect local variables:
   1. Use the “dv” command.
   2. Then use the “dt <variablename>” command.
   3. Note: you may not see correct values if values are stored in registers or due to FPO.

More Commands

Feature	Command	What Does it Do	Example / Comments	See Also Related Commands
	Vertarget	Shows information about the system on which you are debugging.
Data breakpoint (hardware bp)	Ba [ba r/w/e size adr]	Sets a data breakpoint. You can break on read/ write/ execute attempt of a memory location.	ba w4 adr
Exceptions	.lastevent	Displays last exception record
Exceptions	Sx, Sxe, sxd, sxn, sxi exception_X	Enable/ disable/ notify-only/ ignore first chance exception /event exception_X. Example of event: module unload/ thread creation.
Display type	Dt	Shows struct and field values.	Dt x; // x: int Dt myStruct; // struct myStruct Dt myStruct myVar1; // shows myStruct.myVar1
Reload symbols	.reload	Reloads symbols using the symbol path you would have set.
Source lines	l+l, l+o, l+s, l+t	Source line options
	.ecxr	If you had an exception, switches context to faulting context.
	.quit_lock
	;	Command separator
	?	Evaluate expression
	|	Display process information
	.chain	Lists all loaded debugger extensions.
	.echo <string>	Echo/ print any string	Echo xyz
	.exr <address_x>	Display exception record at x.
	.cxr <address_x>	Display context record at x.
	.trap	Dump a trap frame.

Handy Extension Commands

• !help – help for WinDbg extension commands.
• !load, !unload – to load and unload debugger extension DLLs.
• !handle – displays information about handles owned by processes.
• !peb - shows the PEB (process environment block) including DLL information.

Example

Attached is a sample application with these example functions:

1. Example1: Program appears hung because a thread waits indefinitely on a critical section that another thread acquired and then exited without releasing.
2. Example2: Exception: division by zero.
3. Example3: Execute a command every time a breakpoint is hit.
4. Example4: Exception: null pointer access
5. Example5: Exception: double deletion
6. Example6: Exception: stack overflow due to infinite recursion

Suggested Exercises

1. Exception: Array out-of-bound access
2. Exception: Deleted pointer access
3. Exception: Stack underflow

Epilogue

Points to Note

Please note that:

• when you run WinDbg, attach to a process and issue kb, you’d be seeing the stack trace of the thread injected by the debugger. All debugging commands are executed in the context of the injected thread.
• Frame Pointer Omission (FPO):

  Means that when your code is compiled, frame pointers (EBP) will not be put on the stack. This makes function calls faster and makes the EBP register available as a scratch register. The optimization option /Oy in the MSC++ compiler => FPO; /O2 or /Ox (full optimization) => /Oy.

Q & A

1. How can I list all symbols exported by a module?

   x !*

2. How can I find help for a specific command?

   .hh , or /?

3. I want a certain application x.exe to run always under WinDbg. How can I configure this?

   Create a key named x.exe under “HKLMSoftwareMicrosoftWindows NTcurrentversionimage file execution options” and add a new string value “Debugger” to it; set its value to the path of windbg.exe.

4. I want to do something every time a breakpoint is hit. How can I do that?

   The bp command accepts a list of commands as argument that you can execute every time a breakpoint is hit. Example:

    

   bp WindbgEx1!Example3+0x3d "dd [ebp-0x14] L1; .echo hello world;g"

   (ref. attached code)

   prints the value of a local variable in each iteration of function Example3.

5. Can I put a breakpoint that is triggered only once?

   Yes:bp /1

6. Can I set a breakpoint such that it will start hitting only after k-1 passes?

   Yes, bp

   k

References

1. WinDbg documentation [from Microsoft]
2. “The Windows 2000 Device Driver Book” – Art Baker, Jerry Lozano

About Saikat Sen

I am a software developer based in Seattle, Washington. I used to work for Lucent in New York. Click here to view Saikat Sen's online profile.

Other popular articles: Debug Tutorial Part 3: The Heap Introduction to the heap. Debug Tutorial Part 1: Beginning Debugging Using CDB and NTSD Learn how to debug problems in software. Finding crash information using the MAP file Finding crash information using the MAP file: how to create and read the file An introduction to debugging in MSVC++ using Pseudoregisters Explanation of the debugger pseudoregisters like @ERR, @TIB