How to use ADPlus.vbs to troubleshoot "hangs" and "crashes"
http://support.microsoft.com/default.aspx?scid=kb;en-us;286350
Linux / aix:
http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=%2Fcom.ibm.java.doc.igaa%2F_1vg0001475cb4a-1190e2e0f74-8000_1006.html
1. decide which thread takes the high cpu. - ps -o thcount -p pid/ ps -mLF -p pid -> get the cpu time and pid for that thread
2. find out what's that thread doing - get the thread stack.
gdb - pid
input -> all thread apply bt
the bt command will ouput the stack, which would tell you what's the thread doing
misc:
(gdb) info thread
(gdb) info proc
(gdb) info proc all
ll /proc/32193/task/ -> get the thread ids for process 32193
cat /proc/32193/maps ->get process memory map
1. decide which native thread is taking high cpu, by launching 'top -H -p pid' / or using 'ps -mLF -p pid'
2. convert the native thread to hex
3. Dump the java vm to produce the javacore.txt
4. Associate by the native thread, and look at what that thread is doing
1. Use 'Perfmon' tool to add counters of threads for this process(thread id, processor time)
2. Export the logs to .csv format
3. Analyze the generated logs. Find out which thread is taking high cpu & what's its according Thread ID(virtual ID and system overal thread id)
4. Use tool - Debug Diagnostic tool to catch a dump file.
5. Use Debug Diagnostic tool to analyze all threads' tasks they were doing. The output html file will give each stack and the according thread id
e.g.
Entry point | mozcrt19!endthreadex+a0 |
Create time | 11-12-23 22:35:09 |
Time spent in user mode | 0 Days 00:00:00.00 |
Time spent in kernel mode | 0 Days 00:00:00.00 |
Function | Source |
---|---|
ntdll!KiFastSystemCallRet | |
ntdll!NtWaitForSingleObject+c | |
kernel32!WaitForSingleObjectEx+a8 | |
kernel32!WaitForSingleObject+12 | |
nspr4!PR_Wait+ed | |
xul!mozilla::layers::ThebesLayer::GetType+23ff | |
xul!gfxPattern::gfxPattern+1d96 | |
nspr4!PR_WaitCondVar+1cd | |
mozcrt19!endthreadex+106 |
The same methodology applies to Java process
1. Use perfmon tool to find out which thread is taking high cpu and find out its native thread ID
2. Get a javacore.txt file to associate with the native thread ID. See the stack to detect what is it doing - root of the high cpu
1. Use Debug Diagnostic Tool to monitor the leaking process for a period
2. Use debug diagnostic tool to open and analyze the user.dmp. The output html file will give the summary about the memory / handle leaks. As well, it will tell you which module and even which function did that.
, as well as illegal memory operation, error releasing mem, uninitialized memory usage -- focus on Heap, but not stack )
1. Use valgrind tool to start the debugged process, with correct parameters.
e.g. valgrind --trace-children=yes --tool=memcheck --log-file=/tmp/val.log --leak-check=full ./processCmd
2. Would print the exact code line if we compile the source code with -g
3. Focus on malloc/free, new/delete, new []/delete[]; focus on Heap memory.
misc : valgrind is included in some linux distro .. such as RHEL.
The link to quick reading valgrind :-- http://www.cprogramming.com/debugging/valgrind.html