Causes and First Aid of JVM Crash Issues?

 

Java Virtual Machine is a Native engine which allows our Java Applications to run. It performs the code optimization to improve the performance. In correct tuning, Low memory allocation, extensive code optimization, bad garbage collection strategy, API code leaking…etc are some of the reasons which may cause the JVM to crash.

.

Analyzing a JVM Crash is one of the very interesting and little time taking process sometimes it is even little complex to find out the root cause of the JVM Crash. Here in this article we will see some of the common mistakes, first aid solutions/debugging techniques to find out what kind of informations we can get by looking into the Core Dump.

 

 

 

======================================================================

 

What is Core Dump & Where to Find It?

 

Code dump is usually a Binary file which gets generated by the Operating System when JVM or any other process crashes. Sometimes it also happens that the JVM will not be able to generate the Crash dump. In Windows Operating Systems it will be generated in the Directore where the “Dr. Watson” tool is installed. In Windows it will be usually:  ”C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson“

.

By default in Unix based Operating Systems the Core Dump files are created in the directory where the Java Program/Server was started even sometimes it is generated in the “/tmp” directory of the Operating System. But using the following Java Option we can change it’s the Crash Dump/Heap   Dump generation locations:  -XX:HeapDumpPath=/opt/app/someLocaton/ and  -XX:+HeapDump JVM Options.

.

NOTE: These Flags does not gurantee that always the Heap/Crash dump will be generated at the time of JVM Crash. There are some more reasons behind the Core Dump not gets generated…like Process Limitations or the Less Disk Quota or unavailability of the Free File Descriptors.

 

 

=======================================================================

Who Generates the Crash/Core Dump?

 

JVM does not generate the Coe dump. Rather it is the Operating System which generates the Core Dump. Core Dump is a Binary file which may be several hundred Mega Bytes or Giga Bytes in size. The Operating systems just log the exception/error messages and the details of the Threads along with the Native libraries loaded with that java process.

.

Many times a brief Textual Crash file is also generated by the JVM itself sometimes during Crash. Usually the file name is “hs_err_pid<WebLogicPID>.log” in case of Sun JDK. Similarly JRockit JVM also generates a Textual file with name “*.dump” in case of JVM Crash.

 

 

=======================================================================

Use of -XX:+ShowMessageBoxOnError?

 

The Thread Dump is also very helpful to analyze the Server Crash. Thread dump tells us what was the status and activities performed by the Threads at the time of crash. It may be possible to get a thread dump before the process exits. HotSpot supports the JAVA_OPTIONS -XX:+ShowMessageBoxOnError

.

The corresponding JRockit JVM Option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user ” Do you want to debug the problem? ” This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

.

Along with above there are various options available to get the Thread Dumps as described in : http://middlewaremagic.com/weblogic/?p=823

 

 

=======================================================================

What May Cause JVM Crash?

 

 

Reason-1). Usually Native Code causes the JVM Crash. Native code is a code written in Languages like C/C++, Java Native Interface APIs (JNI).

.

Reason-2). JDBC Drivers specially the Native Drivers.

.

Reason-3). JVM Code Optimization.

.

Reason-4). Less Memory availabity for Native Area of a Java Process.

.

Reason-5). Application Servers Native Performance Pack Libraries.

.

Reason-6). JVMs library itself can cause the Crash.

.

Reason-7). High CPU Utilization by the Threads. As described in : http://middlewaremagic.com/weblogic/?p=4348

.

Reason-8). Presence of Wrong Native Libraries in the PATH or in the “-Djava.library.path“

.

Reason-9). Presence of A Different Version of Libraries in “java.library.path” or in “SHLIB” or in “LD_LIBRARY_PATH” variables. Like setting a 64-bit version of Library in a 32-bit version of JVM’s library path or vise-versa.

 

 

=======================================================================

 

Tools To Analyze the Core Dumps?

 

 

Core/Crash dump is Operating System specific, So to analyze these Dumps we must use the Tools provided by the same Operating System vendors. Various kind of tools are provided by the Operating Systems to analyze these Core Dumps like:

Tool-1). Dr. Watson Tool  in Windows OS. Windows OS Start (Button)—>Run…—>drwtsn32

.

Tool-2). “pstack” and “pmap” in Solaris Operating System.

.

Tool-3). “procstack” and “procmap” in AIX Operating System.

.

Tool-4). “lsstack” and “pmap” in Linux Operating System.

 

Tool-5). “pflag” if available in HP-UX Operating System.

 

=======================================================================

 

What May Help To Avoid JVM Crash?

 

 

It totally depends on What Caused the Crash or What Libraries caused the JVM crash to avoid the occurance of the JVM crash for next time. But following things should be taken in consideration while analyzing and avoiding the Crash.

.

Point-1). If the Native Jdbc Driver is causing issues, If our appliation is using the Native JDBC Drivers then Switching from Pure Native Jdbc Driver (Type-2 Drivers) to the Pure Java JDBC Driver (Type-4 Driver) may help.

.

Point-2). If the JVM Libraries are causing the Crash then Upgrading to a Later Version of the JDK. If that Application Server has that new JDK in it;s Supported Configuration list.

.

Point-3). If the JVM Code Optimization is causing the Crash, then Disabling the Code Optimization of JVM by applying the JVM Options.

For JRockit JVM Code Optimization can be disabled using JAVA_OPTION  -Xnoopt

For Sun JDK Code Optimization can be disabled using   JAVA_OPTION  -Xint

.

Point-4). Some times the “Just In Time Compiler” code generation also causes the JVM Crash. In these scenarios In Case of Sun JDKs disabling the JIT Compiler can help. We can disable the JIT Compiler by adding the JVM Option:  ”-Djava.compiler=none“

.

Point-5). Presence of a different bit version (32 bit or 64-bit libraries) of Library in the “-Djava.library.path“

.

Point-6). Disabling the Application Servers Native Performance Packs. In WebLogic The Native IO Can be disabled using “-Dweblogic.NativeIOEnabled=false” JVM Option.

 

.