Troubleshoots Java SE 6 Deployment

from:http://java.sun.com/developer/technicalArticles/javase/troubleshoot/

ss=surivor space（幸存者空间）

minor garbage collection : Eden space's data copy to SS,and gc the Eden space.

full gargabe collection: the data put to SS or old generation,than the space is over,than do the full gc.

in the space if copy 16 times ,than will copy to old generation.

Chuk-Munn Lee of Sun Microsystems Troubleshoots Java SE 6 Deployment

 

Articles Index

This article is adapted from a talk that Sun Microsystems Java technology evangelist Chuk-Munn Lee presented at Sun Tech Days in Sydney, Australia, in March 2008. Based in Singapore, Lee works frequently with individual developers and software vendors, helping them to architect and prototype both their server and desktop-based Java applications. His more recent work has focused on Swing-based client applications. He also keeps ISVs up-to-date on the latest developments in the Java platform and what's on the horizon.

 

The talk explored ways to troubleshoot running Java applications, with a focus on Java SE 6 .

Contents

• Troubleshooting and the Java SE 6 Platform
• An Overview of Memory Management
• Tools: jps , jinfo , and jstat
• HPROF
• The jhat Object Query Language
• Visual Tools: JConsole and Java VisualVM
• Common Problems
• Causes of Memory Problems
• Determining Memory-Retention Problems
• Get a Copy of the Heap to Monitor
• Finding Object Retention
• Finalizers
• Summary
• For More Information

Troubleshooting and the Java SE 6 Platform

Lee first defined troubleshooting as "locating the source of the problem and engaging in a postmortem analysis of what caused it." He pointed to many troubleshooting improvements in Java SE 6 that Sun developer Mandy Chung described in her blog.

With JDK 6, said Lee:

• Developers are no longer required to start applications with special options attached by JDK 6 tools. The Attach API enables users to build their own tools to attach to a running Java Virtual Machine (JVM)* and load a Java or native agent.
• Memory problems are easier to diagnose. The Java HotSpot VM enables developers to request a heap dump on demand from the jmap tool. A heap analysis tool, jhat , was added in JDK 6 to browse the heap dump snapshot.
• It's easier to diagnose an OutOfMemoryError thanks to a stack trace to where the allocation failed. The new -XX:+HeapDumpOnOutOfMemoryError option tells the HotSpot VM to generate a heap dump when an allocation from the Java heap or the permanent generation cannot be satisfied. In addition, a new -XX:-OnOutOfMemoryError=<command> option has been added, allowing developers to specify a command that the HotSpot VM will invoke when the OutOfMemoryError is thrown.
• The JDK 6 HotSpot VM provides built-in DTrace probes, enabling developers to trace the complete stack of any running Java application on the Solaris 10 OS.
• In addition, the Java SE Troubleshooting Guide has been updated to include troubleshooting information for JDK 6.

An Overview of Memory Management

Lee first summarized garbage collection. The garbage collector (GC) detects garbage , defined as objects that are no longer reachable, then reclaims it and makes space available to the running program. The GC typically works in a stop-the-world fashion -- that is, it freezes the heap when working. It has various algorithms, like copying, mark-sweep, mark-compact, and others.

Lee then pointed to a common mistake: Garbage collection is not always the cause of an application's slowness, and adding more memory will not always improve its performance. "Giving it more memory may actually make the system slower if memory is not an issue," he observed. "The GCs in the Java HotSpot VM are built around the idea that objects die young. This is empirical data, and some applications may not conform to this. But by and large, most Java applications do. So the HotSpot VM is optimized for this scenario."

Lee advised developers to favor short-lived objects that are used briefly and then discarded, instead of long-lived objects that are repeatedly updated. Long-lived older objects should be managed as little as possible and will be moved to the old generation by the GC.

The Java HotSpot VM keeps old and young objects in separate spaces, with the goal of making the allocate-manage-deallocate cycle as fast and efficient as possible. Developers can exploit different GC algorithms, based on their hardware, to better manage the objects in these spaces. With J2SE 5.0, Sun introduced ergonomics into the HotSpot VM. JVM ergonomics enables developers to specify desired behaviors, for example, that the VM's GC pauses last no longer than 750 milliseconds. The GC will then try to dynamically tune its behavior to meet the stated specification.

Figure 1 shows how the latest version of the JDK enables developers to specify different algorithms on different spaces with the HotSpot VM heap layout, which is broken up into three areas.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 1. HotSpot VM Heap Layout

 

<!-- END IMAGE WITH CAPTION -->

"The perm generation is basically for class loading," explained Lee. "Next are the old and young generation. The young generation is further broken up into three spaces: Eden, Survivor Space 1 (SS#1) and Survivor Space 2 (SS#2). How are they used? I'll give a simplistic explanation. When you have a new object, the object gets created in Eden space. So after running for a while, Eden space will fill up."

Lee pointed out that a minor garbage collection occurs, in which all the objects alive in Eden are copied over to SS#1. Eden is then empty and ready to receive new objects. After the minor GC, objects are allocated to Eden again. After a time, the Eden space fills up again, and another minor GC occurs. The objects surviving in SS#1 and Eden are copied to SS#2, and both SS#1 and Eden are reset. Although objects are frequently recopied, either from Eden or from one SS to another, at any one time, only Eden and one SS are operating.

Every time an object moves from Eden to SS or from one SS to another, a counter and its header is incremented. By default, if the copying occurs 16 times or more, the HotSpot VM stops copying them and moves them to the old generation.

If an object can't be created in Eden, it goes directly to the old generation. Moving an object from SS to the old generation because of its age is called tenuring. Because of tenuring, the old generation becomes full over time. This calls for garbage collection of the old generation, which is called a full GC. A full GC is a compaction process that is slower than a minor GC.

<!-- BEGIN TABLE -->

Table 1. Tools for Diagnosis of Common Problems

 

Problem

Symptom

Diagnostic Tools

<!-- FINAL LINE IS OF A DIFFERENT COLOR -->

Insufficient memory	OutOfMemoryError	Java Heap Analysis Tool (jhat )
Memory leaks	Growing use of memory Frequent garbage collection	Java Monitoring and Management Console (JConsole ) JVM Statistical Monitoring Tool (jstat )
	A class with a high growth rate A class with an unexpected number of instances	Memory Map (jmap ) See jmap -histo option
	An object is being referenced unintentionally	JConsole or jmap with jhat See jmap -dump option
Finalizers	Objects are pending for finalization	JConsole jmap -dump with jhat
Deadlocks	Threads block on object monitor or java.util.concurrent locks	JConsole Stack Trace (jstack )
Looping threads	Thread CPU time is continuously increasing	JConsole with JTop
High lock contention	Thread with high contention statistics	JConsole

 

<!-- END TABLE -->

Tools: jps , jinfo , and jstat

A variety of tools enable developers to look at running Java applications.

The Java Virtual Machine Process Status Tool (jps ) , the Java equivalent of the Unix ps command, lists the running VMs, including embedded ones. It then gives them a process number, which is the name of the application or class, and digs down to differing levels of detail with command lines. It is started by the browser, not explicitly by the developer. "jps is typically the entry point to most diagnostics -- you need to find out your process number first," said Lee.

jps -s gives slightly more information. jps -l gives both the class name and command line that was run. See Figure 2.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 2. jps

 

<!-- END IMAGE WITH CAPTION -->

The jinfo tool extracts configuration information from the VM or core file. It can only read core files collected by jinfo running on the same operating system instance. Other options include looking at the file separator or getting information on the VM flags that are set with the core file. See Figure 3.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 3. jinfo

 

<!-- END IMAGE WITH CAPTION -->

The Java Virtual Machine Statistics Monitoring Tool (jstat ) displays potentially detailed performance statistics for the JVM with two basic options. With the general option, instead of listing numbers, jstat provides one line with the current status. Output options determine the content and format of jstat 's output.

The -gcutil output option, which provides a summary of GC statistics, is among the most commonly used. Figure 4 shows SS#0 and SS#1, Survivor Space 0 and Survivor Space 1, Eden and the percentage that is full. "It provides young GC and young GCT times and full GC, focal GCT time spans," said Lee.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 4. jstat

 

<!-- END IMAGE WITH CAPTION -->

The -gccause output option displays the same summary of garbage collection statistics as the -gcutil option, but it includes the causes of the last GC event and, where applicable, the current GC event. It adds a column that identifies why the GC has happened. When it shows allocation failed , the heap is too small.

The jstack tool provides the stack traces of all the threads attached to a VM, such as application threads and interval VM threads, as shown in Figure 5. It also performs deadlock detection and will perform a stack trace if the VM is hung. "It will perform deadlock detection with -l ," explained Lee, "but this provides only a hint when assessing whether many threads are waiting on an object. If your VM has hung, you can force a stack trace out of it by doing a -F ."

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 5. jstack

 

<!-- END IMAGE WITH CAPTION -->

Lee turned to Figure 6, which provides an example of a Java source worker thread from a Linux machine. "You can see that it is timed waiting and locked. What does that mean? Lock means it is locked on an external process or resource. For instance, it may be waiting for a port to become free. But from the jstack output, we do not know this. You may have to use an external tool like DTrace on Solaris to correlate this. Wait means it is internally waiting for a monitor. In this case, it is waiting on something from JDK 5 called ReentrantLock ."

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 6. jstack -- Sample Output

 

<!-- END IMAGE WITH CAPTION -->

The jmap tool, as shown in Figure 7, prints shared object memory maps or heap memory details of a given process, core file, or remote debug server . It offers an inclusive, detailed memory configuration and information on free space capacity.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 7. jmap -- Sample Output

 

<!-- END IMAGE WITH CAPTION -->

"Figure 7 tells us we are using the mark and sweep collector, how much free space we have, or how much of the heap is used," said Lee. "When we have a certain amount of the heap, it begins requesting memory from the OS. Typically, for large applications, we set this so the VM doesn't do a lot of extra work. Then we have the Eden space, the current capacity, used and free space data, and so on."

HPROF

The Heap and CPU Profiling Agent (HPROF ), a heap-CPU profiling tool that collects information on CPU usage, heap dumps, and thread states, uses the JVM Tool Interface (JVMTI) , so every JVM has a diagnostic interface.

"To start HPROF, go to -Xrunhprof Java , provide options, and run Java," explained Lee. "By default, HPROF will only dump out information after you exit the application. The information can be in text or binaries. For binaries, use -Xrunhprof:format=b , and then specify the file name with file=<filename> . By default, the file name is java.hprof . There are two ways to collect an HPROF dump: You can force it by typing Ctrl-\ on Windows or by sending a SIGHUP on Solaris and other Unixen. The other method is to wait for the Java application to end and HPROF will write out the dump. The latter is the default behavior. While jmap and HPROF collect the same information that jhat analyzes, jmap is much faster than HPROF because jmap is built into the HotSpot VM."

Lee warned that if developers are working with a big heap, HPROF can take a long time to dump something out. Once developers have collected the information using jhat or HPROF, the information is mounted as follows:

<!-- BEGIN VCD7 CODE SAMPLE COMPONENT -->

jhat dump: format=b,file=heap_dump_file

 

<!-- END VCD7 CODE SAMPLE COMPONENT -->

Once jhat starts a web server internally and parses the information, it can be browsed through a standard browser. See Figure 8. A set of predefined queries shows all the classes, objects, and instances -- all the objects reachable from a root set. "Remember," cautioned Lee, "when you are tracking down memory, look at instances. Do not look at the classes." Lee explained that the first time he used the tool, he looked at classes and could not locate the source of his problem.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 8. jhat

 

<!-- END IMAGE WITH CAPTION -->

The jhat tool initially provides a set of standard queries to click on for information. To run a nonstandard query -- for example, to look for all objects A, B, C, and get all current references from another object -- developers can create a custom query through a hot button on an HTML page.

The jhat Object Query Language

The jhat Object Query Language (OQL) is SQL-like and similar to the Java Database Connectivity (JDBC) object-oriented OQL, with built-in functions such as heap, referrers, reachables, sizeof , and others. It can structure queries such as these: Find all String instances that are greater than 1K in size, or find all URL instances that are referenced by two or more objects.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 9. jhat Object Query Language

 

<!-- END IMAGE WITH CAPTION -->

"Be careful," warned Lee, "because if you have a big heap, you must run this on a fast machine! If you have a really complex query on a really big heap, it can take a very long time."

Visual Tools: JConsole and Java VisualVM

The Java Monitoring and Management Console (JConsole ) , a visual tool that is bundled with the JDK, offers a graphical console that enables developers to monitor and manage Java applications. The JConsolePlugin API lets developers create their own plug-ins. JConsole provides information on memory usage and GC activities, threads, thread stack traces, locks, and objects pending finalization. It also provides runtime information such as uptime and CPU time, as well as JVM information such as classpath, properties, command-line arguments, and so on.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 10. JConsole

 

<!-- END IMAGE WITH CAPTION -->

Java VisualVM relies on tools such as jstat , jinfo , jstack , and jmap to obtain detailed information about applications running inside a JVM. It then presents the data in a unified, graphically rich manner. Java VisualVM helps Java application developers to troubleshoot applications and to monitor and improve the applications' performance. Java VisualVM can allow developers to generate and analyze heap dumps, track down memory leaks, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. Plug-ins also exist that expand the functionality of Java VisualVM. For example, most of the functionality of the JConsole tool is available through the MBeans Tab and JConsole Plug-in Wrapper plug-ins .

In July 2008, Sun announced that it had bundled Java VisualVM with JDK 6 update 7 so that the Java VisualVM can be executed by invoking the jvisualvm command under the JDK's main executable directory. Java VisualVM is bundled together with the latest FCS version of JDK 6 update 7 as jvisualvm . See Figure 11.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 11. Visual Tool -- Java VisualVM

 

<!-- END IMAGE WITH CAPTION -->

"We've known for a long time that people have used the NetBeans Profiler to identify memory problems," observed Lee. "So essentially, now it's a stand-alone profiler."

Java VisualVM, according to Lee, looks better than JConsole and goes well with a plug-in architecture, such as the NetBeans IDE , enabling developers to create and download a plug-in and maneuver it around with the NetBeans window. Lee underscored the point that as applications grow in sophistication, it's important for developers to have their own tools, because standard tools will only take them so far.

Common Problems

Lacking enough heap space to accommodate new objects results in the java heap space error. This can happen when there is insufficient memory to run an application. A more common cause might be memory retention by the application of objects that have outlived their usefulness but for some reason cannot be freed by the GC. See Figure 12.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 12. Insufficient Memory

 

<!-- END IMAGE WITH CAPTION -->

A common nonheap error, the PermGen space error, occurs when the JVM runs out of space in the permanent generation heap. Because permanent generation and interned strings are stored in the permanent generation heap, when it is full, it cannot load classes. "Large JavaServer Pages (JSP) applications can cause this problem because JSP is compiled into classes, and you're using many classes from many libraries," said Lee. "But generally, it's not a problem. Developers writing Java Native Interface (JNI) code are prone to see native memory errors, which doesn't necessarily mean a memory leak has occurred. But it does mean that the system doesn't have enough memory. It may just mean that you have sized the heap incorrectly. So don't jump to conclusions and assume that you have a memory leak. Try a bigger-size heap, or look at the consumption in a graph. Memory error messages are simply indications of what may be wrong."

Causes of Memory Problems

Event listeners can cause memory leaks, particularly when developers add them to the pattern and forget about them until they cause leaks. See Figure 13. "Values in maps means we have a key and a value and lose reference to the key," said Lee. "So the value it points to gets retained in a map. Use rich hash map, which tells you when your key is no longer referenceable -- then the value and function will be removed."

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 13. Possible Causes of Memory Problems

 

<!-- END IMAGE WITH CAPTION -->

At times, various resources, such as graphics, JFrame , socket connection, and result sets are not freed. Because finalizers from legacy code may cause the GC to run slowly as the GC finds and runs them, finalizers should be put somewhere else. "Memory pressure causes the GC to run the finalizers -- it takes at least two GC cycles to clear objects with finalizers that aren't guaranteed to run in any particular order," said Lee. He pointed out that there is in fact no guarantee that they will run at all, and he advised against using finalizers. He insisted that if there are resources to be cleared, developers should use explicit methods to clear the resources before nulling the object.

Common deadlocks include threads waiting for resources not yet freed and high lock contention, which means that a lot of thread is accepting a particular locked object. "Synchronized code is slower, so change it to ReentrantLock , and it will be faster than synchronized code," Lee explained.

High lock contention causes numerous blocked and waiting threads, which may not mean the application has frozen. The key point is that excessively synchronized resources in a heavily threaded environment can lead to unresponsiveness.

Determining Memory-Retention Problems

How can you best collect information and analyze the situation? An application that is running out of memory may not have a memory-retention problem. It may mean that there is not enough memory. Lee advised developers to run JConsole visually if the heap is growing. See Figure 14.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 14. Determining Memory Retention

 

<!-- END IMAGE WITH CAPTION -->

"To run troubleshooting tools remotely, you need to set up your remote server to accept JMX connections," remarked Lee. "Generally, you want to run tools through accessing the local system. The tool produces a nice graph but no data that can be used to size the memory. Size the memory with printGC details, capture that to a local file, and run the application. It's usually best to write a shell script or a PERL script and filter out the minor GCs and the full GCs. Then pull up information from the before and after GC column, and then you do an average on them and add perhaps 20 percent to 30 percent to the memory."

Lee advised developers to use jmap -histo to get a histogram of all the objects in use and then look for suspiciously large allocations for objects, as seen in Figure 15.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 15. Get a Copy of the Heap -- 1

 

<!-- END IMAGE WITH CAPTION -->

Get a Copy of the Heap to Monitor

Once developers determine that they have a problem, Lee recommended that they get a copy of the heap and monitor it, either when the app is running or when it dies, as long as the application is in a steady state and no longer loading or initializing. "If you are using JConsole ," said Lee, "you go to the MBeans tab as you see in Figure 16. This will download the heap in the directory that starts JConsole . Alternatively, if the jps is on command line, go to jmap -dump:format=b and then give the process ID and file name."

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 16. Get a Copy of the Heap -- 2

 

<!-- END IMAGE WITH CAPTION -->

"If an application dies on an OutOfMemoryException , then you can find out how the memory is allocated by restarting your JVM with the -XX:+HeapDumpOnOutfMemoryError option. What this does is that when the JVM is out of memory again, it will generate a heap dump before it exits. If developers have forgotten to set, use jinfo and then give the heap dump and the process ID. The jmap histogram will look something like Figure 16. There are many ways to collect heap information, but typically we use jmap and HPROF," concluded Lee.

Lee offered a third option (Figure 17): Use the JVMTI heap-walker demo application under the demo JVM directory to start the application, and send a SIGQUIT signal to dump out information, though he pointed out that this is more like a learning tool.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 17. Get a Copy of the Heap -- 3

 

<!-- END IMAGE WITH CAPTION -->

Finding Object Retention

Next, use jhat on the binary file. Analyze the information by looking at the heap. What objects are still alive? What is keeping them alive? Where are they allocated? If they are alive, where were they created? See Figure 18.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 18. Finding Object Retention

 

<!-- END IMAGE WITH CAPTION -->

"Typically," said Lee, "analyze the binary dump file in jhat , open a browser, and look for a line called show instance count of all classes excluding platform . 'Show all instances of my application. Don't show platform classes.' Get a list, and look at those objects that are quite large, and click on any of the instances. You will see more information on the particular instance. Look at the object allocated from , which is the trace of the object and the track of its creation, in addition to other objects that reference this object. In HPROF, when you don't optimize during compilation (-O ), you will also get to know which line and from what file the object is created."

He observed that jvisualvm also provides similar functions to analyze the heap dump. Developers who don't want to use HPROF or jmap can use all the information created by jmap and Java, and run it in the NetBeans IDE 6 Profiler. When creating a jmap with NetBeans IDE, use the extension .nps in the file name. When using HPROF, the file name should end with .HPROF . The default is java.hprof .

Finalizers

Lee pointed out that one way to look at finalizers is to use jmap -finalizerinfo <pid> to get a count of the objects that are pending finalization. See Figures 19 and 20. To obtain this information from JConsole , developers should look at the Pending finalization field in VM Summary tab.

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 19. Finalizers

 

<!-- END IMAGE WITH CAPTION --> <!-- BEGIN IMAGE WITH CAPTION -->

Figure 20. Detecting Deadlocks -- 1

 

<!-- END IMAGE WITH CAPTION -->

The developer can obtain the same information from the command line by starting the VM with this option, as shown in Figure 21:

<!-- BEGIN VCD7 CODE SAMPLE COMPONENT -->

XX:+ PrintConcurrentLocks

 

<!-- END VCD7 CODE SAMPLE COMPONENT --> <!-- BEGIN IMAGE WITH CAPTION -->

Figure 21. Detecting Deadlocks -- 2

 

<!-- END IMAGE WITH CAPTION -->

And then inputting

<!-- BEGIN VCD7 CODE SAMPLE COMPONENT -->

jstack -l <pid>

 

<!-- END VCD7 CODE SAMPLE COMPONENT -->

Lock contention, by default, is not enabled in the VM. To activate it from JConsole and attach it to the VM, use MBean, go to thread, and turn it on to get thread-contention information.

Lee concluded by referring to Figure 22, which shows an HPROF example in text form that provides the running methods and threads. "This says that it has sampled the thread 2484 times and found it to be quite active," he said. "In the rest of the HPROF information, look for the trace number and get more information."

<!-- BEGIN IMAGE WITH CAPTION -->

Figure 22. Locating CPU Hogs

 

<!-- END IMAGE WITH CAPTION -->

Finally, he mentioned the JTop tool , which is included in JDK 6 and provides the thread CPU usage of all threads running in the application. JTop shows an application's usage of CPU time per thread and allows developers to easily detect a thread that is using inordinate amounts of CPU time. If high-thread CPU consumption is not an expected behavior, the thread may be looping.

Summary

Lee concluded with three key points.

First, there are lots of options to collect data for analysis. The JDK 6 bundle provides many tools to this end.

Second, recent developments show that Sun is committed not only to making data collection easier but also to making it easier for developers to analyze the collection information. JConsole and more recently jvisualvm in JDK 6 update 7 offer proof of this.

Third, Sun offers lots of resources for developers.