This is the second article in the series of "Become a Java GC Expert". In the first issue Understanding Java Garbage Collection we have learned about the processes for different GC algorithms, about how GC works, what Young and Old Generation is, what you should know about the 5 types of GC in the new JDK 7, and what the performance implications are for each of these GC types.
In this article, I will explain how JVM is actually running Garbage Collection in the real time.
What is GC Monitoring?
Garbage Collection Monitoring refers to the process of figuring out how JVM is running GC. For example, we can find out:
- when an object in young has moved to old and by how much,
- or when stop-the-world has occurred and for how long.
GC monitoring is carried out to see if JVM is running GC efficiently, and to check if additional GC tuning is necessary. Based on this information, the application can be edited or GC method can be changed (GC tuning).
How to Monitor GC?
There are different ways to monitor GC, but the only difference is how the GC operation information is shown. GC is done by JVM, and since the GC monitoring tools disclose the GC information provided by JVM, you will get the same results no matter how you monitor GC. Therefore, you do not need to learn all methods to monitor GC, but since it only requires a little amount of time to learn each GC monitoring method, knowing a few of them can help you use the right one for different situations and environments.
The tools or JVM options listed below cannot be used universally regardless of the HVM vendor. This is because there is no need for a "standard" for disclosing GC information. In this example we will use HotSpot JVM (Oracle JVM). Since NHN is using Oracle (Sun) JVM, there should be no difficulties in applying the tools or JVM options that we are explaining here.
First, the GC monitoring methods can be separated into CUI and GUI depending on the access interface. The typical CUI GC monitoring method involves using a separate CUI application called "jstat", or selecting a JVM option called "verbosegc" when running JVM.
GUI GC monitoring is done by using a separate GUI application, and three most commonly used applications would be "jconsole", "jvisualvm" and "Visual GC".
Let's learn more about each method.
jstat
jstat is a monitoring tool in HotSpot JVM. Other monitoring tools for HotSpot JVM are jps and jstatd. Sometimes, you need all three tools to monitor a Java application.
jstat does not provide only the GC operation information display. It also provides class loader operation information or Just-in-Time compiler operation information. Among all the information jstat can provide, in this article we will only cover its functionality to monitor GC operating information.
jstat is located in $JDK_HOME/bin, so if java or javac can run without setting a separate directory from the command line, so can jstat.
You can try running the following in the command line.
|
1
2
3
4
5
6
7
8
|
$> jstat –gc $<vmid$> 1000
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3008.0 3072.0 0.0 1511.1 343360.0 46383.0 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588
3008.0 3072.0 0.0 1511.1 343360.0 47530.9 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588
3008.0 3072.0 0.0 1511.1 343360.0 47793.0 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588
$>
|
Just like in the example, the real type data will be output along with the following columns: S0C S1C S0U S1U EC EU OC OU PC.
vmid (Virtual Machine ID), as its name implies, is the ID for the VM. Java applications running either on a local machine or on a remote machine can be specified using vmid. The vmid for Java application running on a local machine is called lvmid (Local vmid), and usually is PID. To find out the lvmid, you can write the PID value using a ps command or Windows task manager, but we suggest jps because PID and lvmid does not always match. jpsstands for Java PS. jps shows vmids and main method information. Just like ps shows PIDs and process names.
Find out the vmid of the Java application that you want to monitor by using jps, then use it as a parameter in jstat. If you use jps alone, only bootstrap information will show when several WAS instances are running in one equipment. We suggest that you use ps -ef | grep java command along with jps.
GC performance data needs constant observation, therefore when running jstat, try to output the GC monitoring information on a regular basis.
For example, running "jstat –gc <vmid> 1000" (or 1s) will display the GC monitoring data on the console every 1 second. "jstat –gc <vmid> 1000 10" will display the GC monitoring information once every 1 second for 10 times in total.
There are many options other than -gc, among which GC related ones are listed below.
| Option Name | Description |
| gc | It shows the current size for each heap area and its current usage (Ede, survivor, old, etc.), total number of GC performed, and the accumulated time for GC operations. |
| gccapactiy | It shows the minimum size (ms) and maximum size (mx) of each heap area, current size, and the number of GC performed for each area. (Does not show current usage and accumulated time for GC operations.) |
| gccause | It shows the "information provided by -gcutil" + reason for the last GC and the reason for the current GC. |
| gcnew | Shows the GC performance data for the new area. |
| gcnewcapacity | Shows statistics for the size of new area. |
| gcold | Shows the GC performance data for the old area. |
| gcoldcapacity | Shows statistics for the size of old area. |
| gcpermcapacity | Shows statistics for the permanent area. |
| gcutil | Shows the usage for each heap area in percentage. Also shows the total number of GC performed and the accumulated time for GC operations. |
Only looking at frequency, you will probably use -gcutil (or -gccause), -gc and -gccapacity the most in that order.
- -gcutil is used to check the usage of heap areas, the number of GC performed, and the total accumulated time for GC operations,
- while -gccapacity option and others can be used to check the actual size allocated.
You can see the following output by using the -gc option:
|
1
2
3
4
|
S0C S1C … GCT
1248.0 896.0 … 1.246
1248.0 896.0 … 1.246
… … … …
|
Different jstat options show different types of columns, which are listed below. Each column information will be displayed when you use the "jstat option" listed on the right.
| Column | Description | Jstat Option |
| S0C | Displays the current size of Survivor0 area in KB | -gc -gccapacity -gcnew -gcnewcapacity |
| S1C | Displays the current size of Survivor1 area in KB | -gc -gccapacity -gcnew -gcnewcapacity |
| S0U | Displays the current usage of Survivor0 area in KB | -gc -gcnew |
| S1U | Displays the current usage of Survivor1 area in KB | -gc -gcnew |
| EC | Displays the current size of Eden area in KB | -gc -gccapacity -gcnew -gcnewcapacity |
| EU | Displays the current usage of Eden area in KB | -gc -gcnew |
| OC | Displays the current size of old area in KB | -gc -gccapacity -gcold -gcoldcapacity |
| OU | Displays the current usage of old area in KB | -gc -gcold |
| PC | Displays the current size of permanent area in KB | -gc -gccapacity -gcold -gcoldcapacity -gcpermcapacity |
| PU | Displays the current usage of permanent area in KB | -gc -gcold |
| YGC | The number of GC event occurred in young area | -gc -gccapacity -gcnew -gcnewcapacity -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause |
| YGCT | The accumulated time for GC operations for Yong area | -gc -gcnew -gcutil -gccause |
| FGC | The number of full GC event occurred | -gc -gccapacity -gcnew -gcnewcapacity -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause |
| FGCT | The accumulated time for full GC operations | -gc -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause |
| GCT | The total accumulated time for GC operations | -gc -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause |
| NGCMN | The minimum size of new area in KB | -gccapacity -gcnewcapacity |
| NGCMX | The maximum size of max area in KB | -gccapacity -gcnewcapacity |
| NGC | The current size of new area in KB | -gccapacity -gcnewcapacity |
| OGCMN | The minimum size of old area in KB | -gccapacity -gcoldcapacity |
| OGCMX | The maximum size of old area in KB | -gccapacity -gcoldcapacity |
| OGC | The current size of old area in KB | -gccapacity -gcoldcapacity |
| PGCMN | The minimum size of permanent area in KB | -gccapacity -gcpermcapacity |
| PGCMX | The maximum size of permanent area in KB | -gccapacity -gcpermcapacity |
| PGC | The current size of permanent generation area in KB | -gccapacity -gcpermcapacity |
| PC | The current size of permanent area in KB | -gccapacity -gcpermcapacity |
| PU | The current usage of permanent area in KB | -gc -gcold |
| LGCC | The cause for the last GC occurrence | -gccause |
| GCC | The cause for the current GC occurrence | -gccause |
| TT | Tenuring threshold. If copied this amount of times in young area (S0 ->S1, S1->S0), they are then moved to old area. | -gcnew |
| MTT | Maximum Tenuring threshold. If copied this amount of times inside young arae, then they are moved to old area. | -gcnew |
| DSS | Adequate size of survivor in KB | -gcnew |
The advantage of jstat is that it can always monitor the GC operation data of Java applications running on local/remote machine, as long as a console can be used. From these items, the following result is output when –gcutil is used. At the time of GC tuning, pay careful attention to YGC, YGCT, FGC, FGCT and GCT.
|
1
2
3
4
|
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
|
These items are important because they show how much time was spent in running GC.
In this example, YGC is 217 and YGCT is 0.928. So, after calculating the arithmetical average, you can see that it required about 4 ms (0.004 seconds) for each young GC. Likewise, the average full GC time us 33ms.
But the arithmetical average often does not help analyzing the actual GC problem. This is due to the severe deviations in GC operation time. (In other words, if the average time is 0.067 seconds for a full GC, one GC may have lasted 1 ms while the other one lasted 57 ms.) In order to check the individual GC time instead of the arithmetical average time, it is better to use -verbosegc.
-verbosegc
-verbosegc is one of the JVM options specified when running a Java application. While jstat can monitor any JVM application that has not specified any options, -verbosegc needs to be specified in the beginning, so it could be seen as an unnecessary option (since jstat can be used instead). However, as -verbosegc displays easy to understand output results whenever a GC occurs, it is very helpful for monitoring rough GC information.
| jstat | -verbosegc | |
| Monitoring Target | Java application running on a machine that can log in to a terminal, or a remote Java application that can connect to the network by using jstatd | Only when -verbogc was specified as a JVM starting option |
| Output information | Heap status (usage, maximum size, number of times for GC/time, etc.) | Size of ew and old area before/after GC, and GC operation time |
| Output Time | Every designated time | Whenever GC occurs |
| Whenever useful | When trying to observe the changes of the size of heap area | When trying to see the effect of a single GC |
The followings are other options that can be used with -verbosegc.
- -XX:+PrintGCDetails
- -XX:+PrintGCTimeStamps
- -XX:+PrintHeapAtGC
- -XX:+PrintGCDateStamps (from JDK 6 update 4)
If only -verbosegc is used, then -XX:+PrintGCDetails is applied by default. Additional options for –verbosgc are not exclusive and can be mixed and used together.
When using -verbosegc, you can see the results in the following format whenever a minor GC occurs.
[GC [<collector>: <starting occupancy1> -> <ending occupancy1>, <pause time1> secs] <starting occupancy3> -> <ending occupancy3>, <pause time3> secs]
|
| Collector | Name of Collector Used for minor gc |
| starting occupancy1 | The size of young area before GC |
| ending occupancy1 | The size of young area after GC |
| pause time1 | The time when the Java application stopped running for minor GC |
| starting occupancy3 | The total size of heap area before GC |
| ending occupancy3 | The total size of heap area after GC |
| pause time3 | The time when the Java application stopped running for overall heap GC, including major GC |
This is an example of -verbosegc output for minor GC:
|
1
2
3
4
|
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995
|
This is the example of output results after an Full GC occurred.
|
1
|
[Full GC [Tenured: 3485K->4095K(4096K), 0.1745373 secs] 61244K->7418K(63104K), [Perm : 10756K->10756K(12288K)], 0.1762129 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]
|
If a CMS collector is used, then the following CMS information can be provided as well.
As -verbosegc option outputs a log every time a GC event occurs, it is easy to see the changes of the heap usage rates caused by GC operation.
(Java) VisualVM + Visual GC
Java Visual VM is a GUI profiling/monitoring tool provided by Oracle JDK.
Figure 1: VisualVM Screenshot.
Instead of the version that is included with JDK, you can download Visual VM directly from its website. For the sake of convenience, the version included with JDK will be referred to as Java VisualVM (jvisualvm), and the version available from the website will be referred to as Visual VM (visualvm). The features of the two are not exactly identical, as there are slight differences, such as when installing plug-ins. Personally, I prefer the Visual VM version, which can be downloaded from the website.
After running Visual VM, if you select the application that you wish to monitor from the window on the left side, you can find the "Monitoring" tab there. You can get the basic information about GC and Heap from this Monitoring tab.
Though the basic GC status is also available through the basic features of VisualVM, you cannot access detailed information that is available from either jstat or -verbosegc option.
If you want the detailed information provided by jstat, then it is recommended to install the Visual GC plug-in.
Visual GC can be accessed in real time from the Tools menu.
Figure 2: Viusal GC Installation Screenshot.
By using Visual GC, you can see the information provided by running jstatd in a more intuitive way.
Figure 3: Visual GC execution screenshot.
HPJMeter
HPJMeter is convenient for analyzing -verbosegc output results. If Visual GC can be considered as the GUI equivalent of jstat, then HPJMeter would be the GUI equivalent of -verbosgc. Of course, GC analysis is just one of the many features provided by HPJMeter. HPJMeter is a performance monitoring tool developed by HP. It can be used in HP-UX, as well as Linux and MS Windows.
Originally, a tool called HPTune used to provide the GUI analysis feature for -verbosegc. However, since the HPTune feature has been integrated into HPJMeter since version 3.0, there is no need to download HPTune separately.
When executing an application, the -verbosegc output results will be redirected to a separate file.
You can open the redirected file with HPJMeter, which allows faster and easier GC performance data analysis through the intuitive GUI.
Figure 4: HPJMeter.
What is the Next Article About?
In this article I focused on how to monitor GC operation information, as the preparation stage for GC tuning. From my personal experience, I suggest using jstat to monitor GC operation, and if you feel that it takes too lmuch time to execute GC, then try -verbosegc option to analyze GC. The general GC tuning process is to analyze the results after applying the changed GC options after the -verbosegc option has been applied based on the analysis. In the next article, we will see the best options for executing GC tuning by using real cases as our examples.