Symptoms
The CRX/CQ process uses 100% of the CPU, the system doesn't respond, or the system is very slow.
Cause
Such problems can have many causes.
Analysis, Resolution
CRX 2.x
A simple CPU profiling tool is included in CRX 2.0.x. To start it, open:
http://localhost:7402/crx/diagnostic/prof.jsp
CRX 1.x
To help analyzing the problem, create a few full thread dumps. Those thread dumps can then be analyzed (for example using samurai.jar).
Creating Full Thread Dumps
Get the Process ID
To get the process id of your Java process, use
jps -l
If this doesn't work (path not set, JDK not installed, or older Java version), use
ps -el | grep java
Full Thread Dumps
To analyze a performance problem or a blocked process, create about 10 full thread dumps with about 1 second delay. If the problem may be related to clustering, please create at least 10 full thread dumps on each cluster node. If possible, the thread dumps should be created at roughly the same time (it doesn't need to be exact).
If installed, use jstack. This will print the thread dumps to system out:
jstack <pid>
This will append a full thread dump to a file:
jstack <pid> >> threadDumpNode1.txt
If this doesn't work, use kill -QUIT. This will print the thread dumps to the log file:
kill -QUIT <pid>
NOTE: If the steps above steps for obtaining thread dumps do not work in your environment then please see this article.
Check CPU Usage
To analyze the problem, it is important to know if CRX /CQ is running in a endless loop, or if it's merely sleeping. To find it out, type
top
This will get you the list of processes, sorted by CPU usage. If the top process is a 'java' process, and if the PID matches CRX/CQ, then the process is running full speed.
If you are unsure how to interpret the results, run the following statement and then include the filetop.txt in your problem report:
top -l5 -s5 > top.txt
Check Session Count
In many many cases the problem is the number of open sessions it too large. At some point, this slows down processing. To find out if this is the case, run
jps -l (to get the process id of the Java process)
jmap -histo <pid> | grep CRXSession (to get the number of open session)
If this is in fact the problem (the number is higher than a few hundred sessions) then this needs to be analyzed. Possibly a session pool is used (depending on the version of CRX / CQ there might be a hotfix for the given problem), or an internal (possibly application level) cache references sessions. To analyze where those sessions are opened, see the 'Analyze Unclosed Sessions' page.
Do Not Kill the Process
The CRX process should never be killed, also not when stopping takes too long. If you need to kill a process that is not responding, please create a full thread dump first and log a bug.
If you do kill the CRX process, the next time you start it up the Tar PM may create backup_.tar files.