Tomcat Tunning
Measuring performance: specifically request latency. Hey, that sounds a lot like end-user response time.
Advocates that test system should be as closely as possible identical to deployment system: same hardware, same database; same number of records in database, same workload. Aside: Glenn works for Missouri State Lottery, which runs on Tomcat.
Measuring performance tools: JMeter, ab... monitoring in production: mod_jk 1.2 has a configuration directive that creates data (¿which directive is this?); JMX, Introscope monitoring tool; has own shell script that dumps a stack trace of high-latency threads.
JVM tuning: definitely use -server option to HotSpot; memory usage, gc. Dedicate a box to the appserver (we knew that). Says -Xms and -Xmx options control min and max of Java stack... thought they were heap, but he kinda intermingles these terms. Use -Xincgc for incremental GC. May use -Xss to reduce thread stack memory usage (YMMV). Use -verbose:gc to capture GC performance data. Check out a GC article in JavaWorld, and the HotSpot docs.
Tomcat 4.1 in production, includes Jasper v2 which performs a lot better; use Apache in front for medium - high volume sites (duh). Use multiple catbeasts and balance load. Apache 2.0 performs better than 1.3 because of Tomcat connection pooling in Worker.
Tomcat configuration: set reloadable to false (dev use only); set liveDeploy to false (ditto); set debug to 0; set swallowOutput to true. The last one: redirect stdout and stderr to log file instead of standard streams.
Connector Configuration: minProcessors , maxProcessors , acceptCount , enableLookups . acceptCount low, like 10, to prevent backups. enableLookups is dns lookup: turn off.
Jasper: set development to false; use JSP custom tag pooling and configure to fork javac compiles (I suppose only if you're on a JDK and don't have your JSPs precompiled). javac has a memory leak when running embedded.
Databse : use object caching (and more but wasn't listening)
Application design: switch dynamic to static where you can, or cache dynamic output; use database connection pooling and object caching; reduce the amount of HTML generated for large pages (like, how much HTML markup are you wrapping around resultSet items). Links to caching technologies are in the handout.
Design: use JProbe or OptimizeIt to profile; look for thread sync bottlenecks. Tip: java.sql.Timestamp (is that correct?) introduces a synchronization, which can be expensive because it's automatically gnerated in a resultSet.
Review logs.
Tomcat freezes: usually due to GC. Turn the verbose flag on and tune GC according to results.
JVM Memory usage...
No Processors Available: could be Tomcat overload, or maxProcessors too low. Try thread stacktrace dump. These all look like high-load problems.
Share war stories at 8PM at the BOF.
Q&A
Is Tomcat 5 production quality? Unreleased because the Servlet 2.4 spec is not final yet. Code should be solid though because built on 4.1.
When the database goes down, how can Tomcat be informed when it's back up so it can rebuild the connection pool. Using default JDBC that comes with Tomcat. A: One thing one could do is reload the webapp.
Q: Is there an easy way to ignore broken pipe exceptions due to user pressed stop in browser? No, can't suppress. mod_jk source contains log analysis script that analyses broken pipe exceptions and can bring out signal amongst the noise.
Q: Any performance issue on Tomcat behind IIS? No experience.
Advocates that test system should be as closely as possible identical to deployment system: same hardware, same database; same number of records in database, same workload. Aside: Glenn works for Missouri State Lottery, which runs on Tomcat.
Measuring performance tools: JMeter, ab... monitoring in production: mod_jk 1.2 has a configuration directive that creates data (¿which directive is this?); JMX, Introscope monitoring tool; has own shell script that dumps a stack trace of high-latency threads.
JVM tuning: definitely use -server option to HotSpot; memory usage, gc. Dedicate a box to the appserver (we knew that). Says -Xms and -Xmx options control min and max of Java stack... thought they were heap, but he kinda intermingles these terms. Use -Xincgc for incremental GC. May use -Xss to reduce thread stack memory usage (YMMV). Use -verbose:gc to capture GC performance data. Check out a GC article in JavaWorld, and the HotSpot docs.
Tomcat 4.1 in production, includes Jasper v2 which performs a lot better; use Apache in front for medium - high volume sites (duh). Use multiple catbeasts and balance load. Apache 2.0 performs better than 1.3 because of Tomcat connection pooling in Worker.
Tomcat configuration: set reloadable to false (dev use only); set liveDeploy to false (ditto); set debug to 0; set swallowOutput to true. The last one: redirect stdout and stderr to log file instead of standard streams.
Connector Configuration: minProcessors , maxProcessors , acceptCount , enableLookups . acceptCount low, like 10, to prevent backups. enableLookups is dns lookup: turn off.
Jasper: set development to false; use JSP custom tag pooling and configure to fork javac compiles (I suppose only if you're on a JDK and don't have your JSPs precompiled). javac has a memory leak when running embedded.
Databse : use object caching (and more but wasn't listening)
Application design: switch dynamic to static where you can, or cache dynamic output; use database connection pooling and object caching; reduce the amount of HTML generated for large pages (like, how much HTML markup are you wrapping around resultSet items). Links to caching technologies are in the handout.
Design: use JProbe or OptimizeIt to profile; look for thread sync bottlenecks. Tip: java.sql.Timestamp (is that correct?) introduces a synchronization, which can be expensive because it's automatically gnerated in a resultSet.
Review logs.
Tomcat freezes: usually due to GC. Turn the verbose flag on and tune GC according to results.
JVM Memory usage...
No Processors Available: could be Tomcat overload, or maxProcessors too low. Try thread stacktrace dump. These all look like high-load problems.
Share war stories at 8PM at the BOF.
Q&A
Is Tomcat 5 production quality? Unreleased because the Servlet 2.4 spec is not final yet. Code should be solid though because built on 4.1.
When the database goes down, how can Tomcat be informed when it's back up so it can rebuild the connection pool. Using default JDBC that comes with Tomcat. A: One thing one could do is reload the webapp.
Q: Is there an easy way to ignore broken pipe exceptions due to user pressed stop in browser? No, can't suppress. mod_jk source contains log analysis script that analyses broken pipe exceptions and can bring out signal amongst the noise.
Q: Any performance issue on Tomcat behind IIS? No experience.