glassfish：Indexing and Search Service Performance Tuning

Topics:

• Bootstrapping Users
• Restful Web Services Searches
• Setting Up Large Deployments
• Tuning Restful Web Services

Bootstrapping Users

During the bootstrap process (indexSvcBootstrap.sh), frequent access to the dIndex might cause some performance issues, resulting in slow creation of accounts. To reduce some of the overhead on dIndex, increase the value of the following jiss.conf parameter:

# optimizeinterval: number of dIndex writes before optimize is done
iss.store.account.optimizeinterval=5000

Higher values cause less frequent optimization of the dIndex, reducing overhead while bootstrapping.

You can also increase the value of the following parameter to 5:

# optimizelevel: default level of optimize (number of segments left)
iss.store.account.optimizelevel=1

Check the iss-indexsvc.log to see how frequently the dIndex is being optimized, and how long it is taking. You should see a log entry similar to the following one:

INFO: SharedWriter.close: optimizing index:/dIndex Time: 123ms

Restful Web Services Searches

Depending on the incoming search rate, you can modify the following parameter in the jiss.conf file to accommodate a higher search rate:

# Size of proxy connection pool in restful web services
iss.rest.proxypool.size=512

The default is currently set to 512. As this value is larger than 100, you need to make sure that the following parameter is added to the IMQ broker config.properties file, if it doesn't already exist:

imq.autocreate.destination.maxNumProducers=-1

Otherwise, Restful Web Services fail with the following error message:

Could not create connection and producer com.sun.messaging.jms.JMSException: \[ADD_PRODUCER_REPLY(19)\] \[C4036\]: A broker error occurred. :\[409\] \[B4183\]: Producer can not be added to destination SearchTopic \[Topic\], limit of 100 producers would be exceeded

Also, once this value goes above 500, the following parameter also needs to increase to accommodate more than 2 * 500 threads in IMQ broker:

imq.jms.max_threads=2000

Setting Up Large Deployments

For any large deployment, using the indexSvcBootstrap.sh command to assign accounts randomly to groups is not the best approach, both for performance and for data management reasons. Instead, you should pre-allocate accounts by using the issadmin.sh --createaccount command, in conjunction with the --accountlist acctfile option, which contains the desired mapping of accounts to groups. Using this approach minimizes the contention between accounts in the dIndex. Using this method, it is possible to cut the bootstrap time of 20,000 accounts by over 15 percent.

Tuning Restful Web Services

This section contains the following topics:

• Application Server domain.xml File
• JMQ Broker config.properties File

Note This information was gathered as part of testing on CMT platforms.

Application Server domain.xml File

All configuration changes were applied directly to the main Application Server configuration file, domain.xml. If necessary, the Application Server administrative interface can be used. This configuration file is located in the $INSTALL/domains/domain1/config directory.

Application Server modifications are based on the recommendations from the Application Server Performance Tuning Guide and Scott Oak's blog.

http-listener

This the http-listener entry that is used to accept search requests from the imapd process.

<http-listener acceptor-threads="64" address="10.5.185.213" blocking-enabled="false" default-virtual-server="server" enabled="true" family="inet" id="http-listener-1" port="8070" security-enabled="false" server-name="r54s29-02.red.iplanet.com" xpowered-by="true">
</http-listener>

The acceptor thread value was set to 64 to match the number of virtual processors available. The IP address was added to indicate which network interface is dedicated to ISS web service requests.

request-processing

This entry determines how http requests are processed by the Application Server.

<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="32" request-timeout-in-seconds="30" thread-count="128" thread-increment="8"/>

The initial-thread-count, thread-count, and thread-increment settings are modified to make more efficient use of the 64 virtual processors available on a Sun SPARC Enterprise T5220.

keep-alive Connections

The following keep-alive setting is used. Keep alive connections enable connection re-use, saving resources that would be required to open and close a socket connection for every HTTP request that is received.

<keep-alive max-connections="4096" thread-count="64" timeout-in-seconds="30"/>

The thread count is modified to make more efficient use of the 64 virtual processors available on a Sun SPARC Enterprise T5220.

JVM Options

The following JVM options are added based on advice given in the Scott Oaks blog entry for the Sun SPARC Enterprise T5220 platform : 

<jvm-options>-server</jvm-options>
<jvm-options>-d64</jvm-options>
<jvm-options>-XX:LargePageSizeInBytes=256m</jvm-options>
<jvm-options>-Xms5120m</jvm-options>
<jvm-options>-Xmx5120m</jvm-options>
<jvm-options>-Xmn1g</jvm-options>
<jvm-options>-XX:+UseParallelGC</jvm-option>
<jvm-options>-XX:+UseParallelOldGC</jvm-options>
<jvm-options>-XX:ParallelGCThreads=16</jvm-options>
<jvm-options>-XX:+AggressiveOpts</jvm-options>

Access Logging

Access logging, which can be disk intensive, is disabled by setting this entry:

<property name="accessLoggingEnabled" value="false"/>

JMQ Broker config.properties File

Non-default configuration settings are added to the bottom of the /var/imq/instances/imqbroker/props/config.properties file.

Java Message Service Threads

Increasing the jms.max_threads used by the ISS jmqbroker enables greater number of requests to be processed in any given time interval. The default is 1000, but this has been determined to be too small a number of threads for this jmqbroker. The jmqbroker handles both indexing and search requests, unlike the jmqbroker used by the Messaging Server's JMQ notification plugin, which handles only indexing requests.

imq.jms.max_threads=2000

Maximum Number of Producers

The default number of message producers that can be created by the jmqbroker is 100. This value is too low to handle the amount of messages that are typically generated by an active Messaging Server deployment supporting more than 10,000 active users. Disabling the setting allows an unlimited number of producers to be created to handle the load.

imq.autocreate.destination.maxNumProducers=-1

JVM Options
In the file /etc/imq/imqbrokerd.conf, the jvm options are specified:

ARGS=-javahome /usr/jdk/latest -vmargs -d64 -vmargs -Xms4096m -vmargs -Xmx4096m 

Allows use of the 64 bit jvm and reserves 4 GB of memory for the jmq broker. Current scalability test runs have the messaging server and the iss jmq broker use a bit more or less than 1.5 GB of jvm heap.