Tuning the Performance of documentum UCF Content Transfer

In Documentum applications, the speed at which users can retrieve or add content to the system plays a big part in the overall end user experience. Network latency and bandwidth restrictions, physical architecture and client configuration all have an impact on the time it takes between initiating a content transfer, performing the upload or download, and returning control back to the user.

This document will discuss various tuning options available to achieve the best possible performance for UCF content transfer in your network environment.

Overview of UCF

UCF (Unified Client Facilities) is the EMC Documentum framework for performing content transfer between the client and content servers over HTTP. It provides many features that extend beyond simple file transfer protocol, such as:

*
Content compression to optimize transfers of larger files over the network
*
Support for complex documents, such as XML files
*
Support for compound document types, such as OLE objects inside a Microsoft Office document
*
Awareness of locally downloaded files, and the ability to avoid downloading the same content twice
*
Registry support, to allow checkout from one application and check in from another
*
Recoverability in the case of a brief network interruption
*
Location aware, to allow users to transfer content from the closest ACS or BOCS server

High-Level UCF Operations

When a content transfer is initiated, the UCF client running on the end user machine is initiated if it is not already running. The applet loads the UCF client jars and initiates contact with the UCF server, running on the application server.

The UCF client and server pass information back and forth about the environment and requested action before content is transferred between the two machines. Depending on the architecture and configuration, content may be transferred through the application server, directly to or from an ACS server on the content server, or a BOCS server located near the user.

After the content transfer is complete, there will be some additional communication from server to client to provide instructions on registry entries to be added or modified, directions on how to launch the appropriate application for view or edit operations, or instructions on how to delete the local file on check in.

Optimizing UCF for Best Performance

Tip #1 – Use ACS and BOCS for Content Transfer Whenever Possible

The introduction of ACS and BOCS servers in 5.3 SP1 have allowed content transfer to be performed directly from the content server, rather than requiring that the content always pass through the application server. This not only removes the double-hop that is required (which is very costly for small files) but also reduces the load on the application server.

In the 5.3 timeframe, ACS and BOCS could only be used for download operations such as checkout, export and view. All upload operations still passed through the application server.

From D6 onward, ACS and BOCS servers can also be used for upload operations.

Note that not all applications are network location aware and able to take advantage of remote ACS and BOCS servers.

Tip #2 – Optimize UCF Client Initialization

In 5.3 SP1 through SP5, D6 and D6SP1, the UCF client engine will time out after one minute of inactivity, and the javaw.exe process would be terminated. This means that if a user initiates another content transfer request after the timeout has occurred, the browser must re-initialize the UCF client and restart javaw.exe.

This value can be increased in the ucf.client.config.xml file as follows:

Edit the ucf.client.config.xml file (located at C:\Documents and Settings\\Documentum\ucf\\shared\config\ucf.client.config.xml) and add the following option to extend the UCF Client Timeout.

maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" redirectPort="8443" acceptCount="100"

connectionTimeout="20000" disableUploadTimeout="true"
socketBuffer="64000"/>

If you are using 5.3 ACS and BOCS the application server is Tomcat 5.0.28. In versions 5.3 SP1 or SP2, you will need to add the socketBuffer parameter. In 5.3 SP3 through SP6, this parameter should already have been added by the installer.

In 6.5 and beyond, Jboss is the container for ACS and BOCS. In 6.5 SP1, enhancements were added to the embedded Jboss server to enable it to recognize and use the socketBuffer parameter.

BEA Weblogic

If you are using BEA Weblogic as your application server and you are not using ACS, be sure to add the -Dweblogic.ChunkSize parameter. By default, the ChunkSize is 4k which is much too small.

It can be set in the config.xml for the domain:

server1

false

localhost
17001

-Dweblogic.Chunksize=64000

weblogic
{3DES}wmzrpHOymvTy3q8TOeF1qQ==

It can also be defined in the set_domain_env batch file with the other JAVA_PROPERTIES values:

set JAVA_PROPERTIES=-Dplatform.home=%WL_HOME% \

-Dwls.home=%WLS_HOME% -Dwli.home=%WLI_HOME% \

-Dweblogic.Chunksize=64000

In D6 and D6 SP1, the application server that hosts ACS and BOCS is BEA Weblogic, and the installer has automatically added those options to the script that starts the embedded Weblogic server (startMethodServer.cmd for example).

IBM Websphere

If you are not using ACS and content is being transferred through WAS, then you may see improvements in content download performance for large files by modifying the ResponseChunkSize configuration parameter in the plugin-cfg.xml.

where N equal the chunk size in kilobytes. The default value is 64 (kilobytes).

Tip #6 – Improve Upload Performance in D6 and Above

In D6, a parameter was added to the ucf.client.config.xml file that allowed the users to specify a chunk size to be used when uploading content.

Internal testing showed that a value of 49152 worked best with ACS on BEA when simulating various WAN conditions. However, in environments where customers are not using ACS write, or whose network conditions are different from that which was tested, this value may not be optimal. Increasing or decreasing the optimal.chunk.size parameter based on your specific network conditions can be beneficial.

49152

Note that this setting is ignored if the client is using Sun Java version 1.6 due to a Java bug (http://bugs.sun.com/bugdatabase/view_bug.do”bug_id=6631048).

Tip #7 – Use Adaptive Parallel Content Transfer to Consume More Available Bandwidth

By default in most versions of UCF, content download operations will be done in a single thread. If sufficient network bandwidth is available, the download operations could perform significantly faster if the file was broken up into smaller pieces and transferred by multiple concurrent threads.

In 5.3 SP6 and D6.5 the UCF client can be configured to use multiple threads when downloading files. This is controlled in the ucf.client.config.xml file using the following parameters:

5

131072

300

131072

The “max.parallel.download.streams” parameter limits the number of threads that can be run concurrently when performing a parallel download. In this example, 5 separate streams could potentially be initialized. If one of the streams finishes their assigned task ahead of the others, it is then free to request a new range of bytes to be downloaded.

The “min.parallel.segment.size” parameter specifies that if the remaining portion of a document after a byte range is assigned is smaller than a specified value, it should not be assigned to a new thread. Rather the original thread’s byte range should be extended to include those additional bytes. For example, if a thread is supposed to download 500Kb of a 600Kb file, as the remaining number of bytes is less than 128k, no new thread will be started and the original thread will assume ownership of those additional bytes.

The “Adaptive” part of parallel content transfer is controlled by the remaining two parameters. As tests have shown that disk I/O actually becomes a bottleneck and degrades performance if the content is transferred in parallel over LAN conditions, it is important to be able to control when the parallel content transfer is actually turned on. In this case, the UCF client will measure how many bytes were transferred initially by a single thread within a specified timeframe. If the number of bytes is less than what it expected, parallel content transfer will be turned on. If it is more than was expected, then it is assumed that bandwidth and latency are sufficient for a single thread to transfer the content most efficiently.

In the example above, the UCF client will measure the number of bytes that have been downloaded in the first 300ms of transfer (or as close to that time as possible). It will enable parallel content transfer if less than 131072 bytes (128k) have been downloaded, and disable it if more than 128k has been downloaded.

If your remote users have very high latency, it is likely that you will want to increase the “measurement.time.interval” to match the round trip time plus time to download a small portion of the file. For example, if you have 200ms round-trip latency, you might consider increasing the measurement.time.interval to 500ms to compensate for the time it takes for the request to reach the ACS server and start the content transfer.

In order to permanently disable parallel content transfer, all that must be done is changing the “max.parallel.download.streams” to 1, or decreasing the “single.thread.throughput” to a very small number, such as 1. In this case, regardless of the actual throughput, parallel content transfer will not be used. This is especially important in load testing scenarios, where many clients are being simulated from the same client machine.

Tip #8 – Tuning Content Download when Documentum User Directory is on a Network Drive

If the location of the Viewed and Checkout directories are on a network drive, or users regularly export content to a network drive, performance can be improved by setting the value of “ucf.file.buffer.size” to a larger number. The default value is 32768.

To set it for a single user, simply add the following to the ucf.client.config.xml on the client machine:

61440

To set the default for all users, add the same option to the ucf.installer.config.xml file on the application server and force a new version of UCF on the server side using the steps in Appendix B.

Tip #9 – Use UCF Client Logging to Measure performance

UCF Client logs are extremely useful when diagnosing UCF performance.

UCF client logging is enabled on the end user’s machine.

Edit the ucf.client.logging.properties file, located at:
C:\Documents and Settings\\Documentum\ucf\\shared\config

For best results, set the log level to FINEST, as shown below:

handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler

.level=FINEST

#——————-

java.util.logging.FileHandler.pattern=C:/Documentum/logs/ucf.client%u.log

java.util.logging.FileHandler.limit=10485760

java.util.logging.FileHandler.count=10

java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

java.util.logging.FileHandler.encoding=UTF-8

#——————-

java.util.logging.ConsoleHandler.level=OFF

java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter

In addition, to prevent the UCF client log from overwriting itself each time, change the value of java.util.logging.FileHandler.count to a number greater than 1. As each new client initializes, the old logs will be renamed and preserved for further analysis.

In the UCF client logs you will see entries such as “Logging request: getFile” and “Handled request: getFile” with timestamps. This can be used to better measure the time taken in the UCF pre-processing, content transfer and post-processing phases.

Appendix A:

Reduce the Effect of On-Access Scanning on UCF Initialization and Other Java Applets After Reboot

1.
Right-click on the VirusScan icon in the system tray and choose “On Access Scan Properties.
2.
Click on All Processes.
3.
Click on Advanced.
4.
Ensure “Scan inside archives” is NOT selected.
5.
Click on the Detection tab.
6.
Click on the Exclusions button.
7.
Click on Add.
8.
Add C:\Documents and Settings\s38737\Documentum\ucf , with “Also exclude subfolders”, “On Read” and “On Write”.
9.
Click on OK to save.
10.
Click Add to add another path.
11.
Add C:\Program Files\Java with “Also exclude subfolders” and “On Read”.
12.
Click OK to save.

Appendix B:

Changing UCF Client Settings for All Users

1. Edit the \webtop\wdk\contentXfer\ucf.installer.config.xml file.

2. In this file, you will find the following line,

<"xml version="1.0" encoding="UTF-8"">
<"dctm fileVersion="5.3.0.1" compatibilityVersion="5.3.0.1"">

3. Change the highlighted text above from 5.3.0.XXX to 5.3.0.XXXa

4. Under the configurations tag, change or add the desired option.

value

The next time the UCF client engine is initialized on the client side, the new UCF settings will be downloaded and added to the ucf.client.config.xml file.

There may be cases where the “default” values in the ucf.installer.config.xml should not be applied to the client side. In this case, the “persistent” attribute can be added to the option on the client side. This will prevent any changes on the server side from being applied to that particular client.

value

If not specified, then persistent is set to false.

The persistent property can be specified on the ucf.installer.config.xml and in the ucf.client.config.xml. The following chart outlines the expected behavior:

Server Client Description

false false Server value will override client value.

true true Server value will override client value.

true true Server value will override client value.

false true Client value will not be overridden.