CBR Troubleshooting for FileNet CEMP 4.x.x
1. Introduction
2. Indexing data on disk
3. Verity Version
4. Tools
4.1. K2 Dashboard
4.2. Rcvdk
4.3. Rck2
4.4. mkvdk
4.5. Testqp
4.6. Testing configurations
5. Commonly missed points
5.1. K2 Operating System User vs. K2 Security User
5.2. Verity Administration User
5.3. Collection Security
6. Logs
7. Verity processes
8. Mapped drives on Windows
9. Multiple ticket servers and K2 Dashboard
10. Reconfiguring
10.1. Changing Template Type or CBR Locale
10.2. Changing Verity style files
11. Disabling full text indexing and deleting indexes
12. Re-indexing everything
13. Errors
13.1. CBR_CANT_LOGON_TO_VERITY
13.1.1. Invalid User Credentials
13.1.2. Out of memory
13.2. CBR_CANT_CREATE_COLLECTION
13.3. File Not Found
13.4. Error in write_huge
14. Log Warnings
14.1. Collection(s) not recognized
15. Backdating Configurations
16. Performance issues
16.1. File permissions
16.2. Indexing and Optimize times
16.3. CBR Locales
16.4. CPU usage on the Verity server too low
16.5. Retrieval slow during high document ingestion
16.6. CBR batch size and lease duration
16.7. High CPU usage on security server
16.8. Stop word files
17. No indexing activity
17.1. CBR dispatcher enabled
17.2. Requests locked
17.3. Index Job restart
17.4. Verity Index Server hung
18. Search returns no results or unexpected results
18.1. Verify CBR is enabled
18.2. Check if IndexRequests are pending
18.3. Check CE server log
18.4. Check that the CBR Dispatcher is enabled
18.5. Check Verity Collection Document Count
18.6. Query Verity Collection
18.7. Check object in Verity Collection
18.8. Use testqp
18.9. Check Security
19. CBR tracing
1. Introduction
This document describes various problems that can occur during full text indexing operations, and how to diagnose and solve those problems. Some of the more common errors are discussed, as well as methods to eliminate those errors. Performance issues are also described. Plus tools are listed that can be used for diagnostic and test replication purposes.
This document applies to version 4.0.1, fixpack 4 and later fixpacks, of the CEMP server software. Some comments are not applicable to prior releases.
2. Indexing data on disk
The main unit of indexing data is a Verity Collection. A Verity Collection is just a folder in the file system with several subfolders, and files of various types in each folder. The actual data being indexed is in the “parts” subfolder of the collection folder.
Verity Collections are contained within Verity Index Areas. A Verity Index Area is just a base folder which contains one or more Verity Collections. Each Verity Collection belongs to just one Verity Index Area.
The Verity Index Area and Verity Collection are tracked in the CE database with VerityIndexArea and VerityCollection objects. To find the location of a Verity Collection in the filesystem, you can take the VerityIndexArea.RootDirectoryPath property, and append the VerityCollection.CollectionName property, and that will be the folder that contains the collection.
In some instances a collection will need to be sent to another location for problem analysis. To do so, just find the file system directory identified by the VerityIndexArea.RootDirectoryPath and VerityCollection.CollectionName, and put that folder and all contents of it into a zip or tar file.
3. Verity Version
To determine the version of Verity that is running, look in the file C:/Program Files/verity/k2_61/k2/common/releaseinfo_k2tk.txt (windows) or /opt/verity//k2_61/k2/common/releaseinfo_k2tk.txt (UNIX), or a directory of similar name if you didn’t install to the default location. That file will have something like “Autonomy(R) K2 Toolkit 6.1.4” as the first line of the file for a 6.1.4 version of Verity
4. Tools
There are several tools supplied by Verity for analyzing collections or configuration problems in general. These tools are all described in the Verity documentation, which can be found at http://localhost:9990/verity_docs/index.html on the server which has Verity loaded, provided you used the default ports (as most people do). The most commonly used tools and commonly used commands in those tools are listed here to supply a point to focus on for diagnostic capabilities. Note that when submitting problems to Autonomy, reproducing a problem with these tools must typically be done to let Autonomy diagnose it.
4.1. K2 Dashboard
The K2 Dashboard tool is a web based program at http://localhost:9990/verity_dashboard, where “localhost” can also be replaced with a machine name. This tool will show each of the verity server types, including those used for indexing (“index servers”), searching (“servers” and “brokers”), and administration changes (“administration servers”). Servers used for searching can also have a “test search” button to try out searches directly against a collection. Verity logs can also be viewed for each server type. Collections themselves can also be seen and statistics can be displayed.
4.2. Rcvdk
Rcvdk is a search and discovery tool mainly used to search collections. This tool uses the file system directly and only needs the path to the collection. It does not need the collection imported into a Verity K2 configuration and does not need the Verity K2 server processes up and running. A typical session with rcvdk would be something like:
Rcvdk –locale
attach
s
r
Quit
Where in the above:
“attach” specifies that you are attaching to the given collection for subsequent search attempts.
C:/VerityCollections/DaphneStore_Document_20061201105717_117D65E5F3A2468EB77711E30873FB41
If the locale of the collection doesn’t match the locale that rcvdk was started with, an error is displayed showing the locale of the collection. In that case just re-run rcvdk with the correct locale
“s” specifies you are doing a search.
“r” prints the results of the search, which will be the vdkvgwkey of the records found. The vdkvgwkey contains the guid of the document, plus the content element number (99999999 is used for CBR enabled properties which are not content elements).
4.3. Rck2
Rck2 is like rcvdk, except that the collection it searches must be configured within K2, and the K2 services must be up and running. Generally, rcvdk is easier to use than rck2, so rck2 isn’t used much. A typical session for rck2 would enter:
login
c
s
r
quit
Where in the above:
4.4. mkvdk
mkvdk is a tool that does administration functions and indexing. Here’s a command that would create a collection and then index a file into it:
mkvdk -create -collection test –locale english -style style test.txt
This command creates a collection named “test” with style files in directory “style” and locale of “english”, and indexes a file named “test.txt”. Note that this command does not put the vdkvgwkey into the collection with the same value used by the CE software so it may NOT be used to index documents into CE collections, and may only be used to reproduce problems in offline collections not used by CE.
Here’s a command to optimize a collection (and make searches faster):
mkvdk –optimize tuneup
4.5. Testqp
The Verity Documentation contains a description of all the verity operators for searching full text. You can see them in the documentation at http://localhost:9990/verity_docs/index.html, and searching for “operator”. Sometimes, however, why a query is behaving the way it does is not obvious because the query parser is inserting default operators into the query that you don’t expect. There are two query parsers exposed currently by the CONTAINS and FREETEXT constructs in the CE SQL. The CONTAINS keyword uses what Verity calls the “simple” query parser, and “FREETEXT” uses the “freetext” query parser. To see the default operators inserted into a query, enter either:
testqp -parser simple
or
testqp -parser freetext
Then enter the query string that’s passed to Verity and hit enter. Testqp will print out all the default operators inserted. For instance, if the following is entered
x y
the simple parser prints out:
But the freetext parser prints out:
0.40
0.10
0.50
0.75
0.75
Now you can go look up the “many”, “phrase”, “sum”, “stem” and “accrue” operators in the Verity documentation to understand how these queries work.
Note that when using a property name other than * or Content as the first argument in a CONTAINS operator, the property name must be appended with an IN clause. For instance, if I do a search with
CONTAINS(abc, ‘word1 word2’)
then the query string passed to Verity is:
(word1 word2)
4.6. Testing configurations
Note that the tools in this section can be used to test sample configurations instead of configuring an object store and re-indexing it through an Index Job. Plus these tools can be use to build a test collection with just a single document that is misbehaving in either indexing or search, if such a collection needs to be sent off for investigation. For example, to create a collection with the “uni” locale with the standard style files on a typical Windows based server, and index one document, the commands would be:
mkvdk –collection c:/temp/testcol –create –style “c:/Program Files/verity/k2_61/data/stylesets/FileNet_FileSystem_PushAPI” –locale uni testdoc.txt
Where “testdoc.txt” is the file to be indexed.
Then run
rcvdk –locale uni
attach c:/temp/testcol
x
qparser simple
s “cat dog”
r
The above would search the collection for the phrase “cat dog”.
This type of testing is convenient for checking the behavior of changes to the style files, locale, or other options available through the Verity product. These tools also run totally outside of K2, so you don’t have to import style files, start/stop K2, or do any other K2 overhead when you make changes in order to run a new test. Collections create by mkvdk also do not show up in K2, and when you’re done with them you can just delete the directories.
5. Commonly missed points
This section lists some common misconceptions that have cause configuration problems in the past. All points in this section are also documented elsewhere, but are explicitly mentioned in this document since they are easily missed.
5.1. K2 Operating System User vs. K2 Security User
There are two users associated with a Verity configuration. One is the K2 operating system user that the Verity processes run as according to the operating system. These affect what things the Verity processes can do with file systems and other operating system resources. If there is a problem trying to access a file, it’s the operating system user that must be given permission to perform the given task. Note that the operating system user can be determined by looking at the “login user” of the Verity Service (Windows), or by doing a “ps” on the verity process on a Unix machine.
The K2 security user is identified by the name, domain, and password stored in the VerityDomainConfiguration object. This user is used to logon to Verity from tools such as K2 Dashboard and is used by the security checks that Verity does on a collection. This user must be defined in the configured authentication service.
The operating system user is defined in the security provider used by the operating system on the machine hosting the Verity software. The security user is defined in the security provider configured in the Verity Ticket server that does authentication, and that security provider can be hosted on a remote machine or the local machine.
The same concepts of operating system user vs. security user also applies to the CE server. The operating system user is the user the CE server runs as, and this user is given certain privileges to read and write files, and access other operating system resources. The security user is the user specified to the public CE API, and that user determines which documents, folders, etc. that the client can access. Note that a CE server has only one operating system user, but may have many different security users.
5.2. Verity Administration User
Verity has the concept of an “administration user”. Users are designated as administration users via the Verity K2 Dashboard UI, and Verity will maintain a list of these users. The security provider does not store this list.
The first user that is used for a logon when a Verity configuration is secured by the creation and attachment of a ticket server to the administration server is automatically given the status of an administration user. If you wish to logon to Verity K2 Dashboard with any other user, then that user must explicitly be given permission to logon by making that user an administration user.
The installation guidelines also specify to set up an Index Server as being accessible by only an “Authorized Administrator”. This step is taken so that only administrative users can add data to indexes. Note that there is no other way to prevent users from adding to an index. The group names on a collection do not prevent writes – they only prevent reads.
5.3. Collection Security
The group name assigned to a collection under the assign security link affects only reads of that collection. Just setting up this group does not prevent any random user from adding to the collection. To prevent collection additions, the access type of an index server is set to authorized administrator (see prior section).
6. Logs
When problems are encountered with full text indexing, there are two types of logs that must be inspected. First is the CE server log. Note that if a CE server farm is used, multiple logs for servers in that farm will need to be inspected.
The second type of log is the Verity log. Verity maintains a separate log file for each server type. So if there is a problem with searching, see the Verity logs for the Brokers or Servers (used for searching). If there is a problem with collection creation, see the Administration Server log. If there is a problem with indexing, see the Index Server logs.
All Verity logs can be viewed by running the K2 Dashboard tool, clicking on the server of the required type, and then clicking on the “view logs” link.
7. Verity processes
Verity runs as two services in Windows, and a set of processes on Unix. In the event that the Verity subsystem becomes hung, the K2 Dashboard does have facilities to restart various services. But doing a restart of specific services from K2 Dashboard doesn’t restart everything, so the processes themselves may need to be restarted. In Windows, you can go into the Services utility and restart the two Verity services. In Unix, you can use the k2adminstart and k2adminstop scripts.
8. Mapped drives on Windows
On a Windows system, the requirement to write Verity collections to local disks means that you are writing to a path such as D:/collections, and you cannot use UNC path like //servername/collections. So if you want to have a second server that just reads these collections (note that you cannot have a second server that writes the collections, because writing collections over the network will cause corruption problems), you must map a drive on the machine that reads collections to the filesystem on the machine that writes collections. So in the above example, share out the D drive (share it as some name other than D$, since you can't set permissions on D$) as say "DDrive", and then map the D drive on the reader box to //servername/DDrive. Now D:/collections on the reader box references the same file system as the D:/collections on the writer box.
The next issue that must be dealt with is the fact that Verity runs as a windows service, and windows services by default do not mount mapped drives. One solution to this problem is to not run the Verity Administration service as a windows service, and run it from the command line instead. The command is: "C:/Program Files/verity/k2_61/k2/_nti40/bin/k2admin.exe" -cfg "C:/Program Files/verity/k2_61/k2/common/verity.cfg". Note that only the Verity Administration service needs to be started this way. The Verity Administration Web Server may be left as a service.
Another solution is to use a tool like srvany.exe (supplied as part of the Windows Resource kit) to run a ".cmd" file that first maps the drives, and then issues the command above to start the k2admin.exe. The command to map drives is: “net use D: //servername/DDrive”, using the above example names where "D" is the drive that has been shared on server "servername" as "DDrive".
There are also third party products available that do the same thing as svrany.exe available which are a bit fancier.
The one drawback to using svrany is that although it will start up the service correctly, it will not shut it down. You must use the Verity "rcadmin" command line tool to stop the service, or else use TaskManager to end all the processes that start with the prefix "k2".
To use rcadmin to shutdown the server, enter:
"rcadmin" to start the program
"login
"adminsignal" to initiate the shutdown
"2" in reponse to the "Type of signal (Shutdown=2,WSRefresh=3,RestartAllServers=4):" prompt.
You may then restart the service with the Windows Services tool.
9. Multiple ticket servers and K2 Dashboard
If you have multiple Verity machines in your configuration, you will want to have each machine run a ticket server, and also have all the brokers and search servers on that machine use the local ticket server. However, when you set up such a configuration, be aware that the K2 Dashboard tool has a feature on each broker and search server called “Test Search”. This feature will not work because it will log on using one and only one ticket server, and then won’t be able to access the search servers and brokers attached to the other ticket servers. This limitation is for K2 Dashboard only, and does not affect the CE servers ability to do searches. To test search as the CE server does, use the “rck2” command line tool provided by Verity.
10. Reconfiguring
Several parameters are setup in the collection configuration, and you cannot change them until after you have removed all the collections. Parameters that fall into this configuration are:
1. CBR locale of the Object Store
2. TemplateType of the Index Area which indicates what Verity style files to use
3. The contents of the Verity style files.
10.1. Changing Template Type or CBR Locale
In order to change the Template Type or the CBR Locale, you must first delete all the collections and remove any references to these collections. To do so, see the section “Disabling full text indexing and deleting indexes”, and do both steps.
After the full text disable and delete index has been completed, you will be allowed to change the CBR Locale or Template Type via FileNet Enterprise Manager.
10.2. Changing Verity style files
The Verity style files are sometimes changed to accommodate certain special configuration requirements. The master copy of the style file can be changed, but it won’t have any effect until everything is re-indexed. See comments in the “Re-indexing everything” section.
The data in the style files of the collections should not be changed. Doing so may corrupt the collection and render it unusable. There are a few rare exceptions to this rule, but in general assume that style files in collections should not be modified.
11. Disabling full text indexing and deleting indexes
To disable full text indexing, set the IsCBREnabled status of all classes to false. That can be done by running the FileNet Enterprise Manager, selecting the root class of the classes you’re interested in, and setting the IsCBREnabled property for that class and all subclasses of it to false. If the root class itself is already disabled, but subclasses are not, hitting “apply” won’t do anything, so if you don’t want to individually disable all the classes, you can temporarily set the root class IsCBREnable = true and then set it to false and also apply that to all subclasses.
Setting IsCBREnabled to false will make all future inserts or updates skip full text indexing, but it does not remove the existing full text index that has been previously created. To delete that information, run a re-index job on the root class after all the classes have IsCBREnabled set to false. The root classes are Document, Annotation, Folder, and Custom Object. The re-index must be run on the root class and not a subclass of it to fully delete the full text index information, including the collections, and free up the disk space, and also allow various reconfiguration operations that check to see if collections already exist. That means you can’t reindex a subclass of the root class, or re-index individual documents, annotations, etc., or reindex a collection, to achieve this effect.
12. Re-indexing everything
To re-index everything, re-index the root class and all subclasses of those classes. The root classes are Document, Annotation, Folder, and Custom Object. Re-indexing a root class will create new collections for all the new data, and will delete the old collections when the re-index is complete. It is more efficient than re-indexing an individual subclass or object, because it does not have to do individual object deletes, since the entire collection will be removed when the re-index is finished.
Re-indexing all the collections is not the same as re-indexing Document and all subclasses of it. Re-indexing the collections will only index what is in the collections. If something did not get indexed initially for some reason, it will not be re-indexed even if you do re-index all collections.
Re-indexing a collection is intended to be used when a collection has become damaged, and the exact contents of the collection need to be re-indexed in a new collection.
13. Errors
This section describes the common errors and ways to solve them. It also describes solutions for errors that may not be readily apparent. It is not intended to be a complete list of all errors, or to document obvious errors that are easily fixed from reading the error message.
13.1. CBR_CANT_LOGON_TO_VERITY
13.1.1. Invalid User Credentials
One example of this error is:
com.filenet.api.exception.EngineRuntimeException: CBR_CANT_LOGON_TO_VERITY: Can't logon to Verity. Admin host: testhost, Admin port: 9,950, Domain: exampledomain, User: VerityUser, Error: com.verity.api.administration.ConfigurationException: Invalid user credentials. (-1226)
This error indicates that one or more of the properties used by the CE server to logon to Verity is incorrect, or else the server is inaccessible. All the properties used to logon are in the log message except the password. These properties are stored in the VerityDomainConfiguration object. The CE EnterpriseManager program can be run to change any invalid property of VerityDomainConfiguration in order to fix this problem.
13.1.2. Out of memory
Another example of this error is:
com.filenet.api.exception.EngineRuntimeException: CBR_CANT_LOGON_TO_VERITY: Can't logon to Verity. Admin host: testhost, Admin port: 9,950, Domain: exampledomain, User: VerityUser, Error: Out of memory (-22)
This error can be caused by configuring the JVM with too little memory. This error also occurs when a verity 6.1.2 client attempts to call a Verity 6.1.1 server. Make sure the server revision is at least as new as the client. To check the verity server version, look in
CurrentInstalledPatch= SP1P3
PreviousInstalledPatch=SP1
The magic decoder table for the verity patch number to release number is as follows:
SP3=6.1.4
SP2P1=6.1.3.1
SP2=6.1.3
SP1P3=6.1.2.3
SP1P2=6.1.2.2
SP1P1=6.1.2.1
SP1=6.1.2
Then matching CSE releases to Verity releases:
CSE 4.0.1-001 = 6.1.2.1
CSE 4.0.1-002 = 6.1.2.3
CSE 4.0.1-003 = 6.1.4
13.2. CBR_CANT_CREATE_COLLECTION
One example of this error is:
com.filenet.api.exception.EngineRuntimeException: CBR_CANT_CREATE_COLLECTION: Can't create Verity collection. Name: DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4. Styleset: FileNet_Filesystem_PushAPI, Collection path: //MyServer/VerityCollections/ObjectStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4, Locale: englishv, Error: Fail to create the index. (-6220).
In this instance the error is “caused by”:
com.verity.api.administration.ConfigurationException: Fail to create the index. (-6220)
One problem that causes the “Error: Fail to create the index (-6220)” chained exception is a file system error that occurs when the Verity server tries to write the collection to the indicated path. When this error occurs, check that the operating system user (NOT the user specified by VerityDomainConfiguration.UserName) that the Verity server is running as has file system access to the collection path. Typically this error is caused by the file permission errors, or the path name being wrong. Note that the operating system user can be determined by looking at the “login user” of the Verity Service (Windows), or by doing a “ps” on the verity process on a Unix machine.
Another potential cause of this error is that the verity.cfg file has not been updated. For each VerityIndexArea.RootDirectoryPath property, an entry must be made in verity.cfg on each Verity machine to allow Verity to access the RootDirectoryPath. This file is typically located in C:/Program Files/verity/k2_61/k2/common (adjust as needed if you installed in a non-default location). For each RootDirectoryPath, enter the following in Verity.cfg:
aliasN=pathN
mappingN=
dirmodeN=rw
where “N” is an integer of the next available entry, and “
If the Verity log for the Administration server is inspected after the CBR_CANT_CREATE_COLLECTION error occurs, a sequence of statements such as the following will be logged:
2006-12-06 13:04:52 Status: Launching ["C:/Program Files/verity/k2_61/k2/_nti40/bin/mkvdk" -collection C:/VerityCollections/DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4 -create -style "C:/Program Files/verity/k2_61/data/stylesets/FileNet_Filesystem_PushAPI" -locale englishv] to create collection [C:/VerityCollections/DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4]
2006-12-06 13:04:52 Status: VCM starting process("C:/Program Files/verity/k2_61/k2/_nti40/bin/mkvdk" -collection C:/VerityCollections/DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4 -create -style "C:/Program Files/verity/k2_61/data/stylesets/FileNet_Filesystem_PushAPI" -locale englishv) Pid(2720)
2006-12-06 13:04:53 Status: Failed to create the collection C:/VerityCollections/DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4.
Note that these statements tell you how to run mkvdk to create the collection (in red above). You can issue this same statement from a command window, and mkvdk will also print out additional information to the command window that will identify the error. For instance, it may print out:
Error E3-0011 (VDK): Error: Cannot find or create directory C:/VerityCollection/DaphneStore_Document_20061206130452_63EDA7B982DC43ACB6BE96CBCD7CCFF4
13.3. File Not Found
There are two types of file not found errors that may present in the CE server log that are related to CBR. The first is:
2007-02-14 08:41:40,937 ERROR [CBRExecutorForVerity_DaphneStore_#22] - Updating indexing request after failure. Object store: MyObjectStoreName, targetId: {1DE7C305-2FD2-4C9C-8D28-4B0C2AF4B7BF}, elem seq#: 2, operation: Create, indexationId: {DB3F7D8A-E3F0-413A-BBA9-4E00CF80C503}, objectId: {683553C5-F15B-4F06-A3F7-145B7683A9B1}, note: File to index not found
The above error indicates that the Verity Index server cannot access a file to be indexed. If you turn on detailed CBR tracing, or if you look in the log of the Verity index server, you will see the actual file name. This error is typically caused by permissions problems on the directories or files to be indexed. The most common mistakes are that the user that the Verity server is running as is misunderstood (see “security user” vs “operating system user” earlier in this document), and the “operating system user” that Verity is running as does not have permission to access the given directory. Another common mistake is that this directory is not remote mounted or visible on the Verity server. Note that the CE server passes file names to the Verity server, and the Verity server must be able to open and read these files. Lastly, this error may also be caused by the file server hosting the file being down, or being inaccessible via the network.
There are two directories that may be problems for this error. One is the directory of the file store, if a file store is being used. The other is the directory in the VerityIndexArea.TempDirectoryPath property. This directory is used if the storage area is not a file store, and the data must be copied to a file temporarily in order to submit it to Verity. Also note that if you don’t define a TempDirectoryPath, the CE server will use the VerityIndexArea.RootDirectory path instead, which is fine only if everything is co-located on the same machine, and there is only one server.
To determine if you do have a permissions problem, enable all access to this file by all users. If that works then concentrate on figuring out the permissions problem. If not then some other error is occurring. If you’re on a UNIX system, be aware that if you run the Verity server as “root”, there are special permissions that need to be specified when the remote file system is NFS mounted. This is a feature of NFS and not related to Verity in particular. Consult your MAN pages.
The second file not found error is:
2008-02-14 08:53:41,265 ERROR [CBRExecutorForVerity_DaphneStore_#22] - Updating indexing request after failure. Object store: DaphneStore, targetId: {1DE7C305-2FD2-4C9C-8D28-4B0C2AF4B7BF}, elem seq#: 2, operation: Create, indexationId: {DB3F7D8A-E3F0-413A-BBA9-4E00CF80C503}, objectId: {683553C5-F15B-4F06-A3F7-145B7683A9B1}, note: File not found: unspecified error during access attempt
ERROR [CBRExecutorForVerity_DaphneStore_#22] - Updating indexing request after failure. Object store: MyObjectStoreName, targetId: {1DE7C305-2FD2-4C9C-8D28-4B0C2AF4B7BF}, elem seq#: 2, operation: Create, indexationId: {DB3F7D8A-E3F0-413A-BBA9-4E00CF80C503}, objectId: {683553C5-F15B-4F06-A3F7-145B7683A9B1}, note: File not found: unspecified error during access attempt
This error indicates that the CE server cannot access the file. Again this error is due to the inability to open the file, but this time it’s the CE server that can’t find the file. The same problems can be the cause of this error, such as permissions problems, mounting problems (on the CE server), or the file server being down.
If you turn on CBR tracing you may also see file not found messages that do not relate to actual errors. These may occur if you have the CBR dispatcher wait interval set to a short duration (less than 60 seconds), because the file is moving from its temporary location to a permanent location, and the verity server was trying to access the file while it’s being moved. These conditions are normal, only happen while CBR activity is ramping up when the system is lightly loaded, and are recovered from during normal system processing. So if you’re diagnosing file not found errors, do not set the CBR dispatcher wait interval lower than 120 seconds or so to avoid these messages.
13.4. Error in write_huge
Verity has a limit of 4GB for a single partition (i.e. file) that it writes out as part of a collection since it uses the 32 bit APIs to do file I/O. Verity also by default tries to write 64,000 documents to a single partition, so it may exceed the 4GB limit. If the limit is exceeded, you will see an error like the following in the index server error log:
Error E0-0248 (Vct): Maximum file size exceeded (> 4000000000 bytes)
Error E0-0447 (Vdb): Error writing vdb data file (wrote 0 of 684167)
Error E0-0605 (Vdb Access): Error in write_huge: 00000207.did,
FWDATA, 137076
To recover from this error, you must cut back on the number of documents per partition. To do so, you need to create a file called “style.plc”, and put it into the style files of the verity collection that is having this problem (note that this is about the only style file change that can be made on an existing collection – don’t try changing other style files). This style file should contain:
$control: 1
policy:
{
mode: Default
{
merge_maxdocs : 32000 # Default: 65520 documents/partition
}
}
$$
You may need to set the limit even lower than the 32000 limit in the example above. After this style file has been added to the collection, run “mkvdk –collection
You may also place the style.plc file into the style file set used for creation of new collections, so that when a new collection gets created it will not have this same problem.
You will also need a release of Verity that is later than Verity 6.1.4 or 6.2.1 to solve this problem.
14. Log Warnings
14.1. Collection(s) not recognized
Sometimes when a query is issued, the query may return no error to the user and no data, but the following warning may be logged:
2006-12-06 13:48:16,426 WARN [main] - Collection(s) not recognized and not searched: CollectionName
This warning occurs under normal conditions when it’s due to a newly created collection, and the existence of that collection hasn’t propagated to all the components of the K2 yet. In this particular case, the warning will eventually go away in a few minutes and the system will resume behaving normally.
But in some cases this error will persist forever until corrective action is taken. Possible causes of this error are:
1. The user specified in VerityDomainConfiguration.UserName does not have permission to access the collection due to the group name (from VerityDomainConfiguration.UserGroup) placed on it. To see if this is the case, run K2 Dashboard, log on as the user specified in VerityDomainConfiguration.UserName, and see if you can query the collection from there. If you can’t, click on the collection, then click on “Assign Security”, and remove the group name from the list of security groups associated with the collection. If the collection now appears in the list of searchable collections when you enter the “Test Search” screen, that means the user you logged on as isn’t a member of that group (or the group name was spelled wrong – see next item).
2. The group specified in VerityDomainConfiguration.GroupName is spelled wrong or does not have a properly formatted domain name. There is no validation of the group name when it’s placed on the collection, so misspelled names or improperly formatted names just result in disallowing access to the collection. Look at the pick list of group names in the assign security window of the Dashboard utility. That may show how the name was misspelled. Also note that groups from the domain (not the local system) must be preceded by the domain name, and must be separated from the domain name by a double backslash (e.g.
3. The search server isn’t attached to the broker. In this case, just like in item #1, the collection will not appear as a searchable collection when you hit test search. Use K2 Dashboard and verify that the broker used by search is attached to at least one server that is attached to the collection.
If neither of these two problems are evident, try searching the collection with rcvdk. If the problem still happens with rcvdk, more detailed information on the error will be displayed.
15. Backdating Configurations
Verity keeps a versioned copy of its configuration information. If you installed in the default location on Windows, Verity puts this information in directory:
C:/Program Files/verity/k2_61/data/host/admin
In this directory are a series of files named admin0.xml to admin9.xml, plus an admin.dat. Each adminN.xml file (where “N” ranges from 0..9) is a complete copy of the configuration at one point in time. The admin.dat file has an integer in it from 0..9 which indicates which admin0..9.xml contains the current configuration. If you wish to rollback the current configuration, you can change the contents of admin.dat.
A classic case of needing to do so is when changing security to specify who can logon to the Verity server, or when initially setting up security and requiring a user and password to logon. Sometimes mistakes are made, and after you’ve changed the server you will be unable to logon. If this condition occurs, change admin.dat to point to a version of the configuration that occurred before you made the change. But note that if you make more than 10 changes, the history of changes won’t go back far enough. So it’s best to create a copy of the current admin.xml file (or just copy the whole directory) prior to making significant or error prone changes.
16. Performance issues
16.1. File permissions
The Verity software writes to two directories in addition to the collections directory:
1. The installation directory
2. The user’s %TEMP% directory.
The user that the Verity server runs as should have write access to these two directories. If not, the software still runs, but runs very slowly.
16.2. Indexing and Optimize times
Verity has a practical limit on the size of a collection. Sometimes just 500,000 documents can be written to a collection, and sometimes 8 million can be written to a collection. The point at which the practical limit is hit depends on the number of unique words in each document. The more unique words per document, the sooner the limit is hit.
The indication to the user that the practical collection size limit has been exceeded is that indexing becomes excessively slow, and there is a large queue of objects to be indexed. To verify that the collection size is indeed the problem, CBR summary tracing should be enabled. A line such as the following will be logged showing the optimize time:
2006-12-07 07:28:41,263 DEBUG [CBRExecutorForVerity_ObjectStore_#11] - Optimized Verity collection: ObjectStore_Document_20061207072751_8BC96D3C7172421CBC573A05C6FFA2CB in 656 msecs
If this time is very high, then you may be approaching the maximum practical size of a collection. You should compare this time to the time for indexing inbetween optimize times, and if the optimize starts to take the bulk of the elapsed time, consider closing out the current collection and start writing to a new one.
To close a collection, use the FileNet Enterprise Manager to set the ResourceStatus of the current collection that’s incurring low performance to CLOSED. That will cause a new collection to be created, and performance will resume at a normal level. Queries on the large collection should still be good, so the system will start running normally as soon as the last optimize on the closed collection is finished. Also, be sure to set the VerityServerConfiguration.MaxObjectsPerCollection to a lower value so that the server will automatically roll over the collection to a new one when the limit on the number of objects in the collection is reached. Note that the VerityServerConfiguration object is one of those configuration objects that is set at the Domain, Site, VirtualServer and Server levels. So be sure to set it at the proper level.
16.3. CBR Locales
Note that there is a CBR Locale, and an operating system locale. Those are two different things. Verity has their own concept of what a locale is, which is really a hybrid, since they have a “uni” locale which handles multiple languages.
The “english” verity locale is the fastest, and the “uni” locale is the slowest. The english locale can run up to five times faster than the “uni” locale. The “englishv” is somewhere in between the other two in terms of performance.
Verity recommends using the uni locale in their documentation, and discourages using english. However, most customers don’t need the extra features of the uni locale. So take a look at the verity documentation and see where they compare locale features, and determine if the customer needs uni or can get by with english or englishv.
If you want to change a locale, you must first get all collections deleted in order to do so. See the section on “Reconfiguring” on how to make this happen.
16.4. CPU usage on the Verity server too low
If the CE server is running as fast as possible with 8 or more threads, but the Verity server still shows a low CPU usage, and performance is sub-par, there are two things to check.
First is the number of “Asynchronous Threads” configured for the Verity index server. Go into K2 Dashboard, click on the “Index Servers” link, then click on the desired index server, and then on the “properties” link. Then click on the “ODK” tab, and you’ll see the number of “Asynchronous Threads” configured. That number is the maximum number of threads that will be doing indexing on this index server. If that number is less than the number of CBR threads configured for indexing on the CE server, the CE server threads will be idle waiting for a verity index server thread.
The maximum number of Asynchronous Threads to configure is between 1 and 2 for each CPU on the Verity machine. But use fewer is other functions also run on that machine.
Also note that indexing performance is highly dependent on disk speed and network speed. If storage areas that contain documents to index are slow, or are separated from the Verity server by a slow network, you will bottleneck on file I/O, and will not be able to achieve high CPU utilization on the Verity server.
16.5. Retrieval slow during high document ingestion
If the Verity machine has a high indexing load, it will not be as responsive to searches, if search is also on the same machine. So you should also consider search performance when configuring the number of indexing threads. You can also move search off to a separate verity machine if you need to be get better search performance during heavy indexing load. But there will be some blocking on locked files of the index, so you may also need to reduce the number of indexing threads if search performance is too impacted by indexing load.
16.6. CBR batch size and lease duration
The CBR “batch size” is how many objects are submitted to verity at a time for a single call. Status is not logged and requests are not marked as done until an entire batch is completed. Higher batch sizes generate better performance. Lower batch sizes update statuses more frequently and allow an observer to figure out the progress of indexing in shorter intervals.
The “lease duration” is the maximum amount of time a batch should take to execute. If this time is exceeded by the execution of a batch, index requests will get executed multiple times simultaneously, which will result in many errors in your error log. You will also get a warning if a batch takes longer than ½ the lease time to let you avoid this condition.
Both the “lease duration” and the CBR batch size can be viewed or modify by the FileNet Enterprise Manager, on the Verity Server configuration tab. Note that this tab is at 4 levels, domain, site, virtual server, and server, and the lowest (last in this list) level defined is the effective level.
16.7. High CPU usage on security server
If you notice a high CPU load on the server which handles security checks for the Verity ticket server you have configured, you can set the environment variable:
VERITY_CACHE_USER_GROUPS = 1
Setting this environment variable will cache the security group placed on the Verity collections, and prevent the Verity server from accessing the security provider each time a search is done.
16.8. Stop word files
You can put a file named “style.stp” into the style files directory (the default is FileNet_FileSystem_PushAPI), where this file contains a list of words that are to be omitted from the full text indexes being built. These words also cannot be searched on. The format of the file is to put one word per line. Using stop word file can significantly reduce the collection size and increase performance.
The stop word file is case sensitive, so both a lower case, upper case, and capitialized version of the word should be included, plus any other combinations of upper/lower case in which the word may be found. Additionally, for the “uni” locales (“uni”, “uni/en”, “uni/pt”, etc.), the stop word list must contain a upper case version of each word, even if no upper case version of the word is not in the files to be indexed, because the Verity software puts an uppercase version of the word into the collection for word stemming purposes unless you do so, thus effectively eliminating the advantage of the stop word file.
For “uni” locales, the style.prm file in the style files should also be updated to take full advantage of the stop word list. The following line of this file:
$define WORD-IDXOPTS "Stemdex Casedex"
Should be changed to:
$define WORD-IDXOPTS "Stemdex Casedex StopLangdex"
See the Verity documentation for a complete description of stop word files.
17. No indexing activity
An index request record is generated for each document or object that is to be full text indexed. This record is stored in the IndexRequest table in the database. If you save a document or other object to the server which is for a class that is full text enabled (i.e. the IsCBREnabled flag of the class is “true”, and the IsCBREnabled flag of some of the properties may also be true), an index request will be created. Likewise if you run an index job and select documents or other objects which are full text enabled, it will also generate index requests.
To see if index requests are being generated, you can look in the IndexRequest table in the database for an object store. There is one record always there used for locking purposes, and it has a bunch of zeroes in it and a column named “operation” which has a value of “3”. If you have requests other than the lock record, then those are for documents or other objects to be full text indexed. These requests should be processed by the server, but won’t be picked up until a couple minutes has elapsed, because the server waits a little bit to try and batch up requests for better performance.
If you have requests that are not getting processed, first check the error logs to see if there are errors present. Requests that fail get set aside for a little while before they are retried. Similarly, if a verity server is not accessible, requests to that server will be discontinued for a little while before the CE server will try and access that verity server again.
So check the error log, and if errors are present, fix the error conditions and the server should retry the operations in a few minutes (if you don’t reboot – see “Requests Locked”), and if the error condition has been fixed, indexing will resume normally. See the prior section for descriptions of the common errors.
The remainder of this section goes into the different things that should be checked if you feel that no indexing activity is occurring, and you have no errors.
17.1. CBR dispatcher enabled
Check that the CBR dispatcher is enabled. The CBR dispatcher enabled status is in the Verity Server properties that is on the domain, site, virtual server, and server levels. Setting the value at a lower level (server is the lowest, domain the highest) overrides a setting at a higher level. Check each level for the server being investigated, and make sure that the setting at the lowest level is enabled.
In 4.0, there was a last minute interim fix put in due to a Verity limitation. If you have multiple CE servers per site (not per CE server farm, but per site), you must enable the CBR dispatcher at one and only one CE server in the site. That means you need to disable the CBR dispatcher at the domain level (or site and virtual server level, if defined), and then enable it on the server level for one server for each site. Failure to do this will result in no CBR dispatchers running, because the system doesn’t know which server to pick to run the CBR dispatcher. Each enabled server will also log an error in the error log if this is the reason why CBR dispatchers are not running.
If the server with the CBR dispatcher enabled goes down, you will also need to disable the CBR dispatcher on that server and enable it on some other running server.
The 4.5 and later releases will be enhanced to automatically run the CBR dispatcher on one of the enabled servers, and automatically fail over to another available server if the one currently handling dispatching for the site fails.
17.2. Requests locked
If you reboot the server while indexing is in progress, the requests that are in progress are marked as locked via the “status” column of the “IndexRequest” table. A request is locked if the value of this column is “1”, and unlocked if the value is “0”. By default, the server will timeout these requests after 4 hours and re-execute them.
The timeout is large because requests get executed in batches, and bigger batches are better for performance. The typical batches on a heavily loaded system should be taking ½ hour to an hour for best throughput. The 4 hour default on the timeout is just to make sure that the server doesn’t try and re-execute requests of a batch if they really are in progress. Note that requests from CE server #1 will be re-executed by CE server #2 if server #1 goes down. So it’s not just a reboot that causes this problem to occur.
You can also lower the timeout and/or the batch size if desired. Those parameters are in the Verity Server configuration tab of EM, at the four different levels (domain, site, virtual server, and server). But don’t set it too low, because that would cause requests to be executed redundantly and simultaneously as mentioned in the performance section, and that will results in lots of errors in your error log for unexpected conditions.
If requests are locked, there is also a chance that they will hold up other unlocked requests. The queue for an object store is a semi-FIFO queue, but in some cases a locked request will hold up an unlocked request, such as a locked request for the same document which has another unlocked request for it due to an update, or running of an index job.
17.3. Index Job restart
The ability of an index job to restart after a system reboot was not implemented until fix pack 4 of the 4.0 release. So if you rebooted the CE server while an index job was in progress on a release older than fix pack 4, the index job will process out the remaining index requests that have already been queued up, but will not issue any new requests, and will not go to the terminated state. Index jobs in this state should be deleted and then the same function re-executed via the user interface.
For releases of fix pack 4 and later, an index job will restart at the position it left off after the CE server is rebooted, and guarantee that all objects requested to be indexed will be indexed. The index job will wait, however, for any locked index requests to finish or time out before issuing any significant number of new requests. The default request timeout is 4 hours, as mentioned earlier in the “Requests locked” section. You can disable the CBR dispatcher and wait for the in progress (i.e. locked) index requests to finish prior to rebooting the server to avoid this timeout, if desired.
17.4. Verity Index Server hung
If you think that the CE server has dispatched the requests but the Verity server is really hung, you should first verify that the CPU level on both the CE server (if otherwise inactive) is zero, and the CPU level on the verity server is also zero. Then go into the Verity K2 Dashboard application, click on the “Index Servers” link, and then click on the link for the index server you expect to be processing work. The right hand side of the screen will show “Workspaces on this K2 Index Server”. If none are listed, then the index server is not attempting to do any indexing. But if there are workspaces listed, they will also list how long they have been attempting to complete their task, and what the percentage of completion is. If the time is much longer than the expected time of a batch, that would indicate that they are either hung, or the workspace has been abandoned because the CE server was unable to close it. Either case indicates an error which prevents the CE server from communicating with the Verity server.
Another way to determine that the Verity Index server is hung is to go to your documentation for your application server, and find out how to dump the stack traces of all the threads in the java application. If you see the CBRExecutorForVerity thread stack traces, and they are hung in routines that look like they are submitting requests to Verity (the stack trace will have com.verity… package names on top), then you should submit the stack traces with a DTS, and then reboot the Verity server(s) to clear the condition. We have seen this problem prior to Verity release 6.1.4, but have not reproduced it on 6.1.4 or later verity releases. It also only occurs under very heavy load.
If the Verity server is hung, you can stop and restart the verity services via the verity startup/shutdown command (this is the Services program in Windows, or the k2adminstop script on UNIX). Restarting an individual Verity index (or search) server via the K2 Dashboard start/stop functions has not been reliable in these instances to get the system functioning again.
Additionally, sometimes on either UNIX or Windows, you have to kill off some or all of the Verity processes (kill -9 on UNIX or TaskManager/end process on Windows) in order to get the restart to work. Try killing off any process which shows CPU usage first and then retry the stop verity command. Kill off all processes with “k2” in them if that doesn’t work, and then the stop command will be successful.
The CE server will recover with no loss of data if the Verity software is restarted, but it will log several errors in the CE log while attempting to reconnect to the Verity server. It will also not try and re-execute the in progress requests until the lease timeout has expired.
18. Search returns no results or unexpected results
This section goes through a check list to figure out why content based searches are not working.
18.1. Verify CBR is enabled
Check that the IsCBREnabled property is enabled for the class. This property may be enabled for Document, but may be disabled for a subclass of Document. Also be sure to CBR enable the property definition of any string property you want to have full text indexed. Note that this flag used to be on ColumnDefinition, but is now on the PropertyDefinitionString.
18.2. Check if IndexRequests are pending
Inspect the database table IndexRequest, and see if there are any records present beside the lock record (the lock record has all zeroes in it and is used for database locking). If no requests are present, that means either the requests have completed, or CBR was not enabled properly. The requests will be present in this table for at least a minute, so you should be able to see requests in this table immediately after an update was done to a CBR enabled object.
If requests are present and the retry_count column contains a value greater than zero, then an attempt was made to full text index the object, but it failed for some reason, and the failure cause should be in the error log.
18.3. Check CE server log
If a Document or other object has been written and after a reasonable delay for the background process to finish the full text index, a search still does not return any data, check the CE server log for any errors or warnings. Several error and warning messages that can prevent full text indexing from completing are described elsewhere in this document.
18.4. Check that the CBR Dispatcher is enabled
The CBR Dispatcher can be turned off via Enterprise Manager. Verify that the DispatcherEnabled property of the VerityServerConfiguration object is true, and that the DispatcherWaitInterval is not set to an excessively high number. Also note that the VerityServerConfiguration object is at the Domain, Site, Virtual Server, and Server Instance levels, and can be set at each of these levels. Make sure you look at the lowest level to insure these properties are set correctly. Note that they are set such that the CBR dispatcher is enabled by default.
18.5. Check Verity Collection Document Count
Run the K2 Dashboard tool (“http://
Note that the “document” in the collection is a Verity document, which is different than a “document” in CE terminology. It takes one Verity document to save each ContentElement (i.e. file) of a CE document, plus one more for all the CBR enabled string properties.
18.6. Query Verity Collection
K2 Dashboard can also be used to query the verity collection directly. Click on the broker that is attached to your collection (if the Verity box is being used only for CE functions it will be the only broker configured), and check just the collection(s) you wish to search. Use the object store name and Document, Folder, etc. in the collection name to select the collection for the correct object store and object type (Document, Folder, etc.). Note that if no collections show up for searching, then either the collection was not created (full text indexing not enabled or errors in the error log), the collection is not accessible (security error, an error message should be in the log that’s describe elsewhere in the document), or a configuration error is present (search server not attached to broker (an error message should be in the log described elsewhere in this document).
Provided you can select the proper collection for searching, enter the query string passed in as the second argument to the CONTAINS or FREETEXT clause of your search string. If you searched everything (“*” for the first argument), the second argument is all you need. If you had a CBR enabled string property as the first argument (i.e. “CONTAINS (someprop, ‘xxx’)”), then suffix the query with “
You can also use the same search string with the rcvdk utilities if desired. Rcvdk is actually the more exact tool to use if you are using complex queries, because it is the only tool that uses the same parsers as the CE server. If you are using a CONTAINS clause in the CE SQL statement that is the “simple” parser. If you are using the FREETEXT clause in the CE SQL statement that is the “freetext” parser.
To run rcvdk, enter:
rcvdk –locale
attach
x
qparser
s
r
Where:
Rcvdk is also documented in the verity documentation.
18.7. Check object in Verity Collection
If you have the guid of the object that should have been full text indexed, see if an object is in the Verity collection for this guid. To see if a document or annotation with element sequence number
VdkVgwKey
To search for CBR enabled string properties on documents, annotations, custom objects, or folders, use the same string as above, except put in an element sequence number of 999999999.
Alternatively, you can search for just the IZ/guid/Sequence# string (IZ0123456701230123012301234567890AX7) with any query parser, but that will also find that string within any document, in addition to it being the key for the document. Normally that also works due to the uniqueness of the string.
18.8. Use testqp
Use the testqp (test query parser) utility (see earlier in this document under the Tools section) to verify that the query is being parsed as you expected it to.
18.9. Check Security
You may have created and full text indexed the object successfully, but may not have access to retrieve it from the CE server. To determine if this condition is occurring, turn on detailed tracing for “Search”, then do a search request. If the object is not returned because you do not have access to it, a message will be written to the trace log indicating an object was skipped due to security restrictions.
19. CBR tracing
CBR summary and/or detailed tracing can also be turned on to determine what the CBR subsystem is doing. A trace message will get logged each time the CBR dispatcher queries for objects, and each time it tries to full text index objects, and each time a full text search is done.
CBR tracing is one of those options that is specified at the 4 different levels: Domain, Site, Virtual Server, and Server Instance. Be sure to right click on the appropriate object when running FileNet Enterprise Manager when setting up tracing, with the items later in the list above taking precedence over items earlier in the list, provided you are looking at that site, server, etc.
CBR summary tracing is usually all that is needed to resolve CBR issues. It prints out one line per request made to verity, which won’t be a tremendous amount of information, but can get somewhat large if lots of searches are done.
CBR summary tracing can be used to determine if index requests are being processed, how long they are taking, and if full text searches are being issued, and how long the full text part of the search is taking, separate from the relational search. You can also turn on detailed Search tracing to see how long the entire search, including both relational database and full text part, take.
CBR detailed tracing will print out one line per document indexed or retrieved. That will produce a very large amount of output, and should not be enabled on a server under heavy load. This mode of tracing can be used to determine what specific documents are being indexed or found via a search, without requiring the application to log that data. Note that the relational part of a search may filter out some documents, so it can also be used to determine if a row is being returned by the full text search but then discarded by the relational search.