Unable to Rebuild JIRA Index

Symptoms

Accessing certain JIRA pages result in:

SEVERE: Internal server error
com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.util.concurrent.LazyReference$InitializationException: com.atlassian.jira.util.RuntimeIOException: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _ow2: fieldsReader shows 1 but segmentInfo shows 50
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:85)
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:76)
at com.atlassian.jira.issue.index.SearcherCache.retrieveIssueSearcher(SearcherCache.java:38)

Filters are resulting in:

Caused by: java.io.FileNotFoundException: /local/jira/jira444/home/caches/indexes/entities/searchrequest/_o5.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java: 212 )
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java: 78 )
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java: 108 )

When trying to re-index JIRA, the following error will be thrown:

java.io.IOException: read past EOF
Task completed in 2 seconds with unexpected error.
Started Today 8:29 PM.
Finished Today 8:29 PM.
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: read past EOF at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:145) at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:44) at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:170) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:219) ... at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:249) at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: read past EOF

This error can also be thrown during a re-index:

2012-11-25 16:21:36,897 JiraTaskExectionThread-1 ERROR admin 981x203x1 6slf04 10.1.250.116 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED.  Indexer: DefaultIndexManager: paths: [/var/atlassian/application-data/jira/caches/indexes/comments, /var/atlassian/application-data/jira/caches/indexes/issues, /var/atlassian/application-data/jira/caches/indexes/changes]

com.atlassian.jira.util.RuntimeIOException: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null

	at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:146)

	at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:43)

	at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:205)

	at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:824)

	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:335)

	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:301)

Caused by: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null

 

Cause

The re-indexing process fails when JIRA index file(s) is/are corrupted for any reason, for example the JIRA service may be terminated unexpectedly because of an power outage.

Resolution

Icon

A re-index will lock down the JIRA and prevent users from using JIRA. Please keep this in mind when following these steps to resolve this issue - depending on the size of the instance a downtime window may require scheduling. With the release of JIRA 5.2, an option to perform a background re-index was implemented, but it is advised you use the Lock JIRA and perform a re-index in this scenario. See this page for more details:   Search Indexing

1. Stop JIRA.

2. Make a backup of the following directory and all its subdirectories:

   $JIRA_HOME/caches/indexes

3. Remove the following directory and all its subdirectories:

   $JIRA_HOME/caches/indexes

    This will be rebuilt when JIRA is next reindexed.

4. Start JIRA.

5. Perform a re-index of JIRA through the Administration section.