OutOfMemoryError: unable to create new native thread

1. Problem Description

In our production and staging environment, we occasionally encountered this error:

Exception in thread "default-workqueue-7" java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:691)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Exception in thread "default-workqueue-20" Exception in thread "default-workqueue-10"   at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)

This hurt our business heavily, dev team and sys team got nervous.

Environment: 

Linux 64bit, JDK 7, 2 tomcat instances hosting 2 cometD powered web app, we observed that once the 2 tomcats' total threads get greater than 1000, the error would probably reproduce.

2. Analysis and Solution

Reference [2] seems a great methodology to analyse this issue, but sadly just doesn't work for our case,  anyway, I learned a lot from this awesome material.

When test team conducts load test, I monitored the JVM memory/threads, and OS memory/IO, didn't find anything abnormal.

Our heap size is 2048MB-2048MB, perm size is 128MB-256MB, at peak time, heap usage is 1250MB, perm usage is 75MB, tomcats' total threads number is around 1000. Seems not heavy load at all, weird.

Then I went to reference [1], and found our server's open files limit is 1024, after change to 8000, the error seems gone.

Nevertheless, we did another optimization, enabled Tomcat's NIO connector, so our cometD longpolling threads can be released quickly.

Reference:

[1] http://www.devgrok.com/2012/03/resolving-outofmemoryerror-unable-to.html

[2] http://www.javacodegeeks.com/2012/09/outofmemoryerror-unable-to-create-new.html