WAS7 EJB OOM问题

在WAS7.0.0.13版本上分布式发布ejb和web，web和ejb不在一个集群，且不在一个server哦情况下，web调用EJB的时候出现OOM错误，报错信息如下：

[11-3-1 3:32:06:132 CST] 00000021 SystemOut     O ADMINISTRATOR:123456:0:1:undefined:192.168.0.103:CrbGu3tv3gf37HzkT5r3BW7::
[11-3-1 3:32:11:331 CST] 00000021 UserManagerDe E com.xx.xxxxxx.web.UserManagerDefaultImpl loginIn java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: 
	>> SERVER (id=4773e3aa, host=zmt400) TRACE START:
	>>    java.lang.OutOfMemoryError
	>>	 at com.ibm.rmi.iiop.CDRReader.readBytesForString(CDRReader.java:2296)
	>>	 at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:489)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_codebase_URL(CDRReader.java:2890)
	>>	 at com.ibm.rmi.iiop.CDRReader.fast_read_value(CDRReader.java:1910)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2017)
	>>	 at com.xx.xxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:206)
	>>	 at com.xxx.xxxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:127)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:475)
	>>	 at com.ibm.rmi.iiop.ORB.process(ORB.java:513)
	>>	 at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
	>>	 at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2841)
	>>	 at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2714)
	>>	 at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
	>>	 at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
	>>	 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
	>> SERVER (id=4773e3aa, host=zmt400) TRACE END.

[11-3-1 3:32:11:341 CST] 00000021 BaseServer    E com.xx.xxxxx.web.BaseServer processLogin 登录错误
                                 java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: 
	>> SERVER (id=4773e3aa, host=zmt400) TRACE START:
	>>    java.lang.OutOfMemoryError
	>>	 at com.ibm.rmi.iiop.CDRReader.readBytesForString(CDRReader.java:2296)
	>>	 at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:489)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_codebase_URL(CDRReader.java:2890)
	>>	 at com.ibm.rmi.iiop.CDRReader.fast_read_value(CDRReader.java:1910)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2017)
	>>	 at com.xx.xxxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:206)
	>>	 at com.xx.xxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:127)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:475)
	>>	 at com.ibm.rmi.iiop.ORB.process(ORB.java:513)
	>>	 at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
	>>	 at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2841)
	>>	 at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2714)
	>>	 at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
	>>	 at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
	>>	 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
	>> SERVER (id=4773e3aa, host=zmt400) TRACE END.

 

 

当然第一反应是应用写的有问题，ejb里面有内存泄露或者threadpool开太大，后来检查未发现异常。

 

收集CORE文件和HEAPDUMP文件分析，上PP。

 

core文件分析如下：

 

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received 
......
......
......
1XMCURTHDINFO  Current Thread Details
NULL           ----------------------
3XMTHREADINFO      "ORB.thread.pool : 0" TID:0x0000000001ECF600, j9thread_t:0x0000000011EA3360, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x5C7C, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.readBytesForString(CDRReader.java:2279)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.readStringOrIndirection(CDRReader.java:472)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.read_codebase_URL(CDRReader.java:2852)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.fast_read_value(CDRReader.java:1893)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.read_value(CDRReader.java:2000)
4XESTACKTRACE          at com/xx/xxxxxx/common/service/ejb/_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:206)
4XESTACKTRACE          at com/xx/xxxxxx/common/service/ejb/_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:127)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ServerDelegate.dispatch(ServerDelegate.java:475)
4XESTACKTRACE          at com/ibm/rmi/iiop/ORB.process(ORB.java:504)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ORB.process(ORB.java:1571)
4XESTACKTRACE          at com/ibm/rmi/iiop/Connection.respondTo(Connection.java:2771)
4XESTACKTRACE          at com/ibm/rmi/iiop/Connection.doWork(Connection.java:2640)
4XESTACKTRACE          at com/ibm/rmi/iiop/WorkUnitImpl.doWork(WorkUnitImpl.java:63)
4XESTACKTRACE          at com/ibm/ejs/oa/pool/PooledThread.run(ThreadPool.java:118)
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1527)

 

 

 heapdump分析如下：

 

 

发现CORE文件里面很奇怪，Free Java Heap 有2,104,902,904，98%空闲，但是在内存段分析中，Object即heap占用2,147,483,648 100%占用，heapdump里面分析占用heap内存42,565,544，和core文件的第一种说法匹配，即98% free，排除应用占用内存的可能。

 

分析了下出错的代码行 com.ibm.rmi.iiop.CDRReader.readBytesForString，发现是由web端调用ejb的时候，从ejb容器传递ejb stud的时候出现问题，好，那就避开这个，我将应用发布到同一个server上，这样ejb应该优化为本地调用，验证成功。

 

那为了解决分布式调用的问题，我还要想一个办法，让系统在分布式情况下也不用传递ejb stud，这让我想到了早期中间件需要手动打stud然后引入web容器的事情，查了下infocenter，找到{WAS_HOME}\bin\ejbdeploy.sh命令。

 

ejbdeploy tytim.ear . tytim_stub.ear

 

会在当前目录产生tytim_stub.ear，将这个ear中的ejb相关jar覆盖到.war里面的lib中，问题解决。

 

已联系IBM技术人员，看看是什么原因导致ejb stud传递的时候byte数组长度出现异常，等有了回复，在补充上来。

 

 

转载请注明原始地址： http://boriszhang78.iteye.com/blog/935439