KETTLE_内存溢出错误
原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。
深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831
kettle内存溢出错误解决
环境：
源端数据库：oracle 10G R2
目标端数据库：oracle 11G R2
kettle版本：5.0.1-stable
报错：
抽取大数据量表时，报错，日志信息如下：
2015/01/05 11:27:42 - Spoon - 转换已经打开. 2015/01/05 11:27:42 - Spoon - 正在打开转换 [TABLE_BWWN]... 2015/01/05 11:27:42 - Spoon - 开始执行转换. 2015/01/05 11:27:42 - TABLE_BWWN - 为了转换解除补丁开始 [TABLE_BWWN] 2015/01/05 11:27:44 - max.0 - Finished reading query, closing connection. 2015/01/05 11:27:44 - max.0 - 完成处理 (I=1, O=0, R=0, W=1, U=0, E=0 2015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : UnexpectedError: 2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 一个未预期的错误发生在Spoon: probable cause:在停止Spoon前，请先关闭其它spoon窗口! 2015/01/05 12:25:08 - Spoon - Java heap space 2015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space 2015/01/05 12:25:08 - 插入 / 更新.0 - at oracle.jdbc.driver.OraclePreparedStatement.bindBytesAsStream(OraclePreparedStatement.java:6713) 2015/01/05 12:25:08 - 插入 / 更新.0 - at oracle.jdbc.driver.OraclePreparedStatement.setBytesInternal(OraclePreparedStatement.java:6700) 2015/01/05 12:25:08 - 插入 / 更新.0 - at oracle.jdbc.driver.OraclePreparedStatement.setBytes(OraclePreparedStatement.java:6660) 2015/01/05 12:25:08 - 插入 / 更新.0 - at oracle.jdbc.driver.OraclePreparedStatementWrapper.setBytes(OraclePreparedStatementWrapper.java:150) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.core.row.value.ValueMetaBase.setPreparedStatementValue(ValueMetaBase.java:4704) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.core.database.Database.setValue(Database.java:928) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.core.database.Database.setValues(Database.java:943) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.core.database.Database.setValuesInsert(Database.java:888) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:113) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:301) 2015/01/05 12:25:08 - 插入 / 更新.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60) 2015/01/05 12:25:08 - 插入 / 更新.0 - at java.lang.Thread.run(Unknown Source) 2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space 2015/01/05 12:25:08 - Spoon - at java.util.Arrays.copyOf(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.StringCoding.safeTrim(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.StringCoding.access$100(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.StringCoding$StringDecoder.decode(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.StringCoding.decode(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.StringCoding.decode(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.String.<init>(Unknown Source) 2015/01/05 12:25:08 - Spoon - at java.lang.String.<init>(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.core.row.value.ValueMetaBase.convertBinaryStringToString(ValueMetaBase.java:1105) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.core.row.value.ValueMetaBase.getString(ValueMetaBase.java:1452) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.core.row.RowMeta.getString(RowMeta.java:230) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.showPreviewGrid(TransPreviewDelegate.java:302) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.refreshView(TransPreviewDelegate.java:268) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate$1.widgetSelected(TransPreviewDelegate.java:188) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.custom.CTabFolder.setSelection(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.custom.CTabFolder.onMouse(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.custom.CTabFolder$1.handleEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1227) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7368) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8673) 2015/01/05 12:25:08 - Spoon - at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:625) 2015/01/05 12:25:08 - Spoon - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2015/01/05 12:25:09 - 插入 / 更新.0 - 完成处理 (I=20575, O=20574, R=20575, W=20574, U=0, E=1 2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN 2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN 2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Unexpected error 2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException: 2015/01/05 12:25:09 - 表输入.0 - Couldn't get row from result set 2015/01/05 12:25:09 - 表输入.0 - ORA-03111: 通信通道收到中断 2015/01/05 12:25:09 - 表输入.0 - 2015/01/05 12:25:09 - 表输入.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2302) 2015/01/05 12:25:09 - 表输入.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2270) 2015/01/05 12:25:09 - 表输入.0 - at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:153) 2015/01/05 12:25:09 - 表输入.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60) 2015/01/05 12:25:09 - 表输入.0 - at java.lang.Thread.run(Unknown Source) 2015/01/05 12:25:09 - 表输入.0 - Caused by: java.sql.SQLException: ORA-03111: 通信通道收到中断 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1082) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:369) 2015/01/05 12:25:09 - 表输入.0 - at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:273) 2015/01/05 12:25:09 - 表输入.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2290) 2015/01/05 12:25:09 - 表输入.0 - ... 4 more 2015/01/05 12:25:09 - 表输入.0 - Finished reading query, closing connection. 2015/01/05 12:25:09 - 表输入.0 - 完成处理 (I=20580, O=0, R=1, W=20579, U=0, E=1 2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到! 2015/01/05 12:25:09 - Spoon - 转换完成!! 2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到! 2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到! 2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN 2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN
解决方法：
找到kettle目录下的Spoon.bat文件，找到如下部分：
REM ****************************************************************** REM ** Set java runtime options ** REM ** Change 512m to higher values in case you run out of memory ** REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable ** REM ****************************************************************** if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx512m" "-XX:MaxPermSize=256m" set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"
修改参数，将内存值调大（256的倍数），调整后如下所示：
REM ****************************************************************** REM ** Set java runtime options ** REM ** Change 512m to higher values in case you run out of memory ** REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable ** REM ****************************************************************** if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx1024m" "-XX:MaxPermSize=2048m" set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"
重新执行kettle抽取数据，不再报内存溢出错误。
小结：
使用kettle时，如果遇到数据量大、大对象时很容易出现内存溢出现象，此时需要根据实际的硬件配置对预设的java堆栈大小进行调整。
原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。
深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831