kettle-java代码执行hive相关ktr时报错： database type with plugin id [HIVE2] couldn't be found!

kettle-java代码执行hive相关ktr时报错： database type with plugin id [HIVE2] couldn't be found!

转

2018年08月13日 16:47:30 lisery1993 阅读数：305

1.在java项目中执行hive相关的ktr:

1.  

2. KettleEnvironment.init();

3.  

4. TransMeta transMeta = new TransMeta("C:\\Users\\lixiang\\Desktop\\hivetorizhi.ktr");

5. // 设置执行模式

6. TransExecutionConfiguration config = new TransExecutionConfiguration();

7. config.setExecutingClustered(true);

8. config.setExecutingLocally(false);

9. config.setExecutingRemotely(false);

10. config.setClusterPosting(true);

11. config.setClusterPreparing(true);

12. config.setClusterStarting(true);

13. TransSplitter transSplitter = Trans.executeClustered(transMeta, config);

14. System.out.println(transSplitter.getCarteObjectMap());

15. System.out.println(transSplitter.getMaster());

16. System.out.println(transSplitter.getSlaves()[0]);

17. System.out.println(transSplitter.getSlaves()[1].getStepNames()[0]);

18. }

2.报错：

1. Exception in thread "main" org.pentaho.di.core.exception.KettleXMLException:

2. 错误从XML文件读取转换

3.  

4. 错误从XML文件读取转换

5.  

6. Unable to load database connection info from XML node

7.  

8. Unable to create new database interface

9.  

10. database type with plugin id [HIVE2] couldn't be found!

11.  

12. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3402)

13. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2758)

14. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2710)

15. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2687)

16. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2667)

17. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2632)

18. at org.pentaho.di.trans.TransMeta.(TransMeta.java:2595)

19. at com.kettle.KettleExamle.getRemoteServer(KettleExamle.java:55)

20. at com.kettle.KettleExamle.main(KettleExamle.java:140)

21. Caused by: org.pentaho.di.core.exception.KettleXMLException:

22. 错误从XML文件读取转换

23.  

24. Unable to load database connection info from XML node

25.  

26. Unable to create new database interface

27.  

28. database type with plugin id [HIVE2] couldn't be found!

29.  

30. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3384)

31. ... 8 more

32. Caused by: org.pentaho.di.core.exception.KettleXMLException:

33. Unable to load database connection info from XML node

34.  

35. Unable to create new database interface

36.  

37. database type with plugin id [HIVE2] couldn't be found!

38.  

39. at org.pentaho.di.core.database.DatabaseMeta.(DatabaseMeta.java:1013)

40. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2976)

41. ... 8 more

42. Caused by: org.pentaho.di.core.exception.KettleXMLException:

43. Unable to create new database interface

44.  

45. database type with plugin id [HIVE2] couldn't be found!

46.  

47. at org.pentaho.di.core.database.DatabaseMeta.(DatabaseMeta.java:977)

48. ... 9 more

49. Caused by: org.pentaho.di.core.exception.KettleDatabaseException:

50. database type with plugin id [HIVE2] couldn't be found!

51.  

52. at org.pentaho.di.core.database.DatabaseMeta.findDatabaseInterface(DatabaseMeta.java:584)

53. at org.pentaho.di.core.database.DatabaseMeta.getDatabaseInterface(DatabaseMeta.java:558)

54. at org.pentaho.di.core.database.DatabaseMeta.(DatabaseMeta.java:974)

55. ... 9 more

3.报错分析：kettle程序在读取database type的过程中读不到hive2这个类型。

查看kettle-core-7.1.0.0-12.jar中的kettle-database-types.xml文件，此文件定义了database type ,其中没hive2.。

2.  

4. AS/400

5. org.pentaho.di.core.database.AS400DatabaseMeta

7.  

9. Intersystems Cache

10. org.pentaho.di.core.database.CacheDatabaseMeta

12.  

14. IBM DB2

15. org.pentaho.di.core.database.DB2DatabaseMeta

17.  

19. dBase III, IV or 5

20. org.pentaho.di.core.database.DbaseDatabaseMeta

22.  

24. Apache Derby

25. org.pentaho.di.core.database.DerbyDatabaseMeta

27.  

29. Exasol 4

30. org.pentaho.di.core.database.Exasol4DatabaseMeta

32.  

34. ExtenDB

35. org.pentaho.di.core.database.ExtenDBDatabaseMeta

37.  

39. Firebird SQL

40. org.pentaho.di.core.database.FirebirdDatabaseMeta

42.  

44. Generic database

45. org.pentaho.di.core.database.GenericDatabaseMeta

47.  

49. Greenplum

50. org.pentaho.di.core.database.GreenplumDatabaseMeta

52.  

54. Calpont InfiniDB

55. org.pentaho.di.core.database.InfiniDbDatabaseMeta

57.  

59. Gupta SQL Base

60. org.pentaho.di.core.database.GuptaDatabaseMeta

62.  

64. H2

65. org.pentaho.di.core.database.H2DatabaseMeta

67.  

69. Hypersonic

70. org.pentaho.di.core.database.HypersonicDatabaseMeta

72.  

74. Infobright

75. org.pentaho.di.core.database.InfobrightDatabaseMeta

77.  

79. Informix

80. org.pentaho.di.core.database.InformixDatabaseMeta

82.  

84. Ingres

85. org.pentaho.di.core.database.IngresDatabaseMeta

87.  

89. Ingres VectorWise

90. org.pentaho.di.core.database.VectorWiseDatabaseMeta

92.  

94. Borland Interbase

95. org.pentaho.di.core.database.InterbaseDatabaseMeta

97.  

99. KingbaseES

100. org.pentaho.di.core.database.KingbaseESDatabaseMeta

102.  

104. LucidDB

105. org.pentaho.di.core.database.LucidDBDatabaseMeta

107.  

109. MariaDB

110. org.pentaho.di.core.database.MariaDBDatabaseMeta

112.  

114. MonetDB

115. org.pentaho.di.core.database.MonetDBDatabaseMeta

117.  

119. MS Access

120. org.pentaho.di.core.database.MSAccessDatabaseMeta

122.  

124. MS SQL Server (Native)

125. org.pentaho.di.core.database.MSSQLServerNativeDatabaseMeta

127.  

129. MS SQL Server

130. org.pentaho.di.core.database.MSSQLServerDatabaseMeta

132.  

134. MySQL

135. org.pentaho.di.core.database.MySQLDatabaseMeta

137.  

139. Native Mondrian

140. org.pentaho.di.core.database.MondrianNativeDatabaseMeta

142.  

144. Neoview

145. org.pentaho.di.core.database.NeoviewDatabaseMeta

147.  

149. Netezza

150. org.pentaho.di.core.database.NetezzaDatabaseMeta

152.  

154. Oracle

155. org.pentaho.di.core.database.OracleDatabaseMeta

157.  

159. Oracle RDB

160. org.pentaho.di.core.database.OracleRDBDatabaseMeta

162.  

164. PostgreSQL

165. org.pentaho.di.core.database.PostgreSQLDatabaseMeta

167.  

169. Redshift

170. org.pentaho.di.core.database.RedshiftDatabaseMeta

172.  

174. Remedy Action Request System

175. org.pentaho.di.core.database.RemedyActionRequestSystemDatabaseMeta

177.  

179. MaxDB (SAP DB)

180. org.pentaho.di.core.database.SAPDBDatabaseMeta

182.  

184. SQLite

185. org.pentaho.di.core.database.SQLiteDatabaseMeta

187.  

189. Sybase

190. org.pentaho.di.core.database.SybaseDatabaseMeta

192.  

194. SybaseIQ

195. org.pentaho.di.core.database.SybaseIQDatabaseMeta

197.  

199. Teradata

200. org.pentaho.di.core.database.TeradataDatabaseMeta

202.  

204. UniVerse database

205. org.pentaho.di.core.database.UniVerseDatabaseMeta

207.  

209. Vertica

210. org.pentaho.di.core.database.VerticaDatabaseMeta

212.  

214. Vertica 5+

215. org.pentaho.di.core.database.Vertica5DatabaseMeta

217.  

4.报错解决：

hive的api的连接方式是通过jdbc连接的。具体思路与postgresql的连接思路相同。因此仿照postgresql的连接方法自定义一个database-type为hive2。因此，在kettle-core-7.1.0.0-12.jar的kettle-database-types.xml文件中添加

2. HIVE2

3. org.pentaho.di.core.database.Hive2SQLDatabaseMeta

其中会有对应的实现类org.pentaho.di.core.database.Hive2SQLDatabaseMeta。

仿照postgresql的实现类org.pentaho.di.core.database.PostgreSQLDatabaseMeta来写即可，只需将连接的url以及jdbc driver名称更换为hive相关。

将org.pentaho.di.core.database.Hive2SQLDatabaseMeta类编译为.class文件放进kettle-core-7.1.0.0-12.jar的org\pentaho\di\core\database路径下

6.再次执行ktr：

执行成功：结果：

1. {/hivetorizhi (master)=1cc5f5bb-b0c4-4938-96f7-85363f88564d, /hivetorizhi (local:slave1-8081)=09ea3633-f5bd-4e43-9352-62b958255469, /hivetorizhi (local:slave2-8082)=35e00dc2-f570-4f26-8540-156fb84d9524}

2. /hivetorizhi (master)

3. /hivetorizhi (local:slave2-8082)

4.