Kettle
Kettle 已经归属Pentaho门下,4.1版本的下载地址:http://sourceforge.net/projects/pentaho/files/Data%20Integration/4.1.0-stable/
项目中使用软件,我的感觉是不必追求最新版,稳定是优先考虑的。3.2版本也是不错的。使用中感觉kettle总体表现还是不错的,使用方式也都比较好理解。但其中个别的地方还是给初学者制造了些小麻烦。
以下是我的一点经验:
1. 资源库在原有数据库中建立了很多Kettle的表,建议最好不用,就用文件存储
2. 参数的传递问题
Job 定时任务的时候,可以在其所包含的Transformation中,由“获取系统信息”组件定义时间参数,由“表输入”
组件的SQL动态获取。
不过SQL中的参数不能用${param}这种形式获取,而是用?来代替。其中的缘由可以参见官方FAQ里面Argument 和 variables 的区别。
3. UUID的生成
如果你的ID是UUID,需要kettle来生成。就用“脚本”里面的“Modified Java Script Value”组件,代码如下:
//Script here function UUID(){ this.id = this.createUUID(); } UUID.prototype.valueOf = function(){ return this.id; } UUID.prototype.toString = function(){ return this.id; } UUID.prototype.createUUID = function(){ var dg = new Date(1582, 10, 15, 0, 0, 0, 0); var dc = new Date(); var t = dc.getTime() - dg.getTime(); var h = ''; var tl = UUID.getIntegerBits(t,0,31); var tm = UUID.getIntegerBits(t,32,47); var thv = UUID.getIntegerBits(t,48,59) + '1'; var csar = UUID.getIntegerBits(UUID.rand(4095),0,7); var csl = UUID.getIntegerBits(UUID.rand(4095),0,7); var n = UUID.getIntegerBits(UUID.rand(8191),0,7) + UUID.getIntegerBits(UUID.rand(8191),8,15) + UUID.getIntegerBits(UUID.rand(8191),0,7) + UUID.getIntegerBits(UUID.rand(8191),8,15) + UUID.getIntegerBits(UUID.rand(8191),0,15); return tl + h + tm + h + thv + h + csar + csl + h + n; } UUID.getIntegerBits = function(val,start,end){ var base16 = UUID.returnBase(val,16); var quadArray = new Array(); var quadString = ''; var i = 0; for(i=0;i<base16.length;i++){ quadArray.push(base16.substring(i,i+1)); } for(i=Math.floor(start/4);i<=Math.floor(end/4);i++){ if(!quadArray[i] || quadArray[i] == '') quadString += '0'; else quadString += quadArray[i]; } return quadString; } UUID.returnBase = function(number, base){ var convert = ['0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']; if (number < base) var output = convert[number]; else { var MSD = '' + Math.floor(number / base); var LSD = number - MSD*base; if (MSD >= base) var output = this.returnBase(MSD,base) + convert[LSD]; else var output = convert[MSD] + convert[LSD]; } return output; } UUID.rand = function(max){ return Math.floor(Math.random() * max); } var ID=new UUID().id;
4. Java 执行Transportation
@Test public void executeTrans() throws KettleException { try { StepLoader.init(); EnvUtil.environmentInit(); TransMeta transMeta = new TransMeta("d:/test.ktr"); Trans trans = new Trans(transMeta); trans.execute(null); // Pass arguments instead of null. trans.waitUntilFinished(); if (trans.getErrors() > 0) { throw new RuntimeException("There were errors during transformation execution."); } } catch (KettleException e) { System.out.println(e); } }
5. Java 执行Job
@Test public void executeTrans() throws KettleException { EnvUtil.environmentInit(); JobEntryLoader.init(); StepLoader.init(); LogWriter log = LogWriter.getInstance("TransTest.log", true, LogWriter.LOG_LEVEL_DETAILED); JobMeta jobMeta = new JobMeta(log, "d:/testjob.kjb", null); Job job = new Job(log, StepLoader.getInstance(), null, jobMeta); jobMeta.setInternalKettleVariables(job); job.execute(); job.waitUntilFinished(); }