ETL工具kettle之Java脚本+Elasticsearch bulk insert

ETL工具kettle之Java脚本+Elasticsearch bulk insert

本文在这里记录一下使用ETL工具Kettle将数据批量插入到Elasticsearch中,由于需要数据类型和日
期格式的转换此处需要引入java脚本.

编写住转换脚本

如图所示:

• 第一个表输入,就是直接查询表中所有字段返回;

SELECT
  id
, shop
, productsaleid
, digest
, text
, type
, createtime
, ts
FROM shop_item_text limit 100;

• 编写java脚本,如果此处需要依赖其他jar,需要把依赖的jar copy到kettle的lib目录下然后重启,并且在编写java脚本时要用类的全类名,要不然找不到类;

public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException {
  if (first) {
    first = false;

    /* TODO: Your code here. (Using info fields)

    FieldHelper infoField = get(Fields.Info, "info_field_name");

    RowSet infoStream = findInfoRowSet("info_stream_tag");

    Object[] infoRow = null;

    int infoRowCount = 0;

    // Read all rows from info step before calling getRow() method, which returns first row from any
    // input rowset. As rowMeta for info and input steps varies getRow() can lead to errors.
    while((infoRow = getRowFrom(infoStream)) != null){

      // do something with info data
      infoRowCount++;
    }
    */
  }

  Object[] r = getRow();

  if (r == null) {
    setOutputDone();
    return false;
  }

  // It is always safest to call createOutputRow() to ensure that your output row's Object[] is large
  // enough to handle any new fields you are creating in this step.
  r = createOutputRow(r, data.outputRowMeta.size());

  /* TODO: Your code here. (See Sample)

  // Get the value from an input field
  String foobar = get(Fields.In, "a_fieldname").getString(r);

  foobar += "bar";
    
  // Set a value in a new output field
  get(Fields.Out, "output_fieldname").setValue(r, foobar);
  */
  /************此处以下是自己写的代码**************/
  try{
	byte[] textBytes = get(Fields.In, "text").getBinary(r);
  	String text = org.xerial.snappy.Snappy.uncompressString(textBytes);
	get(Fields.Out, "text_str").setValue(r, text);
    java.text.SimpleDateFormat simpleDateFormat = new java.text.SimpleDateFormat("yyyy-MM-dd\'T\'HH:mm:ss.SSSZ");
    simpleDateFormat.setTimeZone(java.util.TimeZone.getTimeZone("UTC"));
	java.sql.Timestamp createtime = get(Fields.Out, "createtime").getTimestamp(r);
    String createtime_str = simpleDateFormat.format(createtime);
	get(Fields.Out, "createtime_str").setValue(r, createtime_str);
	java.sql.Timestamp ts = get(Fields.Out, "ts").getTimestamp(r);
	String ts_str = simpleDateFormat.format(ts);
	get(Fields.Out, "ts_str").setValue(r, ts_str);
  }catch(Exception e){
  	e.printStackTrace();
	logError(e.getMessage(), e);
  }
  /************此处以上是自己写的代码**************/
  // Send the row on to the next step.
  putRow(data.outputRowMeta, r);
  return true;
}

此处java代码主要是格式化了时间字段,然后又将格式化后的时间字符串设置入一个新的字段,解压一个byte[]为String,然后也设置入一个新字段;此处需要在控件中添加新字段以便于下一个控件"Elasticsearch bulk insert"使用.新加字段如图:

• 编写Elasticsearch bulk insert
  
  
  
  
  
• 最后就可以开始执行转换了
  

此处做一个增量迁移数据的补充:
项目要求增量将后续MySQL中变化的数据实时同步到ES中,此处可以使用阿里开源的工具canal: https://github.com/alibaba/canal 由于数据类型需要做一下转换,所以我自己开发了一个监听canal server的client,代码托管在GitHub:
https://github.com/canglang1973/canal_elasticsearch_client 有兴趣大家可以一起学习探讨.