solr dataimport 数据导入源码分析（十三）

本文接下来分析EntityProcessor相关类，我们可以称之为实体处理器，针对不同的数据源有不同的实体处理器，屏蔽了不同数据源的差异

本文只介绍针对数据库数据源的实体处理器，其他实体处理器类似

EntityProcessor类为抽象类，定义了获取数据源的Map类型数据的方法（针对添加 修改 删除的数据）

/**

 * <p>

 * An instance of entity processor serves an entity. It is reused throughout the

 * import process.

 * </p>

 * <p/>

 * <p>

 * Implementations of this abstract class must provide a public no-args constructor.

 * </p>

 * <p/>

 * <p>

 * Refer to <a

 * href="http://wiki.apache.org/solr/DataImportHandler">http://wiki.apache.org/solr/DataImportHandler</a>

 * for more details.

 * </p>

 * <p/>

 * <b>This API is experimental and may change in the future.</b>

 *

 * @version $Id: EntityProcessor.java 824359 2009-10-12 14:31:54Z ehatcher $

 * @since solr 1.3

 */

public abstract class EntityProcessor {



  /**

   * This method is called when it starts processing an entity. When it comes

   * back to the entity it is called again. So it can reset anything at that point.

   * For a rootmost entity this is called only once for an ingestion. For sub-entities , this

   * is called multiple once for each row from its parent entity

   *

   * @param context The current context

   */

  public abstract void init(Context context);



  /**

   * This method helps streaming the data for each row . The implementation

   * would fetch as many rows as needed and gives one 'row' at a time. Only this

   * method is used during a full import

   *

   * @return A 'row'.  The 'key' for the map is the column name and the 'value'

   *         is the value of that column. If there are no more rows to be

   *         returned, return 'null'

   */

  public abstract Map<String, Object> nextRow();



  /**

   * This is used for delta-import. It gives the pks of the changed rows in this

   * entity

   *

   * @return the pk vs value of all changed rows

   */

  public abstract Map<String, Object> nextModifiedRowKey();



  /**

   * This is used during delta-import. It gives the primary keys of the rows

   * that are deleted from this entity. If this entity is the root entity, solr

   * document is deleted. If this is a sub-entity, the Solr document is

   * considered as 'changed' and will be recreated

   *

   * @return the pk vs value of all changed rows

   */

  public abstract Map<String, Object> nextDeletedRowKey();



  /**

   * This is used during delta-import. This gives the primary keys and their

   * values of all the rows changed in a parent entity due to changes in this

   * entity.

   *

   * @return the pk vs value of all changed rows in the parent entity

   */

  public abstract Map<String, Object> nextModifiedParentRowKey();



  /**

   * Invoked for each parent-row after the last row for this entity is processed. If this is the root-most

   * entity, it will be called only once in the import, at the very end.

   * 

   */

  public abstract void destroy();



  /**

   * Invoked after the transformers are invoked. EntityProcessors can add, remove or modify values

   * added by Transformers in this method.

   *

   * @param r The transformed row

   * @since solr 1.4

   */

  public void postTransform(Map<String, Object> r) {

  }



  /**

   * Invoked when the Entity processor is destroyed towards the end of import.

   *

   * @since solr 1.4

   */

  public void close() {

    //no-op

  }

}

继承类EntityProcessorBase是所有具体实体处理器的基类，定义了公用方法，其中最重要的是Map<String, Object> getNext()，从数据迭代器Iterator<Map<String, Object>> rowIterator获取Map类型数据记录（其中DIHCacheSupport cacheSupport对象用于缓存）

protected Map<String, Object> getNext() {

    if(cacheSupport==null) {

      try {

        if (rowIterator == null)

          return null;

        if (rowIterator.hasNext())

          return rowIterator.next();

        query = null;

        rowIterator = null;

        return null;

      } catch (Exception e) {

        SolrException.log(log, "getNext() failed for query '" + query + "'", e);

        query = null;

        rowIterator = null;

        wrapAndThrow(DataImportHandlerException.WARN, e);

        return null;

      }

    } else  {

      return cacheSupport.getCacheData(context, query, rowIterator);

    }      

  }

SqlEntityProcessor类为数据库数据源的实体处理器

/**

 * <p>

 * An {@link EntityProcessor} instance which provides support for reading from

 * databases. It is used in conjunction with {@link JdbcDataSource}. This is the default

 * {@link EntityProcessor} if none is specified explicitly in data-config.xml

 * </p>

 * <p/>

 * <p>

 * Refer to <a

 * href="http://wiki.apache.org/solr/DataImportHandler">http://wiki.apache.org/solr/DataImportHandler</a>

 * for more details.

 * </p>

 * <p/>

 * <b>This API is experimental and may change in the future.</b>

 *

 * @version $Id: SqlEntityProcessor.java 1065312 2011-01-30 16:08:25Z rmuir $

 * @since solr 1.3

 */

public class SqlEntityProcessor extends EntityProcessorBase {

  private static final Logger LOG = LoggerFactory.getLogger(SqlEntityProcessor.class);

  //数据源

  protected DataSource<Iterator<Map<String, Object>>> dataSource;

  //初始化数据源

  @Override

  @SuppressWarnings("unchecked")

  public void init(Context context) {

    super.init(context);

    dataSource = context.getDataSource();

  }

  //初始化数据迭代器（根据查询语句从数据源获取）

  protected void initQuery(String q) {

    try {

      DataImporter.QUERY_COUNT.get().incrementAndGet();

      rowIterator = dataSource.getData(q);

      this.query = q;

    } catch (DataImportHandlerException e) {

      throw e;

    } catch (Exception e) {

      LOG.error( "The query failed '" + q + "'", e);

      throw new DataImportHandlerException(DataImportHandlerException.SEVERE, e);

    }

  }



  @Override

  public Map<String, Object> nextRow() {    

    if (rowIterator == null) {

      String q = getQuery();

      initQuery(context.replaceTokens(q));

    }

    return getNext();

  }



  @Override

  public Map<String, Object> nextModifiedRowKey() {

    if (rowIterator == null) {

      String deltaQuery = context.getEntityAttribute(DELTA_QUERY);

      if (deltaQuery == null)

        return null;

      initQuery(context.replaceTokens(deltaQuery));

    }

    return getNext();

  }



  @Override

  public Map<String, Object> nextDeletedRowKey() {

    if (rowIterator == null) {

      String deletedPkQuery = context.getEntityAttribute(DEL_PK_QUERY);

      if (deletedPkQuery == null)

        return null;

      initQuery(context.replaceTokens(deletedPkQuery));

    }

    return getNext();

  }



  @Override

  public Map<String, Object> nextModifiedParentRowKey() {

    if (rowIterator == null) {

      String parentDeltaQuery = context.getEntityAttribute(PARENT_DELTA_QUERY);

      if (parentDeltaQuery == null)

        return null;

      LOG.info("Running parentDeltaQuery for Entity: "

              + context.getEntityAttribute("name"));

      initQuery(context.replaceTokens(parentDeltaQuery));

    }

    return getNext();

  }

  

  public String getQuery() {

    String queryString = context.getEntityAttribute(QUERY);

    if (Context.FULL_DUMP.equals(context.currentProcess())) {

      return queryString;

    }

    if (Context.DELTA_DUMP.equals(context.currentProcess())) {

      String deltaImportQuery = context.getEntityAttribute(DELTA_IMPORT_QUERY);

      if(deltaImportQuery != null) return deltaImportQuery;

    }

    LOG.warn("'deltaImportQuery' attribute is not specified for entity : "+ entityName);

    return getDeltaImportQuery(queryString);

  }



  public String getDeltaImportQuery(String queryString) {    

    StringBuilder sb = new StringBuilder(queryString);

    if (SELECT_WHERE_PATTERN.matcher(queryString).find()) {

      sb.append(" and ");

    } else {

      sb.append(" where ");

    }

    boolean first = true;

    String[] primaryKeys = context.getEntityAttribute("pk").split(",");

    for (String primaryKey : primaryKeys) {

      if (!first) {

        sb.append(" and ");

      }

      first = false;

      Object val = context.resolve("dataimporter.delta." + primaryKey);

      if (val == null) {

        Matcher m = DOT_PATTERN.matcher(primaryKey);

        if (m.find()) {

          val = context.resolve("dataimporter.delta." + m.group(1));

        }

      }

      sb.append(primaryKey).append(" = ");

      if (val instanceof Number) {

        sb.append(val.toString());

      } else {

        sb.append("'").append(val.toString()).append("'");

      }

    }

    return sb.toString();

  }



  private static Pattern SELECT_WHERE_PATTERN = Pattern.compile(

          "^\\s*(select\\b.*?\\b)(where).*", Pattern.CASE_INSENSITIVE);



  public static final String QUERY = "query";



  public static final String DELTA_QUERY = "deltaQuery";



  public static final String DELTA_IMPORT_QUERY = "deltaImportQuery";



  public static final String PARENT_DELTA_QUERY = "parentDeltaQuery";



  public static final String DEL_PK_QUERY = "deletedPkQuery";



  public static final Pattern DOT_PATTERN = Pattern.compile(".*?\\.(.*)$");

}

 我们接下来分析EntityProcessorWrapper类，该类继承自抽象类EntityProcessor，用于装饰具体的实体处理器（装饰模式）

其重要成员如下

 //被装饰的实体处理器 

 EntityProcessor delegate;

  private DocBuilder docBuilder;



  String onError;

  Context context;

  protected VariableResolverImpl resolver;

  String entityName;



  protected List<Transformer> transformers;



  protected List<Map<String, Object>> rowcache;

在它的构造方法里面，初始化被装饰的成员对象

public EntityProcessorWrapper(EntityProcessor delegate, DocBuilder docBuilder) {

    this.delegate = delegate;

    this.docBuilder = docBuilder;

  }

初始化方法里面调用被装饰对象的初始化方法（获取数据源） 

@Override

  public void init(Context context) {

    rowcache = null;

    this.context = context;

    resolver = (VariableResolverImpl) context.getVariableResolver();

    //context has to be set correctly . keep the copy of the old one so that it can be restored in destroy

    if (entityName == null) {

      onError = resolver.replaceTokens(context.getEntityAttribute(ON_ERROR));

      if (onError == null) {

          onError = ABORT;

      }

      entityName = context.getEntityAttribute(DataConfig.NAME);

    }

    delegate.init(context);



  }

其他相关方法均为调用被装饰的具体实体处理器的相应方法，另外添加了数据转换等功能，本文不再具体分析 

---------------------------------------------------------------------------

本系列solr dataimport 数据导入源码分析系本人原创

转载请注明出处 博客园 刺猬的温驯

本文链接 http://www.cnblogs.com/chenying99/archive/2013/05/04/3059397.html