java es 数据批量导入_基于Spring Batch向Elasticsearch批量导入数据示例

1.介绍

当系统有大量数据需要从数据库导入Elasticsearch时，使用Spring Batch可以提高导入的效率。Spring Batch使用ItemReader分页读取数据，ItemWriter批量写数据。由于Spring Batch没有提供Elastisearch的ItemWriter和ItemReader，本示例中自定义一个ElasticsearchItemWriter(ElasticsearchItemReader)，用于批量导入。

2.示例

2.1 pom.xml

本文使用spring data jest连接ES(也可以使用spring data elasticsearch连接ES),ES版本为5.5.3

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

4.0.0

com.hfcsbc.estl

es-etl

0.0.1-SNAPSHOT

jar

es-etl

Demo project for Spring Boot

org.springframework.boot

spring-boot-starter-parent

2.0.0.M7

UTF-8

UTF-8

1.8

org.springframework.boot

spring-boot-starter

org.springframework.boot

spring-boot-starter-data-jpa

org.postgresql

postgresql

org.springframework.boot

spring-boot-starter-batch

com.github.vanroy

spring-boot-starter-data-jest

3.0.0.RELEASE

io.searchbox

jest

5.3.2

org.projectlombok

lombok

org.springframework.boot

spring-boot-starter-test

test

org.springframework.boot

spring-boot-maven-plugin

spring-snapshots

Spring Snapshots

https://repo.spring.io/snapshot

true

spring-milestones

Spring Milestones

https://repo.spring.io/milestone

false

spring-snapshots

Spring Snapshots

https://repo.spring.io/snapshot

true

spring-milestones

Spring Milestones

https://repo.spring.io/milestone

false

2.2 实体类及repository

package com.hfcsbc.esetl.domain;

import lombok.Data;

import org.springframework.data.elasticsearch.annotations.Document;

import org.springframework.data.elasticsearch.annotations.Field;

import org.springframework.data.elasticsearch.annotations.FieldType;

import javax.persistence.Entity;

import javax.persistence.Id;

import javax.persistence.OneToOne;

/**

* Create by pengchao on 2018/2/23

*/

@Document(indexName = "person", type = "person", shards = 1, replicas = 0, refreshInterval = "-1")

@Entity

@Data

public class Person {

@Id