Elasticsearch 6.4基本操作 - Java版

1. Elasticsearch Java API有四类client连接方式

• TransportClient
• RestClient
• Jest
• Spring Data Elasticsearch

其中TransportClient和RestClient是Elasticsearch原生的api。TransportClient可以支持2.x，5.x版本，TransportClient将会在Elasticsearch 7.0弃用并在8.0中完成删除，替而代之，我们使用Java High Level REST Client，它使用HTTP请求而不是Java序列化请求。

Jest是Java社区开发的，是Elasticsearch的Java Http Rest客户端；Spring Data Elasticsearch是spring集成的Elasticsearch开发包。

建议：TransportClient将会在后面的版本中弃用，因此不推荐后续使用；而Jest由于是社区维护，所以更新有一定延迟，目前最新版对接ES6.3.1，近一个月只有四个issue，说明整体活跃度较低，因此也不推荐使用；Spring Data Elasticsearch主要是与Spring生态对接，可以在web系统中整合到Spring中使用。目前比较推荐使用官方的高阶、低阶Rest Client，官方维护，比较值得信赖。本文主要介绍RestClient，其他的只做简单概述。

2. TransportClient

这里考虑到后面版本将弃用TransportClinet，主要简单介绍TransportClient的创建、以及一些基本增删改查操作。

    org.elasticsearch.client
    transport
    6.4.3


     org.elasticsearch
     elasticsearch
     6.4.3

/**
 * @Author: Yang JianQiu
 * @Date: 2018/11/12 16:40
 *
 * transportClient将会在7.0版本上过时，并在8.0版本上移除掉，建议使用Java High Level REST Client
 */
public class TransportClientUtils {

    private TransportClient client = null;

    /**
     * 这里使用饿汉单例模式创建TransportClient
     */
    public TransportClientUtils() {
        if (client == null){
            synchronized (TransportClientUtils.class){
                if (client == null){
                    client = getClient();
                }
            }
        }
    }

    public TransportClient getClient(){
        TransportClient client = null;
        try {
            Settings settings = Settings.builder()
                    .put("client.transport.sniff", true)
                    .put("cluster.name", "bigdata").build();
            client = new PreBuiltTransportClient(settings)
                    .addTransportAddress(new TransportAddress(new InetSocketAddress("192.168.187.201", 9300)));
        } catch (Exception e) {
            e.printStackTrace();
        }
        return client;
    }

    public viod test(){
        //增，插入记录
        IndexResponse response = client.prepareIndex("twitter", "_doc")
                .setSource(json, XContentType.JSON)
                .get();
   
        //根据Id查询
        GetResponse response = client.prepareGet("twitter", "_doc", 1).get();

        //根据Id删除
        DeleteResponse response = client.prepareDelete("twitter", "_doc", 1).get();

        //根据Id更新对应的字段
        UpdateRequest updateRequest1 = new UpdateRequest("twitter", "_doc", "NpEWCGcBi36MQkKOSdf3")
                .doc(jsonBuilder()
                .startObject()
                .field("user", "tom")
                .endObject()
        );
        client.update(updateRequest1).get();
 
        //另外还有批处理API、search负责查询API、Aggregate聚合API...
    }
    
}    

具体的TransportClient的其他API应用可以参考Elasticsearch的TransportClient API doc：

https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html

3. Rest Client

后面的elasticSearch版本将主要使用Rest Client操作数据。

Rest Client分为Java Low REST Client和Java High Level REST Client。

maven的依赖包：

    org.elasticsearch.client
    elasticsearch-rest-high-level-client
    6.4.3


    org.elasticsearch.client
    elasticsearch-rest-client
    6.4.3


    org.elasticsearch
    elasticsearch
    6.4.3

RestHighLevelClient与RestClient的创建：

/**
 * @author: Swordfall Yeung
 * @date:
 * @desc:
 */
public class RestClientUtils {

    /**
     * 高阶Rest Client
     */
    private RestHighLevelClient client = null;
    /**
     * 低阶Rest Client
     */
    private RestClient restClient = null;

    /**
     * 这里使用饿汉单例模式创建RestHighLevelClient
     */
    public RestClientUtils() {
        if (client == null) {
            synchronized (RestHighLevelClient.class) {
                if (client == null) {
                    client = getClient();
                }
            }
        }
    }

    private RestHighLevelClient getClient() {
        RestHighLevelClient client = null;

        try {
            client = new RestHighLevelClient(
                    RestClient.builder(
                            new HttpHost("192.168.187.201", 9300, "http")
                    )
            );
        } catch (Exception e) {
            e.printStackTrace();
        }
        return client;
    }

    private RestClient getRestClient() {
        RestClient client = null;

        try {
            client = RestClient.builder(
                    new HttpHost("192.168.187.201", 9300, "http")
            ).build();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return client;
    }

    public void closeClient() {
        try {
            if (client != null) {
                client.close();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

   /**
    * document API 主要是些简单的增删改查操作
    */
    public void documentAPI() {
       //...
    }
     
   /**
    * Search API 主要是些复杂查询操作
    */
    public void searchAPI() {
       //...
    }
}

RestHighLevelAPI分为Document APIs、Search APIs、Miscellaneous APIs、Indices APIs、Cluster APIs...等等，这里主要介绍常用的Document APIs和Search APIs，其余的APIs可以参考：https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html

3.1 Document APIs

Document APIs主要涉及些增删改查等操作，包括Single document APIs单条操作和Multi-document APIs批量操作。Document APIs均可以设置可选参数，实现同步、异步，也均可抛出异常，这里以Index API为例，后面的不再简述。

 Index API介绍：

    /**
     * 增，插入记录
     * 插入操作有四种方式，分同步异步操作，可选参数设置，结果返回IndexResponse，抛出异常
     * @throws Exception
     */
    public void index() throws Exception{
        
        //第一种方式: String
        IndexRequest request = new IndexRequest("posts", "doc", "1");
        String jsonString = "{" +
                "\"user\":\"kimchy\"," +
                "\"postDate\":\"2013-01-30\"," +
                "\"message\":\"trying out Elasticsearch\"" +
                "}";
        request.source(jsonString, XContentType.JSON);

        //第二种方式: Map
        Map jsonMap = new HashMap<>();
        jsonMap.put("user", "kimchy");
        jsonMap.put("postDate", new Date());
        jsonMap.put("message", "trying out Elasticsearch");
        IndexRequest indexRequest = new IndexRequest("posts", "doc", "1").source(jsonMap);

        //第三种方式: XContentBuilder automatically converted to JSON
        XContentBuilder builder = XContentFactory.jsonBuilder();
        builder.startObject();
        {
            builder.field("user", "kimchy");
            builder.timeField("postDate" , new Date());
            builder.field("message", "trying out Elasticsearch");
        }
        builder.endObject();
        IndexRequest indexRequest1 = new IndexRequest("posts", "doc", "1")
                .source(builder);

        //第四种方式: source -> key-pairs
        IndexRequest indexRequest2 = new IndexRequest("posts", "doc", "1")
                .source("user", "kimchy",
                        "postDate", new Date(),
                        "message", "trying out Elasticsearch"
                );

        //可选的参数设置
        request.routing("routing");
        request.parent("parent");
        request.timeout(TimeValue.timeValueSeconds(1));
        request.timeout("1s");
        request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
        request.setRefreshPolicy("wait_for");
        request.version(2);
        request.versionType(VersionType.EXTERNAL);
        request.opType(DocWriteRequest.OpType.CREATE);
        request.opType("create");
        request.setPipeline("pipeline");

        //同步执行
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);

        //异步执行
        ActionListener listener = new ActionListener() {
            @Override
            public void onResponse(IndexResponse indexResponse) {

            }

            @Override
            public void onFailure(Exception e) {

            }
        };
        client.indexAsync(request, RequestOptions.DEFAULT, listener);

        //Index Response
        String index = indexResponse.getIndex();
        String type = indexResponse.getType();
        String id = indexResponse.getId();
        long version = indexResponse.getVersion();

        //抛出异常
        IndexRequest request1 = new IndexRequest("posts", "doc", "1")
                .source("field", "value")
                .version(1);
        try {
            IndexResponse response = client.index(request, RequestOptions.DEFAULT);
        } catch (ElasticsearchException e) {
            if (e.status() == RestStatus.CONFLICT){

            }
        }
    }

Get API介绍

    /**
     * 根据 id 获取数据
     * @throws Exception
     */
    public void get() throws Exception{
        GetRequest request = new GetRequest("posts", "doc", "1");

        //可选参数设置
        request.fetchSourceContext(FetchSourceContext.DO_NOT_FETCH_SOURCE);
        String[] includes = new String[]{"message", "*Date"};
        String[] excludes = Strings.EMPTY_ARRAY;
        FetchSourceContext fetchSourceContext = new FetchSourceContext(true, includes, excludes);
        request.fetchSourceContext(fetchSourceContext);

        //同步执行
        GetResponse getResponse = client.get(request, RequestOptions.DEFAULT);

        //异步执行 listener的写法参照Index的异步执行的listener
        client.getAsync(request, RequestOptions.DEFAULT, listener);

        //Get Response 获取信息

        //抛出异常
    }

Exists API介绍

    /**
     * 是否存在
     * @throws Exception
     */
    public void exists() throws Exception{
        GetRequest getRequest = new GetRequest("posts", "doc", "1");
        getRequest.fetchSourceContext(new FetchSourceContext(false));
        getRequest.storedFields("_none_");

        //同步执行
        boolean exists = client.exists(getRequest, RequestOptions.DEFAULT);

        //异步执行 listener的写法参照Index的
        client.existsAsync(getRequest, RequestOptions.DEFAULT, listener);
        
        if (exists){
            System.out.println("存在");
        }else {
            System.out.println("不存在");
        }
    }

Delete API介绍

    /**
     * 根据id删除
     * @throws Exception
     */
    public void delete() throws Exception{
        DeleteRequest request = new DeleteRequest("posts", "doc", "1");

        //同步执行
        DeleteResponse deleteResponse = client.delete(request, RequestOptions.DEFAULT);

        //异步执行 listener参照index的
        client.deleteAsync(request, RequestOptions.DEFAULT, listener);

        //Delete Response
        String index = deleteResponse.getIndex();

        // document was not found
        if (deleteResponse.getResult() == DocWriteResponse.Result.NOT_FOUND) {

        }

        //抛出异常
      
    }

Update API介绍

/**
     * 根据id更新
     * @throws Exception
     */
    public void update() throws Exception{
        UpdateRequest request = new UpdateRequest("posts", "doc", "1");
        Map parameters = Collections.singletonMap("count", 4);

        //第一种方式：inline script
        Script inline = new Script(ScriptType.INLINE, "painless", "ctx._source.field += params.count", parameters);
        request.script(inline);

        //第二种方式：stored script
        Script stored = new Script(ScriptType.STORED, null, "increment-field", parameters);
        request.script(stored);

        //第三种方式：partial document String
        String jsonString = "{" +
                "\"updated\":\"2017-01-01\"," +
                "\"reason\":\"daily update\"" +
                "}";
        request.doc(jsonString, XContentType.JSON);

        //第四种方式：partial document Map
        Map jsonMap = new HashMap<>();
        jsonMap.put("updated", new Date());
        jsonMap.put("reason", "daily update");
        request.doc(jsonMap);

        //第五种方式：partial document XContentBuilder
        XContentBuilder builder = XContentFactory.jsonBuilder();
        builder.startObject();
        {
            builder.timeField("updated", new Date());
            builder.field("reason", "daily update");
        }
        builder.endObject();
        request.doc(builder);

        //第六种方式：partial document Object key-pairs
        request.doc("updated", new Date(),
                   "reason", "daily update");

        //upserts
        String jsonString1 = "{\"created\":\"2017-01-01\"}";
        request.upsert(jsonString1, XContentType.JSON);

        //同步执行
        UpdateResponse updateResponse = client.update(request, RequestOptions.DEFAULT);

        //异步执行， listener创建参考index的
        client.updateAsync(request, RequestOptions.DEFAULT, listener);

        //update Response
        GetResult result = updateResponse.getGetResult();
        if (result.isExists()) {
            String sourceAsString = result.sourceAsString();
            Map sourceAsMap = result.sourceAsMap();
            byte[] sourceAsBytes = result.source();
        } else {

        }

        //抛出异常
        
    }

 Bulk API介绍

    /**
     * 批量处理
     * @throws Exception
     */
    public void bulk() throws Exception{
        BulkRequest request = new BulkRequest();

        //Other
        request.add(new DeleteRequest("posts", "doc", "3"));
        request.add(new UpdateRequest("posts", "doc", "2")
                .doc(XContentType.JSON, "other", "test"));
        request.add(new IndexRequest("posts", "doc", "4")
                .source(XContentType.JSON, "field", "baz"));

        //同步执行
        BulkResponse bulkResponses = client.bulk(request, RequestOptions.DEFAULT);

        //异步执行
        ActionListener listener = new ActionListener() {
            @Override
            public void onResponse(BulkResponse bulkResponse) {

            }

            @Override
            public void onFailure(Exception e) {

            }
        };
        client.bulkAsync(request, RequestOptions.DEFAULT, listener);

        //Bulk Response 批处理结果
        for (BulkItemResponse bulkItemResponse: bulkResponses){
            DocWriteResponse itemResponse = bulkItemResponse.getResponse();

            if (bulkItemResponse.getOpType() == DocWriteRequest.OpType.INDEX
                    || bulkItemResponse.getOpType() == DocWriteRequest.OpType.CREATE) {
                IndexResponse indexResponse = (IndexResponse) itemResponse;

            } else if (bulkItemResponse.getOpType() == DocWriteRequest.OpType.UPDATE) {
                UpdateResponse updateResponse = (UpdateResponse) itemResponse;

            } else if (bulkItemResponse.getOpType() == DocWriteRequest.OpType.DELETE) {
                DeleteResponse deleteResponse = (DeleteResponse) itemResponse;
            }
        }

        for (BulkItemResponse bulkItemResponse : bulkResponses) {
            if (bulkItemResponse.isFailed()) {
                BulkItemResponse.Failure failure = bulkItemResponse.getFailure();

            }
        }

        //Bulk Processor 自定义批处理器
        BulkProcessor.Listener listener1 = new BulkProcessor.Listener() {
            @Override
            public void beforeBulk(long l, BulkRequest bulkRequest) {

            }

            @Override
            public void afterBulk(long l, BulkRequest bulkRequest, BulkResponse bulkResponse) {

            }

            @Override
            public void afterBulk(long l, BulkRequest bulkRequest, Throwable throwable) {

            }
        };

        BiConsumer> bulkConsumer =
                (request1, bulkListener) -> client.bulkAsync(request1, RequestOptions.DEFAULT, bulkListener);
        BulkProcessor bulkProcessor = BulkProcessor.builder(bulkConsumer, listener1).build();

        BiConsumer> bulkConsumer1 =
                (request2, bulkListener) -> client.bulkAsync(request2, RequestOptions.DEFAULT, bulkListener);
        BulkProcessor.Builder builder = BulkProcessor.builder(bulkConsumer1, listener1);
        builder.setBulkActions(500);
        builder.setBulkSize(new ByteSizeValue(1L, ByteSizeUnit.MB));
        builder.setConcurrentRequests(0);
        builder.setFlushInterval(TimeValue.timeValueSeconds(10L));
        builder.setBackoffPolicy(BackoffPolicy
                .constantBackoff(TimeValue.timeValueSeconds(1L), 3));

        //Once the BulkProcessor is created requests can be added to it:
        IndexRequest one = new IndexRequest("posts", "doc", "1").
                source(XContentType.JSON, "title",
                        "In which order are my Elasticsearch queries executed?");
        IndexRequest two = new IndexRequest("posts", "doc", "2")
                .source(XContentType.JSON, "title",
                        "Current status and upcoming changes in Elasticsearch");
        IndexRequest three = new IndexRequest("posts", "doc", "3")
                .source(XContentType.JSON, "title",
                        "The Future of Federated Search in Elasticsearch");
        bulkProcessor.add(one);
        bulkProcessor.add(two);
        bulkProcessor.add(three);

        boolean terminated = bulkProcessor.awaitClose(30L, TimeUnit.SECONDS);
    }

Multi-Get API介绍

/**
     * 根据id批量获取数据
     * @throws Exception
     */
    public void multiGet() throws Exception{
        MultiGetRequest request = new MultiGetRequest();
        request.add(new MultiGetRequest.Item("index","type","example_id"));
        request.add(new MultiGetRequest.Item("index", "type", "another_id"));

        //optional arguments
        request.add(new MultiGetRequest.Item("index", "type", "example_id") .fetchSourceContext(FetchSourceContext.DO_NOT_FETCH_SOURCE));

        //同步执行
        MultiGetResponse responses = client.mget(request, RequestOptions.DEFAULT);

        //异步执行 listener参考Index的
        client.mgetAsync(request, RequestOptions.DEFAULT, listener);

        //Multi Get Response
        MultiGetItemResponse firstItem = response.getResponses()[0];
        GetResponse firstGet = firstItem.getResponse();
        if (firstGet.isExists()) {

        }
    }

3.2 Search APIs

 public void search() throws Exception{
        //match all query 查询所有数据
        SearchRequest searchRequest = new SearchRequest();
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        searchRequest.source(searchSourceBuilder);

        //使用SearchSourceBuilder查询指定字段 
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(QueryBuilders.termQuery("user", "kimchy"));
        sourceBuilder.from(0);
        sourceBuilder.size(5);
        sourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
        SearchRequest searchRequest2 = new SearchRequest();
        //index 数据库
        searchRequest2.indices("posts");
        searchRequest2.source(sourceBuilder);

        //Building queries
        //One way, QueryBuilder can be created using its constructor 使用QueryBuilder的构造函数
        MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("user", "kimchy");
        matchQueryBuilder.fuzziness(Fuzziness.AUTO);
        matchQueryBuilder.prefixLength(3);
        matchQueryBuilder.maxExpansions(10);
        //Two way, QueryBuilder objects can also be created using the QueryBuilders utility class. 直接使用matchQuery
        QueryBuilder matchQueryBuilder1 = matchQuery("user", "kimchy")
                                                       .fuzziness(Fuzziness.AUTO)
                                                       .prefixLength(3)
                                                       .maxExpansions(10);

        searchSourceBuilder.query(matchQueryBuilder1);

        //Specifying Sorting 指定排序
        sourceBuilder.sort(new ScoreSortBuilder().order(SortOrder.DESC));
        sourceBuilder.sort(new FieldSortBuilder("_uid").order(SortOrder.ASC));

        //Source filtering, turn off _source retrieval completely
        sourceBuilder.fetchSource(false);
        //an array of one or more wildcard patterns to control which fields get included or excluded in a more fine grained way
        String[] includeFields = new String[] {"title", "user", "innerObject.*"};
        String[] excludeFields = new String[] {"_type"};
        sourceBuilder.fetchSource(includeFields, excludeFields);

        //Requesting Aggregations
        SearchSourceBuilder searchSourceBuilder2 = new SearchSourceBuilder();
        TermsAggregationBuilder aggregation = AggregationBuilders.terms("by_company")
                .field("company.keyword");
        aggregation.subAggregation(AggregationBuilders.avg("average_age")
                .field("age"));
        searchSourceBuilder2.aggregation(aggregation);

        //Requesting Suggestions
        SearchSourceBuilder searchSourceBuilder3 = new SearchSourceBuilder();
        SuggestionBuilder termSuggestionBuilder = SuggestBuilders.termSuggestion("user").text("kmichy");
        SuggestBuilder suggestBuilder = new SuggestBuilder();
        suggestBuilder.addSuggestion("suggest_user", termSuggestionBuilder);
        searchSourceBuilder3.suggest(suggestBuilder);

        //同步执行
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

        //异步执行 listener参考index的
        client.searchAsync(searchRequest, RequestOptions.DEFAULT, listener);

        //SearchResponse
        RestStatus status = searchResponse.status();
        for (ShardSearchFailure failure : searchResponse.getShardFailures()) {
            // failures should be handled here
        }

        //Retrieving SearchHits 获取结果数据
        SearchHits hits = searchResponse.getHits();
        long totalHits = hits.getTotalHits();
        float maxScore = hits.getMaxScore();
        SearchHit[] searchHits = hits.getHits();
        for (SearchHit hit : searchHits) {
            // do something with the SearchHit
            String index = hit.getIndex();
            String type = hit.getType();
            String id = hit.getId();
            float score = hit.getScore();

            String sourceAsString = hit.getSourceAsString();
            Map sourceAsMap = hit.getSourceAsMap();
            String documentTitle = (String) sourceAsMap.get("title");
            List