从零学Elasticsearch系列——JAVA API操作

系列文章：

• 从零学Elasticsearch系列——基础概念
• 从零学Elasticsearch系列——环境搭建
• 从零学Elasticsearch系列——使用kibana实现ES基本的操作
• 从零学Elasticsearch系列——深入搜索（Query、Filter、Aggregation）
• 从零学Elasticsearch系列——JAVA API操作
• 从零学Elasticsearch系列——集成中文分词器IK
• 从零学Elasticsearch系列——构建ES集群
• 从零学Elasticsearch系列——搭建ELK Nginx日志分析平台

---

参考资料：https://spring.io/projects/spring-data-elasticsearch#overview

测试代码GitHub地址：https://github.com/gaozhy520/es_springdata_demo

查询方式

• Restful API

  基于http协议，使用JSON为数据交换格式，通过9200端口的与Elasticsearch进行通信，您可使用HttpClient类库通过9200端口操作Elasticsearch。

• JAVA API(Spring Data ElasticSearch)

  Spring Data ElasticSearch封装了与ES交互的实现细节，可以使系统开发者以Spring Data Repository 风格实现与ES的数据交互。Elasticsearch为Java用户提供了两种内置客户端：

  节点客户端(node client)：
  节点客户端以无数据节点(none data node)身份加入集群，换言之，它自己不存储任何数据，但是它知道数据在集群中的具体位置，并且能够直接转发请求到对应的节点上。

  传输客户端(Transport client)：
  这个更轻量的传输客户端能够发送请求到远程集群。它自己不加入集群，只是简单转发请求给集群中的节点。两个Java客户端都通过9300端口与集群交互，使用Elasticsearch传输协议(Elasticsearch Transport Protocol)。集群中的节点之间也通过9300端口进行通信。如果此端口未开放，你的节点将不能组成集群。

Spring Data ElasticSearch实践

Maven依赖

<dependencies>
    <dependency>
        <groupId>org.springframework.datagroupId>
        <artifactId>spring-data-elasticsearchartifactId>
        <version>3.1.3.RELEASEversion>
    dependency>
    <dependency>
        <groupId>org.springframeworkgroupId>
        <artifactId>spring-testartifactId>
        <version>5.1.3.RELEASEversion>
    dependency>
    <dependency>
        <groupId>junitgroupId>
        <artifactId>junitartifactId>
        <version>4.12version>
    dependency>
dependencies>

准备配置文件

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticesearch="http://www.springframework.org/schema/data/elasticsearch"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/data/elasticsearch http://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch.xsd">

    
    <elasticsearch:repositories base-package="com.baizhi.es.dao">elasticsearch:repositories>

    
    <elasticesearch:transport-client id="client" cluster-name="elasticsearch"
                                     cluster-nodes="192.168.23.143:9300">elasticesearch:transport-client>
    
    
    <bean id="elasticsearchTemplate" class="org.springframework.data.elasticsearch.core.ElasticsearchTemplate">
        <constructor-arg name="client" ref="client">constructor-arg>
    bean>
    
    <bean id="customUserRepository" class="com.baizhi.es.dao.CustomUserRepositoryImpl">

    bean>
beans>

准备映射实体类

package com.baizhi.entity;

import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.PersistenceConstructor;
import org.springframework.data.elasticsearch.annotations.Document;

import java.util.Date;

/**
 * @author gaozhy
 * @date 2018/12/28.17:05
 */
// 文档注解 用于描述索引及其相关信息
@Document(indexName = "zpark",type = "user")
public class User {

    // 主键
    @Id
    private String id;

    private String name;

    private String realname;

    private Integer age;

    private Double salary;

    private Date birthday;
    
	// 指定address域的类型 并明确索引和检索使用的分词器（需安装IK分词器）
    @Field(type = FieldType.Text,searchAnalyzer = "ik_max_word",analyzer = "ik_max_word")
    private String address;

	// 省略get/set toString方法 ......
}

spring data repository

spring data elsaticsearch提供了三种构建查询模块的方式：

• 基本的增删改查：继承spring data提供的接口就默认提供
• 接口中声明方法：无需实现类，spring data根据方法名，自动生成实现类，方法名必须符合一定的规则（这里还扩展出一种忽略方法名，根据注解的方式查询）
• 自定义repository：在实现类中注入elasticsearchTemplate，实现上面两种方式不易实现的查询（例如：聚合、分组、深度翻页等）

上面的第一点和第二点只需要声明接口，无需实现类，spring data会扫描并生成实现类

基础的repository接口：提供基本的增删改查和根据方法名的查询

package com.baizhi.es.dao;

import com.baizhi.entity.User;
import org.springframework.data.elasticsearch.annotations.Query;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

import java.util.List;

/**
 * 基础操作的es repository接口（定义的有通用的增删改查方法）
 *
 * @author gaozhy
 * @date 2018/12/29.9:26
 */
public interface UserRepository extends ElasticsearchRepository<User,String> {

    /**
     * 根据年龄区间查询数据 并根据年龄降序排列
     */
    public List<User> findByAgeBetweenOrderByAgeDesc(int start,int end);

    /**
     * 查询真实姓名已“王”开头的数据
     */
    public List<User> findByRealnameStartingWith(String startStr);

    /**
     * 通过Query注解自定义查询表达式
     */
    @Query("{\"bool\" : {\"must\" : {\"fuzzy\" : {\"name\" : \"?0\"}}}}")
    public List<User> findByNameLike(String name);
}

测试代码如下：

package com.baizhi.es.test;

import com.baizhi.entity.User;
import com.baizhi.es.dao.UserRepository;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.data.domain.Sort;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import java.util.Date;
import java.util.List;
import java.util.Optional;

/**
 * @author gaozhy
 * @date 2018/12/29.9:30
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:applicationContext-es.xml")
public class UserRepositoryTest {
	//=============================ElasticsearchRepository接口方法测试====================
    @Autowired
    private UserRepository userRepository;

    /**
     * 查所有
     */
    @Test
    public void testQueryAll(){
        Iterable<User> users = userRepository.findAll();
        for (User user : users) {
            System.out.println(user);
        }
    }

    /**
     * 查询所有 并根据年龄倒序排列
     */
    @Test
    public void testQueryBySort(){
        Iterable<User> users = userRepository.findAll(Sort.by(Sort.Direction.DESC, "age"));
        for (User user : users) {
            System.out.println(user);
        }
    }

    /**
     * 根据id查询
     */
    @Test
    public void testQueryById(){
        Optional<User> user = userRepository.findById("1");
        System.out.println(user.get());
    }

    /**
     * 新增或者修改数据
     */
    @Test
    public void testAdd(){
        User user = userRepository.save(new User("6", "wb", "王八", 26, 10000D, new Date(), "河南省郑州市二七区德化街南路33号"));
        System.out.println(user);
    }

    //================================自定义方法==================================

    /**
     * 接口中声明方法查询：
     *    根据年龄区间查询数据 并根据年龄降序排列
     */
    @Test
    public void testQueryByRange(){
        List<User> users = userRepository.findByAgeBetweenOrderByAgeDesc(20, 28);
        users.forEach(user -> System.out.println(user));
    }

    /**
     * 接口中声明方法查询：
     *    查询真实姓名已“王”开头的数据
     *
     *    响应结果：
     *    User{id='6', name='wb', realname='王八', age=26, salary=10000.0, birthday=Sat Dec 29 14:38:39 CST 2018, address='河南省郑州市二七区德化街南路33号'}
          User{id='3', name='ww', realname='王五', age=25, salary=4300.0, birthday=Tue Mar 15 08:00:00 CST 2016, address='北京市海淀区中关村大街新中关商城2楼511室'}
     */
    @Test
    public void testQueryByPrefix(){
        List<User> users = userRepository.findByRealnameStartingWith("王");
        users.forEach(user -> System.out.println(user));
    }
    
    //==================================================================
    /**
     * 通过Query注解自定义查询表达式
     */
    @Test
    public void testQueryByNameLike(){
        List<User> users = userRepository.findByNameLike("zs");
        users.forEach(user -> System.out.println(user));
    }
}

自定义Repository接口：使用elasticsearchTemplate实现复杂查询

自定义CustomUserRepository接口

package com.baizhi.es.dao;

import com.baizhi.entity.User;

import java.util.List;
import java.util.Map;

/**
 * @author gaozhy
 * @date 2019/1/1.23:10
 */
public interface CustomUserRepository {

    public List<User> findByPageable(int nowPage,int pageSize);

    public List<User> findByFieldDesc(String field);

    public List<User> findByRealNameLikeAndHighLight(String realName);

    public List<User> findByNameWithTermFilter(String ...terms);

    public List<User> findByAgeWithRangeFilter(int start,int end);

    public Map findByNameStartingWithAndAggregations(String prefixName);

    /**
     * 嵌套查询：
     *
     * 先按年龄直方图（桶聚合）统计
     * 然后再统计区间内员工的最高工资（度量聚合）
     */
    public Map aggregationsWithHistogramAndMax();

    /**
     * 日期直方图（桶聚合）
     */
    public Map aggregationsWithDateHistogram();

}

自定义CustomUserRepositoryImpl实现类

package com.baizhi.es.dao;

import com.baizhi.entity.User;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.Aggregation;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.Aggregations;
import org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramInterval;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.search.sort.SortOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.core.ElasticsearchTemplate;
import org.springframework.data.elasticsearch.core.ResultsExtractor;
import org.springframework.data.elasticsearch.core.SearchResultMapper;
import org.springframework.data.elasticsearch.core.aggregation.AggregatedPage;
import org.springframework.data.elasticsearch.core.aggregation.impl.AggregatedPageImpl;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;

import static org.elasticsearch.index.query.QueryBuilders.*;

/**
 * @author gaozhy
 * @date 2019/1/1.23:11
 */
public class CustomUserRepositoryImpl implements CustomUserRepository {

    @Autowired
    private ElasticsearchTemplate template;

    /**
     * ====================================
     * {
     * "query": {
     * "match_all": {}
     * },
     * "from":1,     //从第几条开始    (从0开始)
     * "size":1      //大小
     * }
     * ====================================
     *
     * @param nowPage
     * @param pageSize
     * @return
     */
    @Override
    public List<User> findByPageable(int nowPage, int pageSize) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(matchAllQuery())
                .withPageable(new PageRequest(nowPage - 1, pageSize))
                .build();

        return template.queryForList(query, User.class);
    }

    /**
     * @param field
     * @return
     */
    @Override
    public List<User> findByFieldDesc(String field) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(matchAllQuery())
                .withSort(SortBuilders.fieldSort(field).order(SortOrder.DESC))
                .build();
        return template.queryForList(query, User.class);
    }

    /**
     * 高亮
     *
     * @param realName
     * @return
     */
    @Override
    public List<User> findByRealNameLikeAndHighLight(String realName) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(matchQuery("realname", realName)
                )
                .withHighlightFields(new HighlightBuilder.Field("realname"))
                .build();
        AggregatedPage<User> users = template.queryForPage(query, User.class, new SearchResultMapper() {
            @Override
            public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
                ArrayList<User> users = new ArrayList<>();
                SearchHits searchHits = searchResponse.getHits();
                for (SearchHit searchHit : searchHits) {
                    if (searchHits.getHits().length <= 0) {
                        return null;
                    }
                    User user = new User();
                    user.setId(searchHit.getId());
                    // searchHit.getSourceAsMap().forEach((k, v) -> System.out.println(k + " " + v));
                    user.setName(searchHit.getSourceAsMap().get("name").toString());
                    user.setAddress(searchHit.getSourceAsMap().get("address").toString());
                    user.setAge(Integer.parseInt(searchHit.getSourceAsMap().get("age").toString()));
                    user.setBirthday(new Date(Long.parseLong(searchHit.getSourceAsMap().get("birthday").toString())));
                    user.setSalary(Double.parseDouble(searchHit.getSourceAsMap().get("salary").toString()));
                    String realname = searchHit.getHighlightFields().get("realname").fragments()[0].toString();
                    user.setRealname(realname);

                    users.add(user);
                }
                return new AggregatedPageImpl<T>((List<T>) users);
            }
        });
        return users.getContent();
    }

    @Override
    public List<User> findByNameWithTermFilter(String... terms) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(matchAllQuery())
                .withFilter(termsQuery("name",terms))
                .build();
        System.out.println(query.getFilter());
        return template.queryForList(query,User.class);
    }

    @Override
    public List<User> findByAgeWithRangeFilter(int start, int end) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withFilter(rangeQuery("age").gte(start).lte(end))
                .build();
        System.out.println(query.getQuery());
        System.out.println(query.getFilter());
        return template.queryForList(query,User.class);
    }

    @Override
    public Map<String, Aggregation> findByNameStartingWithAndAggregations(String prefixName) {
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(prefixQuery("name",prefixName))
                // result为度量聚合结果的别名
                .addAggregation(AggregationBuilders.avg("result").field("age"))
                .build();
        Aggregations aggregations = template.query(query, new ResultsExtractor<Aggregations>() {
            @Override
            public Aggregations extract(SearchResponse searchResponse) {
                Aggregations aggregations = searchResponse.getAggregations();
                return aggregations;
            }
        });
        Map<String, Aggregation> map = aggregations.getAsMap();
        return map;
    }

    @Override
    public Map aggregationsWithHistogramAndMax() {
        SearchQuery query = new NativeSearchQueryBuilder()
                .addAggregation(AggregationBuilders.histogram("result").field("age").interval(5)
                        .subAggregation(AggregationBuilders.max("max_salary").field("salary")))
                .build();
        Aggregations aggregations = template.query(query, new ResultsExtractor<Aggregations>() {

            @Override
            public Aggregations extract(SearchResponse searchResponse) {
                return searchResponse.getAggregations();
            }
        });
        return aggregations.getAsMap();
    }

    @Override
    public Map aggregationsWithDateHistogram() {
        SearchQuery query = new NativeSearchQueryBuilder()
                .addAggregation(AggregationBuilders.dateHistogram("result").field("birthday").format("yyyy-MM-dd").dateHistogramInterval(DateHistogramInterval.YEAR))
                .build();
        Aggregations aggregations = template.query(query, new ResultsExtractor<Aggregations>() {

            @Override
            public Aggregations extract(SearchResponse searchResponse) {
                return searchResponse.getAggregations();
            }
        });
        return aggregations.getAsMap();
    }
}

自定义CustomUserRepositoryTest测试类

package com.baizhi.es.test;

import com.baizhi.entity.User;
import com.baizhi.es.dao.CustomUserRepository;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import java.util.List;
import java.util.Map;

/**
 * @author gaozhy
 * @date 2019/1/1.23:26
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:applicationContext-es.xml")
public class CustomUserRepositoryTest {

    @Autowired
    private CustomUserRepository repository;

    @Test
    public void testQueryByPage(){
        List<User> users = repository.findByPageable(0, 2);
        users.forEach(user -> {
            System.out.println(user);
        });
    }
    @Test
    public void testQueryBySort(){
        List<User> users = repository.findByFieldDesc("_id");
        users.forEach(user -> {
            System.out.println(user);
        });
    }
    @Test
    public void testQueryByHighLight(){
        List<User> users = repository.findByRealNameLikeAndHighLight("王八");
        users.forEach(user -> {
            System.out.println(user);
        });
    }
    @Test
    public void testQueryByNameWithTermFilter(){
        List<User> users = repository.findByNameWithTermFilter("zs","ls");
        users.forEach(user -> {
            System.out.println(user);
        });
    }

    @Test
    public void testQueryByAgeWithRangeFilter(){
        List<User> users = repository.findByAgeWithRangeFilter(21,30);
        users.forEach(user -> {
            System.out.println(user);
        });
    }

    @Test
    public void testQueryByNameStartingWithAndAggregations(){
        Map map = repository.findByNameStartingWithAndAggregations("z");
        System.out.println(map.get("result"));
    }

    @Test
    public void testAggregationsWithHistogramAndMax(){
        Map map = repository.aggregationsWithHistogramAndMax();
        System.out.println(map.get("result"));
    }

    @Test
    public void testAggregationsWithDateHistogram(){
        Map map = repository.aggregationsWithDateHistogram();
        System.out.println(map.get("result"));
    }
}