ElasticSearch-SpringBoot中三种分页查询总结

目录

一、from+size 浅分页

DSL 查询方式

RestHighLevelClient 查询方式

二、scroll 深分页

DSL 查询方式

RestHighLevelClient 查询方式

三、search_after 深分页

DSL 查询方式

RestHighLevelClient 查询方式

---

一、from+size 浅分页

1. 浅分页的原理很简单，就是查询前20条数据，然后截断前10条，只返回10-20的数据。这样其实白白浪费了前10条的查询
2. es默认采用的是from+size形式，在深度分页的情况下，这种效率是非常低的，但是可以随机跳转页面
3. es为了性能，会限制我们分页的深度，es目前支持最大的max_result_window = 10000，也就是from+size的大小不能超过10000

DSL 查询方式

GET demo_index/_search
{
  "query":{
    "match_all": {}
  },
  "from": 0,
  "size": 10,
  "sort": [
    {
      "id": {
        "order": "asc"
      },
      "publish_time": {
        "order": "asc"
      }
    }
  ]
}

注意：es是基于分片的，假设有3个分片，from=100，size=10。则会根据排序规则从3个分片中各取回100条数据数据，然后汇总成300条数据后选择最前边的10条数据

RestHighLevelClient 查询方式

    /**
     * @Description from+size浅分页查询
     * @create by meng
     */
    private List docSearch(Date time, String title) {
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        builder.must(QueryBuilders.matchAllQuery())
                .filter(QueryBuilders.rangeQuery("publish_time").gt(time.getTime()));
        try {
            searchSourceBuilder.query(builder)
                    .sort("id", SortOrder.ASC)
                    .sort("publish_time", SortOrder.ASC)
                    .from(0)
                    .size(10);
            SearchRequest searchRequest = new SearchRequest("demo_index").source(searchSourceBuilder);
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            SearchHit[] hits = searchResponse.getHits().getHits();
            if (hits.length > 0) {
                return Arrays.asList(hits);
            } else {
                return null;
            }
        } catch (IOException e) {
            log.error("doc分页查询异常:{} ", e);
        }
        return null;
    }

二、scroll 深分页

1. from+size查询在10000-50000条数据（1000到5000页）以内的时候还是可以的，但是如果数据过多的话，就会出现深分页问题。为了这个问题，es提出了scroll滚动查询方式
2. scroll滚动搜索，会在第一次搜索的时候，保存一个当下的快照。之后只会基于该快照提供数据搜索。在这个期间数据如果发生变动，是不会让用户看到的。推荐非实时处理大量数据的情况可以使用
3. 不适用于有跳页的情景

DSL 查询方式

GET demo_index/_search?scroll=3m
{
  "query":{
    "match_all": {}
  },
  "from": 0,
  "size": 10,
  "sort": [
    {
      "id": {
        "order": "asc"
      },
      "publish_time": {
        "order": "asc"
      }
    }
  ]
}

1. scroll=3m表示设置scroll_id保留3分钟可用
2. 使用scroll必须要将from设置为0
3. size决定后面每次调用_search搜索返回的数量

通过数据返回的_scroll_id读取下一页内容，每次请求将会读取下10条数据，直到数据读取完毕或者scroll_id保留时间截止：

GET _search/scroll
{
  "scroll_id":"mengliulUaGVuRmV0Y2g7NTsxMDk5NDpkUmpiR2FjOFNhNnlCM1ZDMWpWYnRRO==",
  "scroll": "3m"
}

注意：我们需要再次设置游标查询过期时间为3分钟，GET和POST请求均可，scroll是非常消耗资源的，所以当不需要scroll数据的时候，尽可能快的把scroll_id显式删除掉

清除指定的scroll_id：

DELETE _search/scroll/mengliulUaGVuRmV0Y2g7NTsxMDk5NDpkUmpiR2FjOFNhNnlCM1ZDMWpWYnRRO==

清除所有的scroll：

DELETE _search/scroll/_all

RestHighLevelClient 查询方式

    /**
     * @Description scroll 深分页
     * @create by meng
     */
    private void docSearch(Date time, String title) {
        List searchHits = new ArrayList<>();
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        builder.must(QueryBuilders.matchAllQuery())
                .filter(QueryBuilders.rangeQuery("publish_time").gt(time.getTime()));
        try {
            searchSourceBuilder.query(builder)
                    .sort("id", SortOrder.ASC)
                    .sort("publish_time", SortOrder.ASC)
                    .from(0)
                    .size(10);
            SearchRequest searchRequest = new SearchRequest("demo_index").source(searchSourceBuilder);
            //失效时间为3min
            Scroll scroll = new Scroll(TimeValue.timeValueMinutes(3));
            //封存快照
            searchRequest.scroll(scroll);
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            //计算总页数
            long totalCount = searchResponse.getHits().getTotalHits().value;
            int pageSize = (int) Math.ceil((float) totalCount / 2);
            //多次遍历分页，获取结果
            String scrollId = searchResponse.getScrollId();
            for (int i = 1; i <= pageSize; i++) {
                //获取scrollId
                SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
                searchScrollRequest.scroll(scroll);
                SearchResponse response = restHighLevelClient.scroll(searchScrollRequest, RequestOptions.DEFAULT);
                SearchHits hits = response.getHits();
                scrollId = response.getScrollId();
                Iterator iterator = hits.iterator();
                while (iterator.hasNext()) {
                    SearchHit next = iterator.next();
                    searchHits.add(next);
                }
            }
        } catch (IOException e) {
            log.error("doc分页查询异常:{} ", e);
        }
    }

三、search_after 深分页

1. 可以在实时数据的情况下深度分页
2. 为了找每一页最后一条数据，每个文档必须有一个全局唯一值
3. 不适用于有跳页的情景

DSL 查询方式

GET demo_index/_search?scroll=3m
{
  "query":{
    "match_all": {}
  },
  "from": 0,
  "size": 10,
  "sort": [
    {
      "id": {
        "order": "asc"
      },
      "publish_time": {
        "order": "asc"
      }
    }
  ]
}

1. 使用search_after必须要将from设置为0
2. 上边的DSL查询中id是唯一不重复字段，publish_time可能会重复

注意：查询结果会返回sort字段，我们在返回的结果集中，获取最后一条数据的sort属性值，提供给下次查询中search_after

GET demo_index/_search?scroll=3m
{
  "query":{
    "match_all": {}
  },
  "size": 10,
  "search_after": [
    1638374400000,
    "mengliu20211202"
  ],
  "sort": [
    {
      "id": {
        "order": "asc"
      },
      "publish_time": {
        "order": "asc"
      }
    }
  ]
}

RestHighLevelClient 查询方式

    /**
     * @Description search_after 深分页
     * @create by meng
     */
    private void docSearch(Date time, String title) {
        List searchHits = new ArrayList<>();
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        builder.must(QueryBuilders.matchAllQuery())
                .filter(QueryBuilders.rangeQuery("publish_time").gt(time.getTime()));
        try {
            searchSourceBuilder.query(builder)
                    .sort("id", SortOrder.ASC)
                    .sort("publish_time", SortOrder.ASC)
                    .from(0)
                    .size(10);
            SearchRequest searchRequest = new SearchRequest("demo_index").source(searchSourceBuilder);
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            SearchHit[] hits = searchResponse.getHits().getHits();
            //查询最后一个数据
            SearchHit result = hits[hits.length - 1];
            //分页查询下一页数据
            SearchSourceBuilder searchSourceBuilder2 = new SearchSourceBuilder();
            searchSourceBuilder2.query(builder)
                    .sort("id", SortOrder.ASC)
                    .sort("publish_time", SortOrder.ASC)
                    .size(10);
            //存储上一次分页的sort信息
            searchSourceBuilder2.searchAfter(result.getSortValues());
            SearchRequest searchRequest2 = new SearchRequest("demo_index").source(searchSourceBuilder2);
            SearchResponse searchResponse2 = restHighLevelClient.search(searchRequest2, RequestOptions.DEFAULT);
            SearchHit[] hits2 = searchResponse2.getHits().getHits();
        } catch (IOException e) {
            log.error("doc分页查询异常:{} ", e);
        }
    }