ElasticSearch结合springboot常规用法总结
ES结合springboot的常规用法总结：
引jar、配置工具类，客户端等忽略。
建议先熟悉利用kibana查询ES的语句，那样写起来更容易，也方便问题排查。
一、es查询必须的几个对象：
1. 普通查询的条件容器：BoolQueryBuilder 多条件查询容器
BoolQueryBuilder builder=QueryBuilders.boolQuery(); 普通的QueryBuilders查询的boolQuery()方法，变成多条件查询
比如时间条件：boolQueryBuilder.filter(QueryBuilders.rangeQuery(EsConstant.ES_INSERT_TIME).gte(twoHoursInfo.get(0)).lte(twoHoursInfo.get(1)));2. 分组查询的条件容器：AggregationBuilder
AggregationBuilder field = AggregationBuilders.terms(AGG_NAME).field(Constant.ES_TITLE);3. 普通的分组查询：
注意如果要分组的字段是text类型，需要转成key类型才能分组，直接字段.key就行。在分组后直接.subAggregation(。。。。。) 可以添加子分组。
查询条件总容器：SearchSourceBuilder
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
比如：searchSourceBuilder.query(boolQueryBuilder).aggregation(termsAggregationBuilder);4. 创建查询容器：SearchRequest
方式一：SearchRequest searchRequest = new SearchRequest(INDEX_NAME).source(searchSourceBuilder);
方式二：SearchRequest searchRequest = new SearchRequest().indices(INDEX_NAME).source(searchSourceBuilder);
5. 放入提前配置好的客户端：
SearchResponse是返回值的容器
SearchResponse search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);二、解析容器分为两种：
普通条件结果解析：
方式一:
TrendingResponse trendingResponse = new TrendingResponse(); //要装载结果的实体类
SearchHit[] hits = searchResponse.getHits().getHits();
int length = hits.length;
if(length>0){
Map sourceAsMap = hits[0].getSourceAsMap();
trendingResponse = modelMapper.map(sourceAsMap, TrendingResponse.class);
}
方式二：
SearchHit[] hits = searchResponse.getHits().getHits();
if(hits.length>0){
Arrays.asList(hits).stream().forEach(hit -> {
TrendingResponse trendingResponse = new TrendingResponse(); //要装载结果的实体类
Map sourceAsMap = hit.getSourceAsMap();
trendingResponse = modelMapper.map(sourceAsMap, TrendingResponse.class);
trendingResponseList.add(trendingResponse);
});
}
分组条件结果解析：
方式一：
ParsedStringTerms aggregation = searchResponse1.getAggregations().get(AGG_NAME);
List buckets = aggregation.getBuckets();
for(Terms.Bucket bucket:buckets){
TrendingResponse trendingResponse = new TrendingResponse(); //要装载结果的实体类
trendingResponse.setPlatform(bucket.getKeyAsString()); //封装的解析方法
trendingResponseList.add(trendingResponse);
}
方式二：
ParsedTerms terms = response.getAggregations().get("termsAgg")
return terms.getBuckets().stream().map(it -> new Target(it.getKeyAsString(),it.getDocCount(),
dateHistogramAgg(it.getAggregations(), "yyyy-MM-dd HH:mm"))).collect(Collectors.toList()); 注意：多利用类的有参构造：直接转换成需要的类对象这是查询数据内容的方式，每次只能查出10000条，这是es默认的。如果想查出更多，可以采用滚动查询，这个以后再总结
三、日常用法：
1. 只是查询总数的写法：
logger.info("getAllCount 统计总数");
long value = 0;
List twoHoursInfo = new ArrayList<>();
twoHoursInfo.add(System.currentTimeMillis()-86400000);
twoHoursInfo.add(System.currentTimeMillis());
String[] indexs = new String[]{EsConstant.ES_INDEX_BASEDATA,EsConstant.ES_INDEX_BOARD,EsConstant.ES_INDEX_APPS,EsConstant.ES_INDEX_ACCOUNT};
CountRequest countRequest = new CountRequest(indexs,QueryBuilders.rangeQuery(EsConstant.ES_INSERT_TIME).gte(twoHoursInfo.get(0)).lte(twoHoursInfo.get(1)));
try {
CountResponse count = restHighLevelClient.count(countRequest, RequestOptions.DEFAULT);
value = count.getCount();
} catch (IOException e) {
logger.error("插入基础数据getMsgTypeData 报错:{}",e);
}
return value;
2.利用es的分词进行，匹配实现模糊查询：
boolQueryBuilder.filter(QueryBuilders.wildcardQuery(EsConstant.ES_PRODUCT_NAME,"*"+productName+"*"));
注意：ElasticSearchTemplate加中文筛选后无结果的解决办法：title.keyword
3. es实现简单的分页查询：(不推荐)
new SearchSourceBuilder().query(boolQueryBuilder).from(pageNum*pageSize).size(pageSize);4. 常规查询工具类
@Component
public class BaseSearchSourceBuilder extends BaseAggregationBuilder {
@Resource
private DetectionConfigRepository detectionConfigRepository;
public SearchSourceBuilder searchSourceBuilder(HeartbeatDetectionListVO condition){
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
Long startTime = null;
Long endTime = null;
if(Objects.nonNull(condition.getStartTime())){
startTime = condition.getStartTime().getTime();
endTime = condition.getEndTime().getTime();
}
if (Objects.nonNull(startTime) && Objects.nonNull(endTime)){
boolQueryBuilder.filter(QueryBuilders.rangeQuery(EsConstant.ES_DETECTION_TIME_STMP).gte(startTime).lte(endTime));
}
String keyword = condition.getStatus();
if (StringUtils.isNotBlank(keyword)){
long state = 0;
if(keyword.equals("ONLINE")){
state=1;
}else {
state=2;
}
//TODO 与或非
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(EsConstant.ES_RESULT_STATUS, state);
boolQueryBuilder.filter(termQueryBuilder);
}
String productCode = condition.getProductCode();
if (StringUtils.isNotBlank(productCode)){
boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_PRODUCT_CODE,productCode));
}
String resultCode = condition.getResultCode();
if (StringUtils.isNotBlank(resultCode)){
boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_RESULT_CODE,resultCode));
}
List messageList = condition.getMessageList();
if(CollectionUtils.isNotEmpty(messageList)){
boolQueryBuilder.filter(QueryBuilders.termsQuery(EsConstant.ES_RESULT_CODE,messageList));
}
String productName = condition.getProductName();
if (StringUtils.isNotBlank(productName)){
boolQueryBuilder.filter(QueryBuilders.wildcardQuery(EsConstant.ES_PRODUCT_NAME,"*"+productName+"*"));
}
String message = condition.getMessage();
// if (StringUtils.isNotBlank(message)){
// boolQueryBuilder.filter(QueryBuilders.queryStringQuery(message).defaultField(EsConstant.ES_RESULT_REASON));
// }
if (StringUtils.isNotBlank(message)){
boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_RESULT_CODE,message));
}
Integer pageSize = condition.getSize();
if (Objects.isNull(pageSize)){
pageSize = 0;
}
Integer pageNum = condition.getNumber();
if (Objects.isNull(pageNum)){
pageNum = 0;
}
return new SearchSourceBuilder().query(boolQueryBuilder).from(pageNum*pageSize).size(pageSize);
}
} 5. 聚合查询工具类：
public class BaseAggregationBuilder {
/**
* TopHitsAggregation sort <>pt asc
* @param topHitsName
* @return
*/
public TopHitsAggregationBuilder topHitsAggregationBuilder(String topHitsName, Integer size){
return AggregationBuilders.topHits(topHitsName).sort(EsConstant.ES_DETECTION_TIME, SortOrder.DESC).size(size);
}
/**
* TermsAggregation
* @param termsAggName
* @param fieldName
* @param size
* @return
*/
public TermsAggregationBuilder termsAggregationBuilder(String termsAggName, String fieldName, Integer size){
return AggregationBuilders.terms(termsAggName).field(fieldName).size(size).executionHint("map");
}
/**
* AggregationBuilders
* @param cardinalityAggName
* @param fieldName
* @param
* @return
*/
public CardinalityAggregationBuilder cardinalityAggregationBuilder(String cardinalityAggName, String fieldName){
return AggregationBuilders.cardinality(cardinalityAggName).field(fieldName);
}
/**
* DateHistogramAggregation
* @param dateHistogramFieldName
* @return
*/
public DateHistogramAggregationBuilder dateHistogramAggregationBuilder(String dateHistogramName, String dateHistogramFieldName){
return AggregationBuilders.dateHistogram(dateHistogramName).fixedInterval(new DateHistogramInterval("1d"))
.field(dateHistogramFieldName);
}
/**
* DateHistogramAggregation
* @param dateHistogramFieldName
* @return
*/
// public DateHistogramAggregationBuilder dateHistogramAggregationFormatBuilder(String dateHistogramName, String dateHistogramFieldName,
// DateHistogramInterval interval,Long startTime,Long endTime){
// return AggregationBuilders.dateHistogram(dateHistogramName).fixedInterval(interval)
// .field(dateHistogramFieldName).format("yyyy-MM-dd HH:mm").minDocCount(0).extendedBounds(new LongBounds(startTime,endTime)).timeZone(ZoneId.of("Asia/Shanghai"));
// }
/**
* SumAggregation
* @param sumAggName
* @param fieldName
* @return
*/
public SumAggregationBuilder sumAggregationBuilder(String sumAggName,String fieldName){
return AggregationBuilders.sum(sumAggName).field(fieldName);
}
/**
* AvgAggregation
* @param avgAggName
* @param fieldName
* @return
*/
public AvgAggregationBuilder avgAggregationBuilder(String avgAggName, String fieldName){
return AggregationBuilders.avg(avgAggName).field(fieldName);
}
/**
* MaxAggregation
* 最大值
*/
public MaxAggregationBuilder maxAggregationBuilder(String maxAggName, String fieldName){
return AggregationBuilders.max(maxAggName).field(fieldName);
}
/**
* MinAggregation
* 最小值
*/
public MinAggregationBuilder minAggregationBuilder(String maxAggName, String fieldName){
return AggregationBuilders.min(maxAggName).field(fieldName);
}
}6. 多查询并列查询：
当需要多条件查询的时候，尽量一条语句全部查出来，这样性能好很多，类似sql语句的
where (条件一 and 条件二 and 条件三) or (条件四 and 条件五 and 条件六) or 。。。
public List> getImageInfoEs(List listSearchImageInfo){
String[] indexs = new String[]{EsConstant.BASE_DATA,EsConstant.INDEX_APPS,EsConstant.INDEX_ACCOUNT,EsConstant.INDEX_ACCOUNT};
SearchSourceBuilder searchSourceBuilder = searchSourceBuilder(listSearchImageInfo);
SearchResponse search = esService.search(indexs, searchSourceBuilder);
SearchHit[] hits = search.getHits().getHits();
log.info("================从es查询的结果:hits.length : ==============={}",hits.length);
Map searchImageInfoMap = new HashMap<>();
listSearchImageInfo.stream().forEach(t->{
String key = t.getMedia_id()+""+t.getMsg_id_s()+""+t.getMsg_type();
searchImageInfoMap.put(key,t);
});
List> listMap = new ArrayList<>();
if(hits.length>0){
Arrays.asList(hits).stream().forEach(hit -> {
Map sourceAsMap = hit.getSourceAsMap();
String keylog = sourceAsMap.get(EsConstant.MEDIA_ID)+"------"+sourceAsMap.get(EsConstant.MSG_ID_S)+"-------"+sourceAsMap.get(EsConstant.MSG_TYPE);
log.info("-------------------------从es查询的结果--------------------{}",keylog);
sourceAsMap.put("_id",hit.getId());
sourceAsMap.put("index",hit.getIndex());
......业务代码处理
listMap.add(sourceAsMap);
});
}
return listMap;
}
public SearchSourceBuilder searchSourceBuilder(List listSearchImageInfo){
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
listSearchImageInfo.stream().forEach(t->{
//通过should进行多条件拼接
boolQueryBuilder.should(QueryBuilders.boolQuery().must(QueryBuilders.termQuery(EsConstant.MEDIA_ID,t.getMedia_id())).must(QueryBuilders.termQuery(EsConstant.MSG_ID_S,t.getMsg_id_s())).must(QueryBuilders.termQuery(EsConstant.MSG_TYPE,t.getMsg_type())));
});
return new SearchSourceBuilder().query(boolQueryBuilder).from(0).size(1000);
} 7. 只根据数据的es id查询，效率高
根据数据的id直接查询：
方式一：get方式不能用索引别名，但是效率高
public GetResponse getOne(String index, String docId) {
GetRequest getRequest = new GetRequest(index,docId);
getRequest.preference(prefrence());
GetResponse getResponse = null;
try {
getResponse = client.get(getRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
log.error("getOne fail,cause by {}", e.getCause());
}
return getResponse;
}
方式二：这种方式可以，但是套的层比get方式多，效率略不如。
//id可以是一个字符串类型的数组
public SearchResponse getOneById(String index, String docId) {
SearchRequest searchRequest = new SearchRequest(index);//指定索引
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.idsQuery().addIds(docId));
searchRequest.source(searchSourceBuilder);
SearchResponse search = null;
try {
search = client.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("getOne fail,cause by {}", e.getCause());
}
return search;
}