第一种解决方法(取固定的数量 不超过1万条):
1.代码如下:
// 操作类型 操作时间条件查询
if ( StringUtils.isNotBlank(operationType)&&startTime>0L&&endTime>0L) {
searchQuery = new NativeSearchQueryBuilder().withPageable(pageable).withQuery(QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("resName", "\"" + resName + "\""))//条件查询
.must(QueryBuilders.rangeQuery("operationTime").from(startTime).to(endTime))//时间范围查询
).build();
Page
long totalElements = searchPageResults.getTotalElements();
HashMap
map.put("totalElements", totalElements);
map.put("contentList", searchPageResults.getContent());
return map;
}
2.调用ElasticSearch做分页查询时报如下错误:
QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [10000] but was [666000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]; }
从上面的报错信息,可以看到ES提示我结果窗口太大了,目前最大值为10000,而我却要求给我666000。上网查了下大部分都是设置max_result_window的方法,全都是用crul或者http方式改的 这种感觉解决不了根本性的问题 考虑到我们在Es里面存放的是日志 几乎很少人去看 所以我后来我想到从结果集中取5000-10000条 但这样还是报那个错 这种用传统方式(from + size)查询占用内存空间且比较消耗时间,所以做了限制 经过一系列的百度获知 用scroll游标方式的去解决这个问题 还是限制取10000条 具体代码如下:
TransportClient client = null;
try {
client = new PreBuiltTransportClient(Settings.EMPTY)
.addTransportAddress(new TransportAddress(InetAddress.getByName("127.0.0.1"), 9300));//此IP随便写的本地的 具体根据项目ES配置来
} catch (UnknownHostException e) {
e.printStackTrace();
}
HashMap map = new HashMap();
SearchResponse scrollResp = client.prepareSearch("sysoplog")
.addSort(FieldSortBuilder.DOC_FIELD_NAME, SortOrder.ASC)
.setScroll(new TimeValue(6000))
.setQuery(QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("userName", "\"" + userName + "\""))
.must(QueryBuilders.matchQuery("operationType", "\"" + operationType + "\""))
.must(QueryBuilders.matchQuery("resName", "\"" + resName + "\""))
.must(QueryBuilders.rangeQuery("operationTime").from(startTime).to(endTime))
).setSize(10000).get();
List sysOperationLogList = new ArrayList<>();
for (SearchHit hit : scrollResp.getHits().getHits()) {
String sourceAsString = hit.getSourceAsString();
JSONObject object=JSONObject.parseObject(sourceAsString);
SysOperationLog sysOperationLog = object.toJavaObject(object, SysOperationLog.class);
sysOperationLogList.add(sysOperationLog);
}
//用工具类进行分页
List sysOperationLogList1= PageUtils.PageHelper(pageNumber, pageSize, sysOperationLogList);
map.put("totalElements", sysOperationLogList.size());
map.put("contentList", sysOperationLogList1);
return map;
//工具类如下:
public class PageUtils {
public static List PageHelper(int page,int rows,List list) {
int size = list.size();
int pageCount=size/rows;
int fromIndex = rows * (page - 1);
int toIndex = fromIndex + rows;
if (toIndex >= size) {
toIndex = size;
}
if(page>=pageCount+1){
fromIndex=0;
toIndex=0;
}
List sysOperationLogs = list.subList(fromIndex, toIndex);
return sysOperationLogs;
}
}
3.以上代码只能解决查询10000条的数据 .setSize(10000)设置超过10000条之后还是会报那样的错 百度了好久 也看了好久的ES API都没有找到解决办法 个人想要的结果是像数据库那样查询的效果
第二种解决方法(修改内存大小):
*/_settings?preserve_existing=true 请求:put
{
"max_result_window": "2000000000"
}