es中springboot的两种连接方式
使用客户端调用es服务的两种方式
有两种方式,一种 是9200端口或叫rest 接口，
另一种是用 节点的9300端口或叫 TransportClient
说明
es的官网中https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-api.html
We plan on deprecating the TransportClient in Elasticsearch 7.0 and removing it completely in 8.0. Instead, you should be using the Java High Level REST Client, which executes HTTP requests rather than serialized Java requests. The migration guidedescribes all the steps needed to migrate.
The Java High Level REST Client currently has support for the more commonly used APIs, but there are a lot more that still need to be added. You can help us prioritise by telling us which missing APIs you need for your application by adding a comment to this issue: Java high-level REST client completeness.
Any missing APIs can always be implemented today by using the low level Java REST Client with JSON request and response bodies.
简单点说就是 TransportClient 在es7后不支持了,所以你要用 java high level rest client，这个是用http的请求)
所以 ，后面都 是采用9200端口的方式！！！
官网中会说到low-level REST client f High Level REST Client,这两个震得了是rest方式,替换 transportclient方式的!!!
aws 连elasticsearch 的 坑
项目 中是使用 transport来调用es的，但是在aws中，这种 方式是不支持的!!!,要用9200的方式
具体情况:
https://forums.aws.amazon.com/thread.jspa?messageID=683536
不过这个还好，官网也说了，之后高版本会放弃这个,所以早做改变也是OK的.
org.springframework.data.elasticsearch.core中的ElasticsearchOperations 是一个接口，可以多种 实现!
使用springframework.data的情况(Transport), 不建议使用，请查看下一章.
怎么连es，使用第一种 方式
这边是使用spring boot的方式
org.springframework.boot
spring-boot-starter-data-elasticsearch
spring对es地址及相关配置.
spring.data.elasticsearch.cluster-nodes=xxxx:9300,xxx:9301
spring.data.elasticsearch.cluster-name=b_cluster
怎么使用
直接引用即可
聚合,
//聚合操作
TermsAggregationBuilder tbuild = AggregationBuilders.terms("sa_groupid").field("said");
AggregationBuilder sab = AggregationBuilders.stats("a_stat").field("player_count");
tbuild.subAggregation(sab);
tbuild.size(saidlist.size());
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices("aaa_online_log").withTypes("doc").withSearchType(SearchType.DEFAULT)
.addAggregation(tbuild)
.build();
Aggregations aggregations = elasticsearchTemplate.query(searchQuery, new ResultsExtractor() {
@Override
public Aggregations extract(SearchResponse response) {
return response.getAggregations();
}
});
//聚合的内容获取!
MultiBucketsAggregation targetAggregation = aggregations.get("sa_groupid");
Map sa_stat = new HashMap<>();
targetAggregation.getBuckets().forEach(tmpbuck ->{
DAggOnlineStat dAggOnlineStat = new DAggOnlineStat();
dAggOnlineStat.setApp_id(((Long)tmpbuck.getKey()).intValue());
dAggOnlineStat.setDoc_count((int) tmpbuck.getDocCount());
Aggregation tmpa = tmpbuck.getAggregations().getAsMap().get("a_stat");
Stats tmpstats = (Stats) tmpa;
dAggOnlineStat.setAvg(tmpstats.getAvg());
dAggOnlineStat.setMax(((Double)tmpstats.getMax()).intValue());
dAggOnlineStat.setMin(((Double)tmpstats.getMin()).intValue());
sa_stat.put(dAggOnlineStat.getApp_id(), dAggOnlineStat);
});
return sa_stat;
查询单表
String indexName = String.format("a_ooo_price_%s_log", currency_flag);
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(new TermQueryBuilder("appid", appid));
//按时间从早到晚排序
SortBuilder sortBuilder = new FieldSortBuilder("current_time").order(SortOrder.ASC);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices(indexName).withTypes("doc").withSort(sortBuilder)
.build();
SearchHits searchHits = elasticsearchTemplate.query(searchQuery, new ResultsExtractor() {
@Override
public SearchHits extract(SearchResponse response) {
return response.getHits();
}
});
List sfoogamePriceLogs = new ArrayList<>();
for (SearchHit searchHit : searchHits) {
Map hitMap = searchHit.getSource();
sfoogamePriceLog sfoogamePriceLog = new sfoogamePriceLog();
sfoogamePriceLog.setId(searchHit.getId());
sfoogamePriceLog.setDiscount_percent((Integer) hitMap.get("discount_percent"));
sfoogamePriceLog.setUpdated((Long) hitMap.get("updated"));
sfoogamePriceLog.setCdest((Integer) hitMap.get("cdest"));
sfoogamePriceLog.setInitial((Integer) hitMap.get("initial"));
sfoogamePriceLog.setCurrent_time((Long) hitMap.get("current_time"));
sfoogamePriceLog.setCdate((String) hitMap.get("cdate"));
sfoogamePriceLog.setAppid((Integer) hitMap.get("appid"));
sfoogamePriceLog.setCurrency_flag((String) hitMap.get("currency_flag"));
sfoogamePriceLogs.add(sfoogamePriceLog);
}
return sfoogamePriceLogs;
查询percent
PercentilesAggregationBuilder tbuild = AggregationBuilders.percentiles("b_sta").field("score");
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(new TermQueryBuilder("language", "english"));
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices("sfoogame_detail").withTypes("doc").withSearchType(SearchType.DEFAULT)
.addAggregation(tbuild)
.build();
Aggregations aggregations = elasticsearchTemplate.query(searchQuery, new ResultsExtractor() {
@Override
public Aggregations extract(SearchResponse response) {
return response.getAggregations();
}
});
InternalTDigestPercentiles tmp = aggregations.get("b_sta");
Double percentile_val = tmp.percentile(50);
return percentile_val;
怎么连jest与使用,即9200端口.
官网:
https://github.com/VanRoy/spring-data-jest
官网使用的是xml的配置
由于我们是使用springboot的,上面的配置可以不用，
我们使用注解的方式进行配置
@Value("${project.es.serveruri}")
public String serverUri ;
@Bean
public io.searchbox.client.config.HttpClientConfig.Builder jestClientConfigBuilder() {
io.searchbox.client.config.HttpClientConfig.Builder builder = new io.searchbox.client.config.HttpClientConfig.Builder(serverUri);
return builder;
}
@Bean
public HttpClientConfig jestClientConfig(@Qualifier("jestClientConfigBuilder") io.searchbox.client.config.HttpClientConfig.Builder builder) {
return builder.build();
}
@Bean
public io.searchbox.client.JestClientFactory jestClientFactory(@Qualifier("jestClientConfig") HttpClientConfig httpClientConfig){
io.searchbox.client.JestClientFactory factory = new io.searchbox.client.JestClientFactory();
factory.setHttpClientConfig(httpClientConfig);
return factory;
}
@Bean
public JestClient jestClient(@Qualifier("jestClientFactory")JestClientFactory jestClientFactory) {
return jestClientFactory.getObject();
}
@Bean
public com.github.vanroy.springdata.jest.JestElasticsearchTemplate jestElasticsearchTemplate(@Qualifier("jestClient")JestClient jestClient){
JestElasticsearchTemplate jestElasticsearchTemplate = new JestElasticsearchTemplate(jestClient);
return jestElasticsearchTemplate;
}
jest怎么使用?
@Autowired
private JestElasticsearchTemplate elasticsearchTemplate;
直接使用是OK的，不过我找了很久都 有发现有靠谱的文档，对于使用template的资料很少，只能自己摸索,
使用template查单表数据
String indexName = "aaa_index"
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(new TermQueryBuilder("aid", aid));
//按时间从早到晚排序
SortBuilder sortBuilder = new FieldSortBuilder("current_time").order(SortOrder.ASC);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices(indexName).withTypes("doc").withSort(sortBuilder)
.build();
List> searchHits = elasticsearchTemplate.query(searchQuery, new JestResultsExtractor>>() {
@Override
public List> extract(SearchResult searchResult) {
return searchResult.getHits(AAA.class);
}
});
List aaas = new ArrayList<>();
for (SearchResult.Hit searchHit : searchHits) {
AAA aaa = searchHit.source;
}
AAA.class是一个自定义的pojo类,用来装内容的.
查max
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
TermsQueryBuilder queryBuilder1 = new TermsQueryBuilder("aid",String.valueOf(aid));
boolQueryBuilder.must(queryBuilder1);
MaxAggregationBuilder tbuild = AggregationBuilders.max("count_max").field("player_count");
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices("project_online_peak").withTypes("doc").withSearchType(SearchType.DEFAULT)
.addAggregation(tbuild)
.build();
MetricAggregation aggregations = elasticsearchTemplate.query(searchQuery, new JestResultsExtractor() {
@Override
public MetricAggregation extract(SearchResult searchResult) {
return searchResult.getAggregations();
}
});
Double count_max_double = null;
try {
// count_max_double = ((InternalMax)aggregations.get("count_max")).value();
count_max_double = aggregations.getMaxAggregation("count_max").getMax();
} catch (Exception e) {
log.error("count_max获取失败", e);
}
return count_max_double.intValue();
查percentiles数据
PercentilesAggregationBuilder tbuild = AggregationBuilders.percentiles("b_sta").field("score");
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(new TermQueryBuilder("language", "english"));
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQueryBuilder).withIndices("sfoogame_detail").withTypes("doc").withSearchType(SearchType.DEFAULT)
.addAggregation(tbuild)
.build();
MetricAggregation aggregations = elasticsearchTemplate.query(searchQuery, new JestResultsExtractor() {
@Override
public MetricAggregation extract(SearchResult searchResult) {
return searchResult.getAggregations();
}
});
// InternalTDigestPercentiles tmp = aggregations.get("b_sta");
Map tmp = aggregations.getPercentilesAggregation("b_sta").getPercentiles();
Double targetVal = null;
for (String percentilestr : tmp.keySet()) {
Double tdouble = new Double(percentilestr);
if(75 == tdouble.intValue()){ // %50以上
targetVal = tmp.get(percentilestr);
}
}
terms聚合
QueryBuilder queryBuilder = QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("cdatestr", "20170912"));
//聚合操作
TermsAggregationBuilder tbuild = AggregationBuilders.terms("sa_groupid").field("appid");
AggregationBuilder sab = AggregationBuilders.stats("a_stat").field("player_count");
tbuild.subAggregation(sab);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(queryBuilder).withIndices("sa_online_log").withTypes("doc").withSearchType(SearchType.DEFAULT)
.addAggregation(tbuild)
.build();
MetricAggregation aggregations = elasticsearchTemplate.query(searchQuery, new JestResultsExtractor() {
@Override
public MetricAggregation extract(SearchResult response) {
return response.getAggregations();
}
});
//聚合的内容获取!
List tmpList = aggregations.getTermsAggregation("sa_groupid").getBuckets();
Map sa_stat = new HashMap<>();
for (TermsAggregation.Entry entry : tmpList) {
System.out.println(entry);
DAggOnlineStat dAggOnStat = new DAggOnlineStat();
dAggOnStat.setApp_id(Integer.valueOf(entry.getKey()));
dAggOnStat.setDoc_count(entry.getCount().intValue());
StatsAggregation statsAggregation = entry.getStatsAggregation("a_stat");
dAggOnStat.setAvg(statsAggregation.getAvg());
dAggOnStat.setMax(statsAggregation.getMax().intValue());
dAggOnStat.setMin(statsAggregation.getMin().intValue());
sa_stat.put(dAggOnStat.getApp_id(), dAggOnStat);
}
System.out.println("-----------");
System.out.println(sa_stat);
两种的一些使用差异.
在有些方法，会出现unsupportedOpertationException.对这些方法，我们要区别去改代码…【竟然不是在接口层去封装而是改业务代码，真是诡异】,
调这些方法会抛错
java.lang.UnsupportedOperationException
at com.github.vanroy.springdata.jest.JestElasticsearchTemplate.query(JestElasticsearchTemplate.java:413)
现在来看，换个方法即可
参考:
https://help.aliyun.com/document_detail/69194.html?spm=a2c4g.11186623.6.552.45263075k2X9j5
如果要连aws，可参考:
https://blog.csdn.net/mingover/article/details/82984579