java 对es的操作

1.创建连接：

        SearchRequest request = new SearchRequest(GlobalConstant.INDEX_NAME_NEW);
                SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
                 request.source(searchSourceBuilder);
        RestHighLevelClient client = null;
        try {
            client = getNewClient();
            SearchResponse search = client.search(request, RequestOptions.DEFAULT);
            Aggregations aggregations = search.getAggregations();
            Terms byCompanyAggregation2 = aggregations.get("by_code");

 //新的ES地址
    public static volatile String CLUSTER_NODES_NEW;
    //新的ES地址端口
    public static volatile Integer CLUSTER_NODES_NEW_PORT;
    //新的ES日志索引
    public static volatile String INDEX_NAME_NEW;
    //报文日志索引
    public static volatile String API_MSG_INDEX;
    //新的ES登录账号
    public static volatile String ES_USER_NAME;
    //新的ES地址登录密码
    public static volatile String ES_USER_PWD;

public static RestHighLevelClient getNewClient(){
        HttpHost host=new HttpHost(GlobalConstant.CLUSTER_NODES_NEW, GlobalConstant.CLUSTER_NODES_NEW_PORT, HttpHost.DEFAULT_SCHEME_NAME);
        RestClientBuilder builder= org.elasticsearch.client.RestClient.builder(host);
        builder.setMaxRetryTimeoutMillis(300 * 1000);
        CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(GlobalConstant.ES_USER_NAME, GlobalConstant.ES_USER_PWD));
        builder.setHttpClientConfigCallback(f -> f.setDefaultCredentialsProvider(credentialsProvider));
        RestClientBuilder.RequestConfigCallback configCallback = new RestClientBuilder.RequestConfigCallback() {
            @Override
            public org.apache.http.client.config.RequestConfig.Builder customizeRequestConfig(org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder
                        .setConnectTimeout(30000)
                        .setSocketTimeout(300 * 1000);//更改客户端的超时限制默认30秒现在改为5分钟
            }
        };
        builder.setRequestConfigCallback(configCallback);
        RestHighLevelClient restClient = new RestHighLevelClient( builder);
        return restClient;
    }

2.sql查询

2.1设置查询缓存大小：

默认时100mb下面设置成500

 RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
                    builder.setHttpAsyncResponseConsumerFactory(
                            new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));
                    RequestOptions COMMON_OPTIONS = builder.build();
                    response = newClient.search(request, COMMON_OPTIONS);

2.2设置查询出来的固定字段数据，超时时间

 //过滤字段
 String[] excludes = {};
                    String[] includes = {"logData.apiId", "logData.rawRequest", "logData.rawResponse", "logData.gatewayErrorCode", "logData.backendResponseCode"};
                    searchSourceBuilder.fetchSource(includes, excludes);
            // 限制查询出来的条数       searchSourceBuilder.query(boolQueryBuilder).size(LIMIT_SIZE);
                    searchSourceBuilder.timeout(TimeValue.timeValueSeconds(300));
               

2.3查询某个区间范围

  boolQueryBuilder.must(
                            QueryBuilders.rangeQuery("collectTime")
                                    .gte(beginDate.getTime())
                                    .lte(endDate.getTime()));

2.4查询某个并的数据

                    boolQueryBuilder.must(QueryBuilders.matchQuery("busiLogType", "eopori"));

2.5 查询含有某个字段和不含有某个字段

一下意思时 存在logData.gatewayResponseCode数据和不存在
logData.gatewayErrorCode数据或者为空

    boolQueryBuilder.must(QueryBuilders.wildcardQuery("logData.gatewayResponseCode", "*"));
                    boolQueryBuilder.mustNot(QueryBuilders.wildcardQuery("logData.gatewayErrorCode", "*"));

以上完整可以es的sql为

{
    "size":120,
    "timeout":"300s",
    "query":{
        "bool":{
            "must":[
                {
                    "match":{
                        "busiLogType":{
                            "query":"eopori",
                            "operator":"OR",
                            "prefix_length":0,
                            "max_expansions":50,
                            "fuzzy_transpositions":true,
                            "lenient":false,
                            "zero_terms_query":"NONE",
                            "auto_generate_synonyms_phrase_query":true,
                            "boost":1
                        }
                    }
                },
                {
                    "range":{
                        "collectTime":{
                            "from":1647142201124,
                            "to":1649820601124,
                            "include_lower":true,
                            "include_upper":true,
                            "boost":1
                        }
                    }
                },
                {
                    "match":{
                        "logData.apiId":{
                            "query":"10240",
                            "operator":"OR",
                            "prefix_length":0,
                            "max_expansions":50,
                            "fuzzy_transpositions":true,
                            "lenient":false,
                            "zero_terms_query":"NONE",
                            "auto_generate_synonyms_phrase_query":true,
                            "boost":1
                        }
                    }
                },
                {
                    "wildcard":{
                        "logData.gatewayResponseCode":{
                            "wildcard":"*",
                            "boost":1
                        }
                    }
                }
            ],
            "must_not":[
                {
                    "wildcard":{
                        "logData.gatewayErrorCode":{
                            "wildcard":"*",
                            "boost":1
                        }
                    }
                }
            ],
            "adjust_pure_negative":true,
            "boost":1
        }
    },
    "_source":{
        "includes":[
            "logData.apiId",
            "logData.rawRequest",
            "logData.rawResponse",
            "logData.gatewayErrorCode",
            "logData.backendResponseCode"
        ],
        "excludes":[

        ]
    }
}

2.6 or查询和and包含or

should就是or mus 就是and wildcardQuery就是模糊查询

 BoolQueryBuilder requestQueryBuilder = QueryBuilders.boolQuery();
            requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiId.keyword", "*" + apiId + "*"));
            requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiName.keyword", "*" + apiId + "*"));
            boolQueryBuilder.must(requestQueryBuilder);

2.7 分页查询

from 就是从哪里开始查询
size：就是查询数据
sort 就是按照logData.timestamp时间降序排序

  searchSourceBuilder
                .size(size)
                .from(from)
                .trackTotalHits(true)
                .query(
                        boolQueryBuilder
                )
                .sort("logData.timestamp", SortOrder.DESC);

2.8 分组

关键词：aggregation
按照：busiLogType分组数量查查询
别名是：by_busiLogType
统计分组的数据

        AggregationBuilder by_busiLogType = AggregationBuilders.terms("by_busiLogType").field("busiLogType").size(10000);
searchSourceBuilderCount.query(
                boolQueryBuilder
        ).aggregation(
                by_busiLogType
        ).size(0);

3.查询结果获取数据

将其转为map获取某个字段的数据在这里插入代码片

 SearchHit[] results = search.getHits().getHits();
            for (SearchHit hit : results) {
                Map<String, Object>
                        map = hit.getSourceAsMap();
                Map<String, Object> maplogData = (Map<String, Object>) map.get("logData");
                maps.add(maplogData);
            }

将其转为字符串在转为json对象字符串

 if (response.status().equals(RestStatus.OK) && !response.isTimedOut()) {
                        SearchHits hits = response.getHits();
                        List<ApiMsg> apiMsgs = new ArrayList<>();
                        for (SearchHit hit : hits) {
                            String apiMsgStr = hit.getSourceAsString();
                            try {
                                String logDataStr = JSON.parseObject(apiMsgStr).getJSONObject("logData").toJSONString();
                     

获取分组的统计数据：

            Aggregations aggregations = search.getAggregations();
            Terms byCompanyAggregation = aggregations.get("by_busiLogType");
            for (Terms.Bucket bucket2 : byCompanyAggregation.getBuckets()) {
                if (bucket2.getKey() != null && !StringUtils.isEmpty(bucket2.getKey().toString().trim()) && "eoplog".equals(bucket2.getKey().toString())) {
                    //总量
                    total = bucket2.getDocCount();
                }
            }

4.引用es包

    
        org.elasticsearch.client
        transport
        6.5.0
    
    
        org.elasticsearch.client
        elasticsearch-rest-client
        6.5.0
    
    
        org.elasticsearch.client
        elasticsearch-rest-high-level-client
        6.5.0
    

5. elastic相关操作

5.1查询

GET aiolog-eop-eop-comm-eopori-orderquery-2022.04.02/_search

5.2 插入

          POST  aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12/_doc
{}

5.3 删除

DELETE aiolog-eop-eop-comm-eopori-orderquery-2022.03.21

5.4查看模板

右键点击
aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12
{}

5.5插入模板

PUT /_template/aiolog-eop-eop
{
“order”: 1,
“index_patterns”: [
“grp-eop3-full-*”
],
“settings”: {},
}

5.6 查询每小时，每分钟，12小时之前每小时统计数据

一下例子统计前12小时，每小时的数据

{
    "size":0,
    "query":{
        "bool":{
            "must":[
                {
                    "range":{
                        "collectTime":{
                            "from":1676602231188,
                            "to":1676645431188,
                            "include_lower":false,
                            "include_upper":true,
                            "boost":1
                        }
                    }
                }
            ],
            "adjust_pure_negative":true,
            "boost":1
        }
    },
    "aggregations":{
        "collectTime":{
            "date_histogram":{
                "field":"collectTime",
                "format":"HH:mm",
                "time_zone":"+08:00",
                "interval":"1h",
                "offset":-569811,
                "order":{
                    "_key":"asc"
                },
                "keyed":false,
                "min_doc_count":0,
                "extended_bounds":{
                    "min":1676602231188,
                    "max":1676645431188
                }
            }
        }
    }
}

  SearchRequest request = new SearchRequest(GlobalConstant.LOGIN_LOG_INDEX);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        searchSourceBuilder.query(
        QueryBuilders.boolQuery()
                .must(QueryBuilders.rangeQuery("collectTime")
                        .gt(startTime.getTime())
                        .lte(endTime.getTime())));

        DateHistogramAggregationBuilder collectTimeAgg = AggregationBuilders.dateHistogram("collectTime")
                .dateHistogramInterval(DateHistogramInterval.HOUR).format("YYYY-MM-dd HH:mm:ss")
                .timeZone(DateTimeZone.forID("+08:00")).offset(-betterTime).minDocCount(0)
                .extendedBounds(extendedBounds)
                .field("collectTime");
        searchSourceBuilder.aggregation(collectTimeAgg).size(0);
        request.source(searchSourceBuilder);
        RestHighLevelClient client = null;
        try {
            client = getNewClient();
            SearchResponse search = client.search(request, RequestOptions.DEFAULT);
            Aggregations aggregations = search.getAggregations();
            ParsedDateHistogram parsedDateHistogram = aggregations.get("collectTime");
            //系统ID
            for (Histogram.Bucket bucket : parsedDateHistogram.getBuckets()) {
                Date date = DateUtil.parse(bucket.getKeyAsString());
               Date minuteDate= DateUtil.offsetHour(date,1);
               String time = DateUtil.format(minuteDate, "HH:mm");
                StatisticLoginCount statisticLoginCount=StatisticLoginCount.builder()
                        .time(time)
                        .count(bucket.getDocCount())
                        .build();
                statisticLoginCounts.add(statisticLoginCount);
            }
            if(CollectionUtils.isNotEmpty(statisticLoginCounts)&&statisticLoginCounts.size()>12){
                for(int i=statisticLoginCounts.size()-12;i<statisticLoginCounts.size();i++){
                    statisticLoginCountList.add(statisticLoginCounts.get(i));
                }
            }
        } catch (Exception e) {
            log.error(e.toString());
        }finally {
            client.close();
        }

collectTime 是统计的时间字段 range的是规定从某段时间到某段时间查询范围，format是时间格式-interval是没1小时，offset当前时间的每一小时1分向前便宜开始统计，extended_bounds是统计某段时间到某段时间的值，min_doc_count最小值

注意：在这里插入代码片
number_of_shards和number_of_replicas es配置的分区数量
如果只有一个设置一

5.6 es的安装可以参考

https://blog.csdn.net/happyzxs/article/details/89156068
https://www.cnblogs.com/xyddm/p/14188842.html
https://www.elastic.co/cn/downloads/past-releases/#kibana
https://www.cnblogs.com/wwjj4811/p/14700279.html
https://blog.csdn.net/qq_43676531/article/details/113095349
https://blog.csdn.net/he19970408/article/details/107359861/
https://blog.csdn.net/weixin_39887748/article/details/112481376