博客地址：http://www.moonxy.com
一、前言
Elasticsearch 是一个分布式的全文搜索引擎，索引和搜索是 Elasticsarch 的基本功能。同时，Elasticsearch 的聚合（Aggregations）功能也时分强大，允许在数据上做复杂的分析统计。ES 提供的聚合分析功能主要有指标聚合、桶聚合、管道聚合和矩阵聚合。需要主要掌握的是前两个，即指标聚合和桶聚合。
聚合分析的官方文档：Aggregations
二、聚合分析
2.1 指标聚合
指标聚合官网文档：Metric
指标聚合中主要包括 min、max、sum、avg、stats、extended_stats、value_count 等聚合，相当于 SQL 中的聚合函数。
指标聚合中包括如下聚合：
- Avg Aggregation
- Cardinality Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Value Count Aggregation
Aggregations that keep track and compute metrics over a set of documents.
在一组文档中跟踪和计算度量的聚合。如下以 max 聚合为例：
Max Aggregation
max 聚合官网文档：Max Aggregation
max 聚合用于最大值统计，与 SQL 中的聚合函数 max() 的作用类似，其中 "max_price" 为自定义的聚合名称。
##Max Aggregation GET books/_search { "size": 0, "aggs": { "max_price": { "max": { "field": "price" } } } }
返回结果如下：
{ "took": 6, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "max_price": { "value": 81.4 } } }
Cardinality Aggregation
基数统计聚合官网文档：Cardinality Aggregation
Cardinality Aggregation 用于基数查询，其作用是先执行类似 SQL 中的 distinct 操作，去掉集合中的重复项，然后统计排重后的集合长度。
##Cardinality Aggregation GET books/_search { "size": 0, "aggs": { "all_language": { "cardinality": { "field": "language" } } } }
返回结果如下：
{ "took": 41, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "all_language": { "value": 3 } } }
Stats Aggregation
基本统计聚合官网文档：Stats Aggregation
Stats Aggregation 用于基本统计，会一次返回 count、max、min、avg 和 sum 这 5 个指标。如下：
##Stats Aggregation GET books/_search { "size": 0, "aggs": { "stats_pirce": { "stats": { "field": "price" } } } }
返回结果如下：
{ "took": 5, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "stats_pirce": { "count": 5, "min": 46.5, "max": 81.4, "avg": 63.8, "sum": 319 } } }
Extended Stats Aggregation
高级统计聚合官网文档：Extended Stats Aggregation
用于高级统计，和基本统计功能类似，但是会比基本统计多4个统计结果：平方和、方差、标准差、平均值加/减两个标准差的区间。
##Extended Stats Aggregation GET books/_search { "size": 0, "aggs": { "extend_stats_pirce": { "extended_stats": { "field": "price" } } } }
返回响应结果：
{ "took": 14, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "extend_stats_pirce": { "count": 5, "min": 46.5, "max": 81.4, "avg": 63.8, "sum": 319, "sum_of_squares": 21095.46, "variance": 148.65199999999967, "std_deviation": 12.19229264740638, "std_deviation_bounds": { "upper": 88.18458529481276, "lower": 39.41541470518724 } } } }
Value Count Aggregation
文档数量聚合官网文档：Value Count Aggregation
Value Count Aggregation 可按字段统计文档数量。
##Value Count Aggregation GET books/_search { "size": 0, "aggs": { "doc_count": { "value_count": { "field": "author" } } } }
返回结果如下：
{ "took": 6, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "doc_count": { "value": 5 } } }
注意：
text 类型的字段不能做排序和聚合（terms Aggregation 除外），如下对 title 字段做聚合，title 定义为 text：
GET books/_search { "size": 0, "aggs": { "doc_count": { "value_count": { "field": "title" } } } }
返回结果如下：
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [title] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "books", "node": "6n3douACShiPmlA9j2soBw", "reason": { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [title] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead." } } ] }, "status": 400 }
2.2 桶聚合
桶聚合官网文档：Bucket Aggregations
Bucket 可以理解为一个桶，它会遍历文档中的内容，凡是符合某一要求的就放入一个桶中，分桶相当与 SQL 中 SQL 中的 group by。
桶聚合包括如下聚合：
- Adjacency Matrix Aggregation
- Children Aggregation
- Composite Aggregation
- Date Histogram Aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Range Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
terms Aggregation 用于分组聚合，统计属于各编程语言的书籍数量，如下：
GET books/_search { "size": 0, "aggs": { "terms_count": { "terms": { "field": "language" } } } }
返回结果如下：
{ "took": 31, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "terms_count": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "java", "doc_count": 2 }, { "key": "python", "doc_count": 2 }, { "key": "javascript", "doc_count": 1 } ] } } }
在 terms 分桶的基础上，还可以对每个桶进行指标聚合。例如，想统计每一类图书的平局价格，可以先按照 language 字段进行 Terms Aggregation，再进行 Avg Aggregattion，查询语句如下：
GET books/_search { "size": 0, "aggs": { "terms_count": { "terms": { "field": "language" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } }
返回结果如下：
{ "took": 8, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "terms_count": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "java", "doc_count": 2, "avg_price": { "value": 58.35 } }, { "key": "python", "doc_count": 2, "avg_price": { "value": 67.95 } }, { "key": "javascript", "doc_count": 1, "avg_price": { "value": 66.4 } } ] } } }
Range Aggregation
Range Aggregation 是范围聚合，用于反映数据的分布情况。比如，对 books 索引中的图书按照价格区间在 0~50、50~80、80 以上进行范围聚合，如下：
GET books/_search { "size": 0, "aggs": { "price_range": { "range": { "field": "price", "ranges": [ {"to": 50}, {"from": 50, "to": 80}, {"from": 80} ] } } } }
返回结果如下：
{ "took": 16, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 5, "max_score": 0, "hits": [] }, "aggregations": { "price_range": { "buckets": [ { "key": "*-50.0", "to": 50, "doc_count": 1 }, { "key": "50.0-80.0", "from": 50, "to": 80, "doc_count": 3 }, { "key": "80.0-*", "from": 80, "doc_count": 1 } ] } } }
Range Aggregation 不仅可以对数值型字段进行范围统计，也可以作用在日期类型上。Date Range Aggregation 专门用于日期类型的范围聚合，和 Range Aggregation 的区别在于日期的起止值可以使用数学表达式。
2.3 管道聚合
管道聚合官网文档：Pipeline Aggregations
- Avg Bucket Aggregation
- Derivative Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Cumulative Sum Aggregation
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Serial Differencing Aggregation
Pipeline Aggregations 处理的对象是其他聚合的输出（而不是文档）。
2.4 矩阵聚合
矩阵聚合官网文档：Matrix Aggregations
- Matrix Stats
Matrix Stats 聚合是一种面向数值型的聚合，用于计算一组文档字段中的以下统计信息：
计数：计算过程中每种字段的样本数量；
平均值：每个字段数据的平均值；
方差：每个字段样本数据偏离平均值的程度；
偏度：量化每个字段样本数据在平均值附近的非对称分布情况；
峰度：量化每个字段样本数据分布的形状；
协方差：一种量化描述一个字段数据随另一个字段数据变化程度的矩阵；
相关性：描述两个字段数据之间的分布关系，其协方差矩阵取值在[-1,1]之间。
主要用于计算两个数值型字段之间的关系。如对日志记录长度和 HTTP 状态码之间关系的计算。
GET /_search { "aggs": { "statistics": { "matrix_stats": { "fields": ["log_size", "status_code"] } } } }