把Elasticsearch作为时间序列数据库使用

这篇文章算是对另一篇《Elasticsearch as a Time Series Data Store》的简单翻译吧，自己的理解吧。

• 首先_source被关闭了，这样原始的json文档不会被重复存储一遍。
• 其次_all也被关闭了。而且每个字段的store都是False，也就是不会单独被存储。
• 这些都关掉了，那么数据存哪里了？存在doc_values里。doc_values用于在做聚合运算的时候，根据一批文档id快速找到对应的列的值。doc_values在磁盘上一个按列压缩存储的文件，非常高效。

curl -XPOST http://172.16.18.116:9200/test -d '
{
    "settings": { "number_of_shards": 1, "number_of_replicas": 0, "index.query.default_field": "timestamp", "index.mapping.ignore_malformed": false, "index.mapping.coerce": false, "index.query.parse.allow_unmapped_fields": false },
    "mappings": { "test": { "_source": {"enabled": false}, "_all": {"enabled": false}, "properties": { "timestamp": { "type": "date", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } }, "appid": { "type": "string", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } }, "result": { "type": "string", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } }, "cmdid": { "type": "string", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } }, "optime": { "type": "integer", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } }, "total_count": { "type": "integer", "index": "no", "store": false, "dynamic": "strict", "doc_values": true, "fielddata": { "format": "doc_values" } } } } } }'

增加一条数据：

curl -XPOST http://172.16.18.116:9200/test/test/1 -d '
{
    "timestamp": 53534543,
    "appid": 1,
    "result": "test",
    "cmdid": "test",
    "optime": 53534543,
    "total_count": 100 }
'

查询一下：

curl -XGET http://172.16.18.116:9200/test/test/_search
{
    "took": 1,
    "timed_out": false,
    "_shards": { "total": 1, "successful": 1, "failed": 0 },
    "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": "test", "_type": "test", "_id": "1", "_score": 1 } ] } }

能查到数据，但是看不到原始字段内容，因为没存储也没索引，但是doc_values=true，实际上是保存到了磁盘上的

下面做一下聚合操作：

curl -XPOST http://172.16.18.116:9200/test/test/_search
{
    "aggs": { "timestamp": { "terms": { "field": "timestamp" }, "aggs": { "total_count": {"sum": {"field": "total_count"}} } } } }

结果：

{
    "took": 2,
    "timed_out": false,
    "_shards": { "total": 1, "successful": 1, "failed": 0 },
    "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": "test", "_type": "test", "_id": "1", "_score": 1 } ] },
    "aggregations": { "timestamp": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": 53534543, "key_as_string": "1970-01-01T14:52:14.543Z", "doc_count": 1, "total_count": { "value": 100 } } ] } } }

可以看到聚合操作可以获取到total_count值。