Java高级教程之ElasticSearch-11：ElasticSearch查询日志
系统产生的日志一般记录在*.log的文件中，比如tomcat的logs文件夹里各种日志文件，如localhost.log文件。日志数据一般有时间，文本信息等，这里可以使用_bulk批量初始化数据：
curl -X PUT "localhost:9200/user_logs_000001/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d'
{"index":{"_index" : "user_logs_000001", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" }
{"index":{"_index" : "user_logs_000001", "_id" : "2"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" }
{"index":{"_index" : "user_logs_000001", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }
'
日志数据查询出来的结果：
curl -X GET "localhost:9200/user_logs_000001/_search?format=txt&pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} }
}
'
{
"took" : 49,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-12-06T11:04:05.000Z",
"user" : {
"id" : "vlb44hny"
},
"message" : "Login attempt failed"
}
},
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
},
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-12-07T11:07:08.000Z",
"user" : {
"id" : "l7gk7f82"
},
"message" : "Logout successful"
}
}
]
}
}
日志索引包含3条记录，id分别为1，2，3，_type为_doc。
这里的_type可以自定义，比如_log，logtype，type1等。查询时也可以加入查询的字段，比如?q=user.id:8a4f500d只会查询出user字段中id为8a4f500d的记录：
curl -X GET "localhost:9200/user_logs_000001/_search?q=user.id:8a4f500d&pretty"
{
"took" : 126,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.9808291,
"hits" : [
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.9808291,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
}
]
}
}
这里返回的_source字段相当于一个数据对象，包含索引中对应的数据字段和值。
即使可以使用带参数的查询，官方还是推荐JSON数据体查询：
curl -X GET "localhost:9200/user_logs_000001/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"message": "successful"
}
}
}
'
查询出message包含成功登录successful的日志。
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.49917626,
"hits" : [
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
},
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:07:08.000Z",
"user" : {
"id" : "l7gk7f82"
},
"message" : "Logout successful"
}
}
]
}
}
多个索引查询用逗号：
curl -X GET "localhost:9200/user_logs_000001,user_logs_000002/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"message": "successful"
}
}
}
'
这里的user_logs_000002还没有创建，所以会报not found错误。
使用_reindex创建日志索引user_logs_000002，
curl -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "user_logs_000001"
},
"dest": {
"index": "user_logs_000002"
}
}
'
{
"took" : 4021,
"timed_out" : false,
"total" : 3,
"updated" : 0,
"created" : 3,
"deleted" : 0,
"batches" : 1,
"version_conflicts" : 0,
"noops" : 0,
"retries" : {
"bulk" : 0,
"search" : 0
},
"throttled_millis" : 0,
"requests_per_second" : -1.0,
"throttled_until_millis" : 0,
"failures" : [ ]
}
也可以使用匹配索引查询：
curl -X GET "localhost:9200/user_logs*/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"message": "successful"
}
}
}
'
{
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 0.49917626,
"hits" : [
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
},
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:07:08.000Z",
"user" : {
"id" : "l7gk7f82"
},
"message" : "Logout successful"
}
},
{
"_index" : "user_logs_000002",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
},
{
"_index" : "user_logs_000002",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.49917626,
"_source" : {
"@timestamp" : "2020-12-07T11:07:08.000Z",
"user" : {
"id" : "l7gk7f82"
},
"message" : "Logout successful"
}
}
]
}
}
使用from和size进行范围指定查询，这里还加入了term条件，指定查询的字段值，相当于加入参数?q=user.id:8a4f500d：
curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
"from": 0,
"size": 5,
"query": {
"term": {
"user.id": "8a4f500d"
}
}
}
'
{
"took" : 26,
"timed_out" : false,
"_shards" : {
"total" : 24,
"successful" : 24,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.9808291,
"hits" : [
{
"_index" : "user_logs_000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.9808291,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
},
{
"_index" : "user_logs_000002",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.9808291,
"_source" : {
"@timestamp" : "2020-12-07T11:06:07.000Z",
"user" : {
"id" : "8a4f500d"
},
"message" : "Login successful"
}
}
]
}
}