本文针对LogStash常用插件grok和geoip的使用进行说明:
编辑 first-pipeline.conf 文件,修改为如下内容:
input{
#stdin{type => stdin}
file {
# 读取文件的路径
path => ["/tmp/access.log"]
start_position => "beginning"
}
}
filter{
grok{
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
}
output{
stdout{codec => rubydebug}
}
启动./logstash -f ../config/first-pipeline.conf
后输出就为结构化的数据了:
{
"message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
"response" => "200",
"auth" => "-",
"bytes" => "103",
"referrer" => "\"-\"",
"host" => "nb002",
"@version" => "1",
"agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
"@timestamp" => 2022-06-26T00:28:24.302Z,
"timestamp" => "25/Jun/2022:05:11:33 +0800",
"ident" => "-",
"httpversion" => "1.1",
"path" => "/tmp/access.log",
"clientip" => "140.77.188.102",
"verb" => "GET",
"request" => "/api/ss/api/v1/login/getBaseUrl"
}
编辑 first-pipeline.conf 文件,修改为如下内容:
input{
#stdin{type => stdin}
file {
path => ["/tmp/access.log"]
start_position => "beginning"
}
}
filter{
grok{
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
mutate{
# 重命名字段
rename => {"clientip" => "cip"}
}
mutate{
# 移出特定字段
remove_field => ["timestamp","agent"]
}
}
output{
stdout{codec => rubydebug}
}
重新启动./logstash -f ../config/first-pipeline.conf
后,往 /tmp/access.log 中新增一条数据,看输出:发现"clientip" 变成了 “cip” 和timestamp agent 字段已经没有了。NICE
{
"verb" => "GET",
"@timestamp" => 2022-06-26T00:48:28.224Z,
"referrer" => "\"-\"",
"path" => "/tmp/access.log",
"auth" => "-",
"message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
"@version" => "1",
"ident" => "-",
"response" => "200",
"bytes" => "103",
"request" => "/api/ss/api/v1/login/getBaseUrl",
"httpversion" => "1.1",
"host" => "nb002",
"cip" => "140.77.188.102"
}
使用geoip过滤器插件,可以增强数据。
geoip插件可以针对IP地址进行地理位置信息来源的查找
编辑 first-pipeline.conf 文件,修改为如下内容:
input{
#stdin{type => stdin}
file {
path => ["/tmp/access.log"]
start_position => "beginning"
}
}
filter{
grok{
match => {"message" => "%{COMBINEDAPACHELOG}" }
}
mutate{
# 重命名字段
rename => {"clientip" => "cip"}
}
mutate{
# 移出特定字段
remove_field => ["timestamp","agent"]
}
geoip{
# 由于上面将clientip修改为了cip,故此处配置cip,如果没有rename字段则用clientip
source => "cip"
}
}
output{
stdout{codec => rubydebug}
}
重新启动./logstash -f ../config/first-pipeline.conf
后,往 /tmp/access.log 中新增一条数据,看输出:发现输出结果中新增了geoip
字段,并展示了地区、国家、省份、经纬度等地理位置信息。
外国ip示例:
{
"host" => "nb002",
"auth" => "-",
"bytes" => "103",
"cip" => "140.77.188.104",
"@version" => "1",
"message" => "140.77.188.104 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
"verb" => "GET",
"request" => "/api/ss/api/v1/login/getBaseUrl",
"referrer" => "\"-\"",
"response" => "200",
"ident" => "-",
"path" => "/tmp/access.log",
"@timestamp" => 2022-06-26T00:58:11.786Z,
"geoip" => {
"country_code3" => "FR",
"longitude" => 4.85,
"ip" => "140.77.188.104",
"continent_code" => "EU",
"region_name" => "Rhône",
"country_code2" => "FR",
"timezone" => "Europe/Paris",
"country_name" => "France",
"region_code" => "69",
"latitude" => 45.748,
"postal_code" => "69007",
"location" => {
"lat" => 45.748,
"lon" => 4.85
},
"city_name" => "Lyon"
},
"httpversion" => "1.1"
}
国内ip示例:
{
"host" => "nb002",
"auth" => "-",
"bytes" => "103",
"cip" => "175.30.108.241",
"@version" => "1",
"message" => "175.30.108.241 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
"verb" => "GET",
"request" => "/api/ss/api/v1/login/getBaseUrl",
"referrer" => "\"-\"",
"response" => "200",
"ident" => "-",
"path" => "/tmp/access.log",
"@timestamp" => 2022-06-26T01:00:11.972Z,
"geoip" => {
"country_code3" => "CN",
"longitude" => 125.3247,
"ip" => "175.30.108.241",
"continent_code" => "AS",
"region_name" => "Jilin",
"country_code2" => "CN",
"timezone" => "Asia/Shanghai",
"country_name" => "China",
"region_code" => "JL",
"latitude" => 43.88,
"location" => {
"lat" => 43.88,
"lon" => 125.3247
},
"city_name" => "Changchun"
},
"httpversion" => "1.1"
}