1. 安装
https://www.elastic.co/cn/downloads/logstash
当前版本6.5.4，下载.tar.gz
2. 测试是否可用（需要基础环境jdk1.7+)
bin/logstash -e 'input { stdin {} } output { stdout {} }'
# 输入 hello，看到返回包含hello的json
如果遇到Unrecognized VM option 'UseParNewGC'错误，且可用通过java --version查看到jdk信息，
则需要修改config/jvm.options
#Remove or comment line:
-XX:+UseParNewGC
#Remove or comment line:
-XX:+UseConcMarkSweepGC
#Add this line instead:
-XX:+UseG1GC
查看已安装插件
./logstash-plugin list
3. 解析mongodb中的数据
安装mongodb的读取插件
bin/logstash-plugin install logstash-input-mongodb
#Validating logstash-input-mongodb
#Installing logstash-input-mongodb
#Installation successful
创建配置文件my.config:
input {
mongodb {
uri => 'mongodb://172.16.1.10:27017/test'
placeholder_db_dir => '/opt/mongodb/'
placeholder_db_name =>'BIZ_CATEGORY.db'
collection => 'BIZ_CATEGORY'
}
}
filter
{
# 把mongodb的_id替换掉，因为_id是跟es中的_id相冲突
mutate {
rename => ["_id", "uid"]
}
}
output {
file {
path => "/opt/mongons.log"
}
stdout {
codec => json_lines
}
# elasticsearch {
# hosts => ["192.168.1.171:9200"]
# index => "testData"
# manage_template=>true
# document_type => "judicial"
# }
}
执行命令:
../bin/logstash -f ./my.config
可获取mongons.log文件和控制台输出（处理后logstach会进入持续监视）
mongons.log内容：
{
"@timestamp": "2019-01-29T03:19:20.765Z",
"@version": "1",
"host": "fireflyMacBook-Pro.local",
"label": "xxxxx",
"log_entry": "{\"_id\"=>BSON::ObjectId('5c2c800a868cde4a10000c61'), \"value\"=>\"NationalProjects\", \"label\"=>\"xxxxx\"}",
"logdate": "2019-01-02T09:10:34+00:00",
"mongo_id": "5c2c800a868cde4a10000c61",
"uid": "5c2c800a868cde4a10000c61",
"value": "NationalProjects"
}
{
"@timestamp": "2019-01-29T03:19:20.682Z",
"@version": "1",
"host": "fireflyMacBook-Pro.local",
"label": "yyyyyy",
"log_entry": "{\"_id\"=>BSON::ObjectId('5c2c7e6c868cde4a10000c55'), \"value\"=>\"985School\", \"label\"=>\"yyyyyy\"}",
"logdate": "2019-01-02T09:03:40+00:00",
"mongo_id": "5c2c7e6c868cde4a10000c55",
"uid": "5c2c7e6c868cde4a10000c55",
"value": "985School"
}
@timestamp、@version、host、log_entry、logdate、mongo_id为附加信息
API解释：
Name Type Description
uri [String] A MongoDB URI for your database or cluster (check the MongoDB documentation for further info on this) [No Default, Required]
placeholder_db_dir [String] Path where the place holder database will be stored locally to disk [No Default, Required]
This gets created by the plugin so the directory needs to be writeable by the user that logstash is running as
placeholder_db_name [String] Name of the database file that will be created [Default: logstash_sqlite.db]
collection [String] A regex that will be used to find desired collecitons. [No Default, Required]
generateId [Boolean] If true, this will add a field '_id' that contains the MongoDB Document id
batch_size [Int] Size of the batch of mongo documents to pull at a time [Default: 30]
parse_method [String] Built in parsing of the mongodb document object [Default: 'flatten']
dig_fields [Array] An array of fields that should employ the dig method
dig_dig_fields [Array] This provides a second level of hash flattening after the initial dig has been done
4.输出到mongodb
安装mongodb的写入插件
bin/logstash-plugin install logstash-output-mongodb
API解释：
Name Type Description
bulk [boolean] Bulk insert flag, set to true to allow bulk insertion, else it will insert events one by one.[Default: false]
bulk_interval [number] Bulk interval, Used to insert events periodically if the "bulk" flag is activated.[Default: 2]
bulk_size [number] Bulk events number, if the number of events to insert into a collection raise that limit, it will be bulk inserted whatever the bulk interval value (mongodb hard limit is 1000).[Default: 900]
collection [String] The collection to use. This value can use %{foo} values to dynamically select a collection based on data in the event. [no default, Required]
database [String] The database to use. [Default: 2,Required]
generateId [boolean] If true, an "_id" field will be added to the document before insertion. The "_id" field will use the timestamp of the event and overwrite an existing "_id" field in the event.[Default: false]
isodate [boolean] If true, store the @timestamp field in MongoDB as an ISODate type instead of an ISO8601 string. For more information about this,see (http://www.mongodb.org/display/DOCS/Dates).[Default: false]
retry_delay [number] The number of seconds to wait after failure before retrying.[Default: 3]
uri [String] A MongoDB URI to connect to. [no default,Required]
5.java插件
https://github.com/logstash-plugins/logstash-filter-java_filter_example
grok常量
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns