ELK基础环境搭建-logstash部署
1 logstash部署
1.1 修改系统参数
1. hosts文件
vi /etc/hosts
127.0.0.1 localhost
192.168.2.21 logstash
2. limits及pam添加以下内容
vi /etc/security/limits.conf
* soft nofile 1024
* hard nofile 65536
* soft nproc 2047
* hard nproc 16384
* soft stack 10240
* hard stack 32768
vi /etc/pam.d/login
session required/lib64/security/pam_limits.so
1.2 安装java
rpm -ivh jdk-8u101-linux-x64.rpm
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_101
export PATH=$JAVA_HOME/bin:$PATH
1.3 安装logstash
1. 解压
tar -xvzf logstash-2.4.0.tar.gz
2. 创建相关目录
cd /home/cendish/software/logstash-2.4.0/bin
mkdir conf logs
3. 启动进程
./logstash -e 'input { stdin {} } output { stdout {} }'
[cendish@logstash bin]$./logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipelineworkers: 4
Pipeline main started
hello world
2016-10-09T05:51:11.735Zlogstash hello world
2016-10-09T05:51:11.994Zlogstash
4. 创建pipeline
CTRL-D退出命令行。logstash架构如下：
logstash的pipeline包含input和output两个必须部分，另外filter是可选的。配置文件的结构如下：
#The # character at the beginning of a line indicates a comment. Use
#comments to describe your configuration.
input{
}
#The filter part of this file is commented out to indicate that it is
#optional.
#filter {
#
#}
output{
}
以下是官方的一个例子，读取文件的日志，输出到elasticsearch。参数文件如下：
input{
file {
path =>"/home/cendish/logs/*.log"
start_position => beginning
ignore_older => 0
}
}
filter{
grok {
match => { "message" =>"%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output{
elasticsearch {
hosts => ["192.168.2.11:9200" ]
}
}
5. 测试
./logstash -f conf/first-pipeline.conf --configtest
[cendish@logstash bin]$./logstash -f conf/first-pipeline.conf --configtest
Configuration OK
[cendish@logstash bin]$
6. 正式运行
./logstash -f conf/first-pipeline.conf
然后运行查询命令：
[cendish@logstash conf]$ curl-XGET '192.168.2.11:9200/logstash-2016.10.09/_search?pretty&q=response=200'
{
"took" : 91,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 98,
"max_score" : 5.053008,
"hits" : [ {
"_index" :"logstash-2016.10.09",
"_type" : "logs",
"_id" :"AVeoF0hU7h3h6d65347z",
"_score" : 5.053008,
"_source" : {
"message" : "83.149.9.216- - [04/Jan/2015:05:13:45 +0000] \"GET/presentations/logstash-monitorama-2013/images/frontend-response-codes.pngHTTP/1.1\" 200 52878\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"@version" : "1",
"@timestamp" :"2016-10-09T06:17:58.042Z",
"path" :"/home/cendish/logs/logstash-tutorial.log",
"host" :"logstash",
"clientip" :"83.149.9.216",
"ident" : "-",
"auth" : "-",
"timestamp" :"04/Jan/2015:05:13:45 +0000",
"verb" : "GET",
"request" :"/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
"httpversion" : "1.1",
"response" : "200",
"bytes" : "52878",
"referrer" :"\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
"agent" :"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"geoip" : {
"ip" :"83.149.9.216",
"country_code2" :"RU",
"country_code3" :"RUS",
"country_name" :"Russian Federation",
"continent_code" :"EU",
"region_name" :"48",
"city_name" :"Moscow",
"latitude" :55.75219999999999,
"longitude" : 37.6156,
"timezone" :"Europe/Moscow",
"real_region_name" :"Moscow City",
"location" : [ 37.6156,55.75219999999999 ]
}
}
},
...
过滤查询：
[cendish@logstash conf]$ curl-XGET'192.168.2.11:9200/logstash-2016.10.09/_search?pretty&q=geoip.city_name=Buffalo'
{
"took" : 29,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.1011324,
"hits" : [ {
"_index" :"logstash-2016.10.09",
"_type" : "logs",
"_id" :"AVeoF0hW7h3h6d65349B",
"_score" : 1.1011324,
"_source" : {
"message" :"108.174.55.234 - - [04/Jan/2015:05:27:45 +0000] \"GET /?flav=rss20HTTP/1.1\" 200 29941 \"-\" \"-\"",
"@version" : "1",
"@timestamp" :"2016-10-09T06:17:58.619Z",
"path" :"/home/cendish/logs/logstash-tutorial.log",
"host" :"logstash",
"clientip" :"108.174.55.234",
"ident" : "-",
"auth" : "-",
"timestamp" :"04/Jan/2015:05:27:45 +0000",
"verb" : "GET",
"request" :"/?flav=rss20",
"httpversion" :"1.1",
"response" : "200",
"bytes" : "29941",
"referrer" :"\"-\"",
"agent" :"\"-\"",
"geoip" : {
"ip" :"108.174.55.234",
"country_code2" :"US",
"country_code3" :"USA",
"country_name" :"United States",
"continent_code" :"NA",
"region_name" :"NY",
"city_name" :"Buffalo",
"postal_code" :"14221",
"latitude" : 42.9864,
"longitude" : -78.7279,
"dma_code" : 514,
"area_code" : 716,
"timezone" :"America/New_York",
"real_region_name" :"New York",
"location" : [ -78.7279,42.9864 ]
}
}
} ]
}
}