ELK配置记录

Elasticsearch

Elasticsearch是一个基于lucene的全文搜索引擎

1.安装

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.0.tar.gz

tar -xzvf elasticsearch-5.3.0.tar.gz

mv elasticsearch-5.3.0  /usr/share/elasticsearch-5.3.0 

2. 配置

修改配置文件

path.data: /var/lib/elasticsearch //数据存放的路径
path.logs: /var/log/elasticsearch //log的路径
node.name: wddlc //节点的名字
network.host: _网卡编号_  
http.port: 9206  //监听的端口号

配置文件：/usr/share/elasticsearch-5.3.0/config/elasticsearch.yml

如果需要的话修改jvm的内存参数

-Xms1g   初始内存大小
-Xmx1g   最大内存大小

配置文件：/usr/share/elasticsearch-5.3.0/config/jvm.options

3. 启动

nohup /usr/share/elasticsearch-5.3.0/bin/elasticsearch -Epath.conf=/usr/share/elasticsearch-5.3.0/config/ > /dev/null 2>&1 &

ps -ef | grep elas  //关闭
kill pid

检测启动是否成功.png

4.一个小问题

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

这是因为系统进程最大虚拟内存映射的数量不足，用下面的命令设置一下就可以了

sysctl -w vm.max_map_count=262144

5.监控插件

git clone git://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head

npm install

npm run start

open
 [http://localhost:9100/](http://localhost:9100/)

注意：使用这个插件的时候，要打开es的跨域

http.cors.enabled: true
http.cors.allow-origin: "*"

logstash

logstash是一个开源的数据收集引擎，负责从多个数据源汇总数据、过滤、并输出到指定的输出中

1.安装

wget wget https://artifacts.elastic.co/downloads/logstash/logstash-5.3.0.tar.gz

tar -xzvf logstash-5.3.0.tar.gz

mv logstash-5.3.0  /usr/share/logstash-5.3.0

2.配置

input {
    beats {
        #  监听filebeat的输入
        port => 5043
    }
}
filter {
  # 删除无用的key
    if ([type] == "log") {
        ruby {
            code => "
                event.to_hash.each { |k,v|
                    if (!['json', 'type'].include?(k))
                        event.remove(k)
                    end
                }
            "
        }
# 根据ip，补充地理位置信息
        geoip {
            source => "[json][ip]"
        }
    }
}
output {
    if ([type] == "log") {
# 输出到标准输出
        stdout { codec => rubydebug}
# 输出到刚刚配置好的es里
        elasticsearch {
            hosts => ["172.16.3.2:9206"]
            index => "%{[json][model]}"
        }
    }
}

2.启动

nohup /usr/share/logstash-5.3.0/bin/logstash -f /usr/share/logstash-5.3.0/config/first-pipeline.conf > /etc/logstash.log 2>&1 &

filebeat

1.安装

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.0-x86_64.rpm
sudo rpm -vi filebeat-5.3.0-x86_64.rpm

2.配置

配置文件在：/etc/filebeat/filebeat.yml

# 设置输入
 input_type: log
# 表示数据的是json
  json.key_under_root: true
# 监控这个路径下的log
  paths:
    - /var/www/prometheus-library/logs/*.log

# 设置控制台输出，方便调试，正式环境可以关掉
output.console:
  pretty: true

# 输出到logstash
output.logstash:
  hosts: ["172.16.3.2:5043"]

3.启动

sudo /etc/init.d/filebeat start

kibana

1.安装

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.0-linux-x86_64.tar.gz

tar -xzvf kibana-5.3.0-linux-x86_64.tar.gz

mv kibana-5.3.0-linux-x86_64 /usr/share/kibana-5.3.0-linux-x86_64

2.配置

配置文件：/usr/share/kibana-5.3.0-linux-x86_64/config/kibana.yml

# 设置es的地址
server.host: "0.0.0.0"

# es请求的路径
elasticsearch.url: "http://172.16.3.2:9206"

2.启动

nohup /usr/share/kibana-5.3.0-linux-x86_64/bin/kibana > kibana.log 2>&1 &