ELK日志框架搭建及实现笔记
logstash配置及启动
进入到安装目录logstash-6.0.0/bin下，测试是否安装成功
./logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'临时启动：
./logstash -f ../config/logstash.conf
永久启动：
nohup ./logstash -f ../config/logstash.conf &示例：多文件读入到ElasticSearch中，logstash.conf配置如下
# 日志输入来源定义
input{
file{
path=>[
"/opt/app/test_log/log/ae.log"
]
start_position => "beginning"
type => "ae.log"
}
}
input{
file{
path=>[
"/opt/app/test_log/log/pay.log"
]
start_position => "beginning"
type => "pay.log"
}
}
# 日志过滤定义
filter {
# 将日志中的时间替换为@timestamp
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
}
date {
match => ["logdate", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
}
# 去除冗余字段
mutate {
remove_field => "_id"
remove_field => "@version"
remove_field => "host"
remove_field => "_score"
}
}
# 日志输出到elasticsearch定义
output{
elasticsearch {
action => "index"
hosts => "ip:9200"
index => "logstash-%{type}-%{+YYYY.MM.dd}"
codec => "json"
}
# dubug
stdout { codec => rubydebug }
}ElasticSearch 安装启动说明
问题一：ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]解决：切换到root用户，编辑limits.conf 添加类似如下内容
vi /etc/security/limits.conf
添加如下内容:
soft nofile 65536
hard nofile 131072
soft nproc 2048
hard nproc 4096
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
原因：
这是在因为Centos6不支持SecComp，而ES5.2.0默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。
解决：
在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
问题二：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
解决：切换到root用户，进入limits.d目录下修改配置文件。
vi /etc/security/limits.d/90-nproc.conf
修改如下内容：
soft nproc 1024
修改为
soft nproc 2048
问题三：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
解决：切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p
然后，重新启动elasticsearch，即可启动成功。
问题四：我们在使用elasticsearch的时候，如果是以root权限来执行elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:160)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
1这是出于系统安全考虑设置的条件。由于ElasticSearch可以接收用户输入的脚本并且执行，为了系统安全考虑，
建议创建一个单独的用户用来运行ElasticSearch
创建elsearch用户组及elsearch用户
groupadd elsearch
useradd elsearch -g elsearch -p elasticsearch更改elasticsearch文件夹及内部文件的所属用户及组为elsearch:elsearch
chown -R elsearch:elsearch elasticsearchelasticsearch为你elasticsearch的目录名称,切换到elsearch用户再启动即可。
nohup ./elasticsearch& 或
nohup ./elasticsearch>/dev/null 2>&1 & 永久启动elasticsearch.yml 部分说明如下：
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
# add
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
# end
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
kibana
nohup ./kibana& 或
nohup ./kibana >/dev/null 2>&1 & 永久启动kibana.yml 部分说明
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "127.0.0.1" #(访问ip)
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://127.0.0.1:9200" #(访问ip)其它知识点
- 一般最新的elk环境jdk需要1.8版本，下载解压版做相应配置即可。
如:logstash 可以在logstash.lib.sh中加入以下设置：
export JAVA_CMD="/opt/app/elk_manager_log/jdk1.8.0_77/bin"
export JAVA_HOME="/opt/app/elk_manager_log/jdk1.8.0_77/"- Linux】如何关闭某个被占用的端口？
1）查找被占用的端口：
abloume@ubuntu:~$ netstat -tln | grep 8000
tcp 0 0 192.168.2.106:8000 0.0.0.0:* LISTEN
2）查看被占用端口的PID：
abloume@ubuntu:~$ sudo lsof -i:8000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 850 root 6u IPv4 15078 0t0 TCP 192.168.2.106:8000 (LISTEN)
nginx 851 www-data 6u IPv4 15078 0t0 TCP 192.168.2.106:8000 (LISTEN)
nginx 852 www-data 6u IPv4 15078 0t0 TCP 192.168.2.106:8000 (LISTEN)
3）kill掉该进程
abloume@ubuntu:~$ sudo kill -9 850参考文章：
https://blog.csdn.net/qq_24879495/article/details/78009562
https://blog.csdn.net/buqutianya/article/details/72027209