介绍
ELK是三个开源项目的首字母缩写,这三个项目分别是:Elasticsearch、Logstash 和 Kibana。
- Elasticsearch 是一个搜索和分析引擎。
- Logstash 是服务器端数据处理管道,能够同时从多个来源采集数据,转换数据,然后将数据发送到诸如 Elasticsearch 等“存储库”中。
- Kibana 则可以让用户在 Elasticsearch 中使用图形和图表对数据进行可视化。
克隆一台虚拟机
- 从之前装好jdk的centos虚拟机快照克隆一台专门搭建elk环境
- 克隆完成以后开机,登录
- 修改主机名
vim /etc/hostname
zmzhou-132-elk
- 修改ip地址
vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
#UUID=d77bb448-a7db-4b0f-9812-b306e44c5d3b
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.163.132
GATEWAY=192.168.163.2
NETMASK=255.255.255.0
DNS1=8.8.8.8
DNS2=114.114.114.114
- 重启
reboot
- 检查IP,网络和Java环境
如果Java环境没有配好,请参考 jdk1.8商用免费版下载地址
下载,并配置环境变量:
vim /etc/profile
# 在最后添加如下内容:ZZ保存退出以后执行 source /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_202
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export ES_JAVA_HOME=/home/elastic/elasticsearch-7.12.1/jdk
安装 elasticsearch
-
lscpu
查看系统架构
- 下载相应版本的 elasticsearch https://www.elastic.co/cn/downloads/elasticsearch
- 创建用户elastic
useradd elastic
上传安装包到/home/elastic/
目录下 - 解压,修改配置文件,启动
tar -zxvf elasticsearch-7.12.1-linux-x86_64.tar.gz
cd elasticsearch-7.12.1/
vim config/elasticsearch.yml
# 修改如下内容
cluster.name: zmzhou-132-elk
node.name: es-node-1
path.data: /home/elastic/elasticsearch-7.12.1/data
path.logs: /home/elastic/elasticsearch-7.12.1/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["127.0.0.1", "zmzhou-132-elk"]
cluster.initial_master_nodes: ["es-node-1"]
# 修改文件夹所属用户权限
chown -R elastic:elastic /home/elastic/
# 切换用户
su - elastic
cd elasticsearch-7.12.1/
# 后台启动
./bin/elasticsearch -d
报错以及解决办法
- 报错1
JAVA_HOME is deprecated, use ES_JAVA_HOME
vim /etc/profile
export ES_JAVA_HOME=/home/elastic/elasticsearch-7.12.1/jdk
source /etc/profile
- 报错2 ERROR: bootstrap checks failed
[2] bootstrap checks failed. You must address the points described in the following [2] lines before starting Elasticsearch.
bootstrap check failure [1] of [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
bootstrap check failure [2] of [2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
解决:[1]
编辑 sysctl.conf
添加如下配置
# echo "vm.max_map_count=262144" >> /etc/sysctl.conf
# sysctl -p #使修改立即生效
[2]
在elasticsearch.yml
中加上如下配置:
discovery.seed_hosts: ["127.0.0.1", "zmzhou-132-elk"]
cluster.initial_master_nodes: ["es-node-1"]
- 报错3 ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [elastic] likely too low, increase to at least [2048]
解决:[1]
切换到root用户,编辑limits.conf 根据错误提示添加如下内容:
vim /etc/security/limits.conf
#添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
[2]
编辑 90-nproc.conf
修改配置
vim /etc/security/limits.d/90-nproc.conf
#修改为
* soft nproc 2048
- 报错4 bootstrap checks failed
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
解决:在elasticsearch.yml
中加上如下配置:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
- 查看日志
tail -100f /home/elastic/elasticsearch-7.12.1/logs/zmzhou-132-elk.log
, 启动成功如下:
安装 Logstash
- 下载相应版本的 Logstash https://www.elastic.co/cn/downloads/logstash
- 解压,修改配置文件
tar -zxvf logstash-7.12.1-linux-x86_64.tar.gz
cd logstash-7.12.1/
cp config/logstash-sample.conf config/logstash.conf
vim startup.sh
#编辑如下内容,保存退出
#!/bin/bash
nohup ./bin/logstash -f config/logstash.conf &
chmod +x startup.sh
vim config/logstash.conf
添加配置如下
input {
beats {
port => 5044
}
tcp {
mode => "server"
host => "0.0.0.0" # 允许任意主机发送日志
type => "elk1" # 设定type以区分每个输入源
port => 4567
codec => json_lines # 数据格式
}
}
output {
if [type] == "elk1" {
elasticsearch {
action => "index" # 输出时创建映射
hosts => "192.168.163.132:9200" # ElasticSearch 的地址和端口
index => "elk1-%{+YYYY.MM.dd}" # 指定索引名
codec => "json"
}
}
}
- 启动 logstash
./startup.sh
#查看日志
tail -100f nohup.out
安装 Kibana
- 下载相应版本的 Kibana https://www.elastic.co/cn/downloads/kibana
- 解压,修改配置文件,启动
tar -zxvf kibana-7.12.1-linux-x86_64.tar.gz
cd kibana-7.12.1-linux-x86_64/
vim config/kibana.yml
#修改如下内容:
server.port: 5601
server.host: "0.0.0.0"
server.name: "zmzhou-132-elk"
elasticsearch.hosts: ["http://localhost:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
# 后台启动
nohup ./bin/kibana &
- 启动成功访问:
http://192.168.163.132:5601/
springboot + logback 输出日志到 logstash
- 添加 logstash 依赖
pom.xml
net.logstash.logback
logstash-logback-encoder
6.6
- 修改
application.yml
,添加配置
logstash:
address: 192.168.163.132:4567
- 修改
logback-spring.xml
配置
INFO
${CONSOLE_LOG_PATTERN}
utf8
false
${LOG_FILE}.%d{yyyy-MM-dd}.log
10
%d{yyyy-MM-dd HH:mm:ss} %-5level logger{39} -%msg%n
${LOGSTASH_ADDRESS}
Asia/Shanghai
{
"app": "${APPLICATION_NAME}",
"level": "%level",
"thread": "%thread",
"logger": "%logger{50} %M %L ",
"message": "%msg"
}
100
true
true
springboot + log4j2 异步输出日志到 logstash
- 修改
log4f2.xml
/opt/web-shell/logs
7
%d{yyyy-MM-dd HH:mm:ss z} [%thread] %-5level %class{36} [%M:%L] - %msg%xEx%n
%d{HH:mm:ss.SSS} [%thread] %-5level %class{36} [%M:%L] - %msg%xEx%n
{"app": "web-shell", "level": "%level", "message": "%thread %M %L - %msg%xEx%n"}
- 本项目地址: https://gitee.com/zmzhou-star/web-shell
测试,启动spring boot项目,访问 http://192.168.163.132:5601/
- 创建索引,我们之前在 logstash 配置的索引规则是以 elk1 开头:
- 筛选我们的应用名
- 至此,我们的ELK环境已经搭好啦,不过还有更多功能等待解锁,比如 Beats等,整个软件目录如下
设置Nginx反向代理账号密码访问
JVM Support Matrix https://www.elastic.co/cn/support/matrix#matrix_jvm
防火墙相关命令
# 启动:
systemctl start firewalld
# 查看状态:
systemctl status firewalld
firewall-cmd --state
# 停止:
systemctl disable firewalld
#禁用:
systemctl stop firewalld
#查看所有打开的端口
firewall-cmd --zone=public --list-ports
#添加一个端口
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=4567/tcp --permanent
firewall-cmd --zone=public --add-port=5601/tcp --permanent
#删除一个端口
firewall-cmd --zone=public --remove-port=80/tcp --permanent
#更新防火墙规则
firewall-cmd --reload