1. 实验环境
目的：使用ELK采集服务器的系统日志，并将数据可视化展示。
介于之前没有这方面的经验，先从简单的开始。
1.1. 准备工作
准备4台虚拟机，分别用于以下功能：
- 客户端。也就是要被采集的服务器，需要配置rsyslog服务。
- IP: 192.167.17.11
- 主机名: client.localdomain
- CPU: 1核
- 内存: 1G
- 硬盘: 40G
- 缓存服务。需要部署logstash服务和redis服务。配置尽量高一些，logstash服务依赖java环境，比较耗内存。
- IP: 192.167.17.12
- 主机名: redis.localdomain
- CPU: 4核
- 内存: 4G
- 硬盘: 40G
- 存储服务。需要部署logstash服务和elasticsearch服务。配置也尽量高一些。
- IP: 192.167.17.13
- 主机名: elasticsearch.localdomain
- CPU: 4核
- 内存: 4G
- 硬盘: 40G
- 展示服务。需要部署kibana服务。
- IP: 192.167.17.14
- 主机名: kibana.localdomain
- CPU: 2核
- 内存: 2G
- 硬盘: 40G
2. 客户端服务部署
配置非常简单。只需要改一个参数即可。
2.1. 修改rsyslog配置
文件路径：/etc/rsyslog.conf。一般在倒数第二行。
*.* @@192.168.17.12:514
2.2. 重启rsyslog服务
[root@client ~]# systemctl restart rsyslog.service
3. logstash服务和redis服务部署
3.1. 部署java环境
- 配置好yum源
[root@redis ~]# mv /etc/yum.repos.d/* /tmp/
[root@redis ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
[root@redis ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
- 安装java
[root@redis ~]# yum install java-11
配置java环境变量，文件路径：/etc/profile
在最后面加上：
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64
JRE_HOME=$JAVA_HOME
CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
PATH=${JAVA_HOME}/bin:$PATH
export JAVA_HOME JRE_HOME CLASSPATH PATH
环境变量生效：重启机器或是source /etc/profile。
[root@redis ~]# echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64
3.2. 部署redis
由于官网的网速比较慢，我是用华为云镜像。
- 部署redis
[root@redis ~]# yum install redis
- 配置redis
文件路径：/etc/redis.conf。添加如下配置：
# 使redis后台运行，守护进程
daemonize yes
# 配置监听ip
bind 192.168.17.12
- 启动redis
[root@redis ~]# systemctl enable redis.service
[root@redis ~]# systemctl start redis.service
3.3. 部署logstash
- 安装logstash
[root@redis ~]# yum install https://mirrors.huaweicloud.com/logstash/7.7.1/logstash-7.7.1.rpm
没有报错就说明安装成功了。
其中有一条警告 OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was...
从网上查到，这是一种内存回收机制，比较耗内存，我们改一下内存回收机制，修改文件/etc/logstash/jvm.options。
将 -XX:+UseConcMarkSweepGC 替换成 -XX:+UseG1GC
- 配置logstash
文件路径：/etc/logstash/conf.d/rsyslog2redis.conf
input {
syslog {
type => "rsyslog"
host => "192.168.17.12"
port => "514"
}
}
output {
redis {
host => "192.168.17.12"
port => "6379"
db => "10"
data_type => "list"
key => "rsyslog"
}
}
- 启动logstash服务
文件路径：/etc/sysconfig/logstash
LS_USER=root
简单的测试
[root@redis ~]# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
...省略WARN和INFO信息...
hello, world!!!
{
"host" => "redis.localdomain",
"@timestamp" => 2020-06-26T16:38:15.075Z,
"@version" => "1",
"message" => "hello, world!!!"
}
[root@redis ~]# systemctl enable logstash.service
[root@redis ~]# systemctl start logstash.service
启动比较慢，大概需要2分钟。查看是否启动成功的方法。
[root@redis ~]# systemctl status logstash.service
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2020-06-27 00:20:52 CST; 1min 24s ago
Main PID: 1473 (java)
CGroup: /system.slice/logstash.service
└─1473 /bin/java -Xms1g -Xmx1g -XX:+UseG1GC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccu...
Jun 27 00:20:52 redis.localdomain systemd[1]: Started logstash.
Jun 27 00:20:52 redis.localdomain systemd[1]: Starting logstash...
Jun 27 00:21:18 redis.localdomain logstash[1473]: WARNING: An illegal reflective access operation has occurred
Jun 27 00:21:18 redis.localdomain logstash[1473]: WARNING: Illegal reflective access by com.headius.backport9.mod...long)
Jun 27 00:21:18 redis.localdomain logstash[1473]: WARNING: Please consider reporting this to the maintainers of c...dules
Jun 27 00:21:18 redis.localdomain logstash[1473]: WARNING: Use --illegal-access=warn to enable warnings of furthe...tions
Jun 27 00:21:18 redis.localdomain logstash[1473]: WARNING: All illegal access operations will be denied in a futu...lease
Jun 27 00:21:55 redis.localdomain logstash[1473]: Sending Logstash logs to /var/log/logstash which is now configu...rties
Hint: Some lines were ellipsized, use -l to show in full.
3.4. 验证
验证client的日志是否存到了redis里面。
在客户端执行生成日志的命令：
[root@client ~]# logger "test"
在redis里面查看是否存储了日志：
[root@redis ~]# redis-cli -h 192.168.17.12
192.168.17.12:6379> ping
PONG
192.168.17.12:6379> info Keyspace
# Keyspace
db10:keys=1,expires=0,avg_ttl=0
192.168.17.12:6379> select 10
OK
192.168.17.12:6379[10]> keys *
1) "rsyslog"
192.168.17.12:6379[10]> llen rsyslog
(integer) 6
192.168.17.12:6379[10]> lindex rsyslog -1
"{\"severity\":6,\"timestamp\":\"Jun 27 00:50:01\",\"logsource\":\"client\",\"@timestamp\":\"2020-06-26T16:50:01.000Z\",\"@version\":\"1\",\"pid\":\"27036\",\"host\":\"192.168.17.11\",\"severity_label\":\"Informational\",\"type\":\"rsyslog\",\"facility\":9,\"facility_label\":\"clock\",\"priority\":78,\"program\":\"CROND\",\"message\":\"(root) CMD (/usr/lib64/sa/sa1 1 1)\\n\"}"
192.168.17.12:6379[10]> exit
[root@redis ~]#
4. logstash服务和elasticsearch服务部署
4.1. 部署java环境
略。与3.1相同
4.2. 部署elasticsearch
- 安装elasticsearch
[root@elasticsearch ~]# yum install https://mirrors.huaweicloud.com/elasticsearch/7.7.1/elasticsearch-7.7.1-x86_64.rpm
- 配置elasticsearch
配置文件：/etc/elasticsearch/elasticsearch.yml
[root@elasticsearch ~]# cat /etc/elasticsearch/elasticsearch.yml |grep ^[a-z]
cluster.name: es
node.name: es-node01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 192.168.17.13
http.port: 9200
discovery.seed_hosts: ["192.168.17.13"]
cluster.initial_master_nodes: ["es-node01"]
http.cors.enabled: true
http.cors.allow-origin: "*"
配置文件：/usr/lib/systemd/system/elasticsearch.service
添加如下参数，将启动时间延长，不然会导致因启动时间长，而无法启动。
TimeoutStartSec=900
- 启动elasticsearch
[root@elasticsearch ~]# systemctl daemon-reload
[root@elasticsearch ~]# systemctl enable elasticsearch.service
[root@elasticsearch ~]# systemctl start elasticsearch.service
启动成功后，测试访问http://192.168.17.13:9200/
[root@elasticsearch ~]# curl http://192.168.17.13:9200/
{
"name" : "es-node01",
"cluster_name" : "es",
"cluster_uuid" : "UiO2khJYSMychDOkLPxM4g",
"version" : {
"number" : "7.7.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
"build_date" : "2020-05-28T16:30:01.040088Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
- 安装插件
[root@elasticsearch ~]# yum install git npm
[root@elasticsearch ~]# git clone git://github.com/mobz/elasticsearch-head.git
[root@elasticsearch ~]# vim elasticsearch-head/_site/app.js
# 将localhost改为192.168.17.13
[root@elasticsearch ~]# cd elasticsearch-head
[root@elasticsearch elasticsearch-head]# npm install
[root@elasticsearch elasticsearch-head]# npm run start
> [email protected] start /root/elasticsearch-head
> grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
最后访问http://192.168.17.13:9100/
4.3. 部署logstash
略。参照3.3。唯一不同的是配置文件。
文件路径：/etc/logstash/conf.d/redis2elasticsearch.conf
input {
redis {
host => "192.168.17.12"
port => "6379"
db => "10"
data_type => "list"
key => "rsyslog"
}
}
output {
elasticsearch {
hosts => ["192.168.17.13:9200"]
index => "rsyslog-%{+YYYY.MM.dd}"
}
}
4.4. 验证数据
访问http://192.168.17.13:9100/
5. 总结
至此，已经将系统日志存储到了elasticsearch。后续可以使用kibana进行数据展示。