ELK安装+Filebeat收集java日志

java日志格式

[2022-07-14 10:03:30.310] [http-nio-8080-exec-2] [ERROR] (com.zz.framework.exception.handler.GlobalExceptionHandler:83) - 数据校验异常,异常字段

下载

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.4-x86_64.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.4-x86_64.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.4-x86_64.rpm
rpm包wget https://artifacts.elastic.co/downloads/enterprise-search/enterprise-search-7.17.4.rpm
或者wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.4-linux-x86_64.tar.gz

安装配置elasticsearch

安装jdk环境

● 下载安装包
○ 登陆官网下载(需要oracle账号)
■ https://www.oracle.com/java/technologies/downloads/#java8
解压并配置环境变量

tar xf jdk-8u311-linux-x64.tar.gz
mv jdk1.8.0_311 /usr/local/jdk
cat >> /etc/profile <<EOF
JAVA_HOME=/usr/local/jdk
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export PATH JAVA_HOME CLASSPATH
EOF
source /etc/profile
● 验证jdk
java -version

配置系统参数

● 设置内核参数
echo "vm.max_map_count=655360" >> /etc/sysctl.conf
● 执行以下命令确保配置生效。
sysctl -p
● 设置资源参数
  ○  # vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 65536
* hard nproc 131072

设置es启动用户资源参数
echo "elastic    soft    nproc     65536" >> /etc/security/limits.d/20-nproc.conf

解压elasticsearch到指定目录

tar xf elasticsearch-7.17.4-linux-x86_64.tar.gz -C /usr/local/
mkdir -p /data/es/{data,logs}
chown -R elastic.elastic /usr/local/elasticsearch-7.17.4
chown -R elastic.elastic /data/es
su - elastic
cat >> /usr/local/elasticsearch-7.17.4/config/elasticsearch.yml <<EOF
cluster.name: application
node.name: node-1
path.data: /data/es/data
path.logs: /data/es/logs
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
plugin.mandatory: ingest-attachment
xpack.security.enabled: 'true'
xpack.security.transport.ssl.enabled: 'true'
EOF

设置密码

/usr/local/elasticsearch-7.17.4/bin/elasticsearch-setup-passwords interactive

后台启动

/usr/local/elasticsearch-7.17.4/bin/elasticsearch -d

查看日志验证

安装配置logstash，并启动

cat > /etc/logstash/conf.d/app-java.conf << EOF
input {
    beats {
        port => 5044
    }
}
filter {
	grok {
		match => [
			"message", "\[%{DATA:logtime}\] "
		]

		break_on_match => false
	}

	date {
		match=> ["logtime","yyyy-MM-dd HH:mm:ss.SSS"]
		target=>"@timestamp"
	}
}
output {
  if "jAPPname" in [tags] {
    elasticsearch {
    hosts => ["elasticsearch集群ip:9200"]
    index => "java-APPname-%{+YYYY-MM-dd}"
    user => "elastic"
    password => "123456"
    }
  }else if "APPname" in [tags] {
    elasticsearch {
    hosts => ["elasticsearch集群ip:9200"]
    index => "java-APPname-%{+YYYY-MM-dd}"
    user => "elastic"
    password => "123456"
    }
  }else {
    elasticsearch {
    hosts => ["elasticsearch集群ip:9200"]
    index => "java-%{+YYYY-MM-dd}"
    user => "elastic"
    password => "123456"
    }
  }
}
EOF
systemctl start logstash

安装配置filebeat,并启动

rpm -ivh filebeat-7.17.4-x86_64.rpm
cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak
cat > /etc/filebeat/filebeat.yml << EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/APPname.log
  tags: ["APPname"]
  fields:
    env: test
    app: APPname
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: "after"
- type: log
  enabled: true
  paths:
    - /tmp/APPname.log
  tags: ["APPname"]
  fields:
    env: stg
    app: APPname
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: "after"
- type: log
  enabled: true
  paths:
    - /tmp/APPname.log
  tags: ["APPname"]
  fields:
    env: prod
    app: APPname
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: "after"
- type: log
  enabled: true
  paths:
    - /tmp/APPname.log
  tags: ["APPname"]
  fields:
    env: test
    app: APPname
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: "after"
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
output.logstash:
  hosts: ["logstashIP:5044"]
EOF

systemctl start filebeat

filebeat排错

如果你发现没有启动成功，可以执行 cd /usr/bin，在这个目录下执行./filebeat -c /etc/filebeat/filebeat.yml -e，这样会提示具体的错误信息

filebeat删除元数据，重新读取数据创建索引（解决）

背景：filebeat搜集同一文件，修改filebeat中自定义tag和logstash中定义的索引名称，重启后不生效，es中未创还能对应的索引
解决办法
mv /var/lib/filebeat/registry/filebeat/log.json /tmp
systemctl restart filebeat
查看对应索引是否创建
curl -u elastic:123456 -XGET 'http://localhost:9200/_cat/indices?v&pretty'

安装配置kibana，并启动

rpm -ivh kibana-7.17.4-x86_64.rpm
cp /etc/kibana/kibana.yml /etc/kibana/kibana.yml.bak
sed -i 's/#elasticsearch.username: "kibana_system"/elasticsearch.username: "kibana"/g' /etc/kibana/kibana.yml
sed -i 's/#elasticsearch.password: "pass"/elasticsearch.password: "123456"/g' /etc/kibana/kibana.yml

systemctl start kibana

防火墙开放5601端口

elfk采集json格式的java日志

背景：经与开发沟通，改造java日志为json格式，并添加相应字段 app、env、timestamp等方便采集
最终日志格式为

{"timestamp":"2022-09-09T01:58:00,151Z","logfilepath":"logs/java-financial-2022-09-09.log","app":"financial","version":"v1.3.0","env":"test","host":"financial-56fb8bf9b9-nmbpz","level":"INFO","pid":"7","thread":"Thread-8","class":"com.zz.mp.user.service.impl.BarcodeServiceImpl","method":"generateBarcode","line":"100","message":"完成释放生成Barcode任务锁","statck_trace":""}

logstash配置

input {
    beats {
        port => 5044
    }
}
output {
  if "financial" in [tags] {
    elasticsearch {
    hosts => ["192.168.11.41:9200"]
    index => "jfinancial-%{+YYYY-MM-dd}"
    user => "elastic"
    password => "123456"
    }
  }else if "java" in [tags] {
    elasticsearch {
    hosts => ["192.168.11.41:9200"]
    index => "%{[app]}-%{+YYYY-MM-dd}"
    user => "elastic"
    password => "123456"
    }
  }
}

filebeat配置

- type: log
  paths:
  - /app/*/*/java-*.log
  exclude_files: ["_filebeat", ".gz$"]
  json.keys_under_root: true
  json.overwrite_keys: true
  tags:
    java