centos7.9安装EFK ELK等日志系统选型

2021年9月5日21:41:08

关闭防火墙

systemctl stop firewalld
systemctl stop iptables

systemctl disable firewalld.service
systemctl disable iptables.service

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

 java安装

地址：http://openjdk.java.net/install/

参考 https://www.cnblogs.com/mabiao008/p/12059069.html

之前是直接写可执行文件的，现在需要换一个方式

注意：现在即使是yum也需要配置环境变量的，很奇葩

yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel.x86_64

jdk在

/usr/lib/jvm/

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64

配置环境变量，虽然现在可以使用java 命令，但是不能使用jps等相关命令

vi /etc/profile

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64
   JRE_HOME=$JAVA_HOME/jre
   CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
   PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
   export JAVA_HOME JRE_HOME CLASS_PATH PATH

注意：

A：https://www.elastic.co/cn/downloads/past-releases#filebeat

历史版本匹配，请先看下elasticsearch-analysis-ik的版本，最好和EFK的版本一致，不然问题多，还得回头重新来

1，版本匹配，建议3个软件版本一致，不然会有问题

2，filebeat和kibana建议yum安装，因为两个没有什么特殊部署需要，而且需要开机启动服务器，自己手动部署，需要接启动脚本

3，Elasticsearch如果需要集群，建议3个以上，测试单机，直接安装即可

4，先运行Elasticsearch，不然kibana无法启动

添加elastic的yum源

https://www.elastic.co/guide/en/beats/filebeat/7.14/setup-repositories.html#_yum

1. 下载并安装公共签名密钥

   sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
   

2. 在您的目录中创建一个带有.repo扩展名(例如，elastic.repo)的/etc/yum.repos.d/文件并添加以下行：

   [elastic-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
   

filebeat安装

下载地址：https://www.elastic.co/cn/downloads/beats/filebeat

filebeat建议使用rpm包安装，直接使用systemctl去启动服务，因为filebeat需要部署到各个子节点

yum安装
https://www.elastic.co/guide/en/beats/filebeat/7.14/setup-repositories.html#_yum

当然也支持widnows

systemctl enable filebeat
systemctl start filebeat

配置文件目录

/etc/filebeat

kibana安装

下载地址：https://www.elastic.co/cn/downloads/kibana

如果你上面的操作是ok的，直接

yum install kibana

启动

systemctl enable kibana
systemctl start kibana

配置文件

/etc/kibana

kibana.yml

端口5601

访问IP+端口

grafana安装

https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.1.3-1.x86_64.rpm
sudo yum install grafana-enterprise-8.1.3-1.x86_64.rpm

启动grafana

systemctl start grafana-server
systemctl enable grafana-server

端口3000

访问ip+端口

elasticsearch安装

yum search elasticsearch
yum install elasticsearch

bin版本下载： wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.1-linux-x86_64.tar.gz

单机运行可以直接yum安装，注意es需要非root账号运行

在配置文件elasticsearch.yml ，这样elasticsearch-head可以访问了

http.cors.enabled: true
http.cors.allow-origin: "*"

重启服务

systemctl stop elasticsearch
systemctl start elasticsearch

 插件地址

usr/share/elasticsearch/plugins

一些参考的基础配置

#集群名称,建议修改成自己的，防止误用默认集群
cluster.name: elasticsearch

#配置外网可以访问
network.host: 0.0.0.0
#单节点需要修改配置
cluster.initial_master_nodes: ["node-1"]

追加配置文件:

echo "bootstrap.memory_lock: false" >>  /opt/elasticsearch-6.4.0/config/elasticsearch.yml
echo "bootstrap.system_call_filter: false" >> /opt/elasticsearch-6.4.0/config/elasticsearch.yml

echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 131072" >> /etc/security/limits.conf
echo "* soft nproc 4096" >> /etc/security/limits.conf
echo "* hard nproc 4096" >> /etc/security/limits.conf

echo "vm.max_map_count=655360" >> /etc/sysctl.conf

保存后执行
sysctl -p

5.配置jvm内存大小
cd /opt/elasticsearch-6.4.0/config/
修改文件jvm.options
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms8g
-Xmx8g

echo "export ES_HEAP_SIZE=8g" >> /opt/elasticsearch-6.4.0/bin/elasticsearch

切换目录：cd /opt/elasticsearch-6.4.0/bin/

执行配置生效：

source "`dirname "$0"`"/elasticsearch-env

 查看已安装插件

http://ip:port/_cat/plugins

elasticsearch安装ik分词器

下载地址： https://github.com/medcl/elasticsearch-analysis-ik/releases

注意和es的匹配的版本

吧插件复制到  /usr/share/elasticsearch/plugins

elasticsearch-head安装

需要安装nodejs，不展开，若不需要提供服务，就不要安装，本地安装即可

git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head/
npm install

运行

npm run start
访问端口
http://localhost:9100/

单机配置

es

# ---------------------------------- Cluster -----------------------------------
#cluster.name: my-application
# ------------------------------------ Node ------------------------------------
node.name: node-1
#node.attr.rack: r1
# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
# --------------------------------- Discovery ----------------------------------
#discovery.seed_hosts: ["host1", "host2"]
cluster.initial_master_nodes: ["node-1"]

#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"

filebeat

# ================================== Intputs ===================================
filebeat.inputs:
#- type: log
#  enabled: false
#  paths:
#    - /var/log/*.log
#    #- c:\programdata\elasticsearch\logs\*
#
#- type: filestream
#  enabled: false
#  paths:
#    - /var/log/*.log
#    #- c:\programdata\elasticsearch\logs\*

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# =================================== Kibana ===================================
setup.kibana:
  host: "127.0.0.1:5601"
  #space.id:
# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["127.0.0.1:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


kibana

server.port: 5601

server.host: "服务器IP"

#server.basePath: ""

#server.rewriteBasePath: false

#server.publicBaseUrl: ""

#server.maxPayload: 1048576

#server.name: "your-hostname"

elasticsearch.hosts: ["http://127.0.0.1:9200"]

#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

#elasticsearch.ssl.verificationMode: full

#elasticsearch.pingTimeout: 1500

# must be a positive integer.
#elasticsearch.requestTimeout: 30000

#elasticsearch.requestHeadersWhitelist: [ authorization ]

#elasticsearch.customHeaders: {}

#elasticsearch.shardTimeout: 30000

#elasticsearch.logQueries: false

#pid.file: /run/kibana/kibana.pid

#logging.dest: stdout

#logging.silent: false

#logging.quiet: false
#logging.verbose: false

#ops.interval: 5000

# Supported languages are the following: English - en , by default , Chinese - zh-CN .
i18n.locale: "en"

#i18n.locale: "zh-CN" 设置成中文