一、CentOS7.4下Elastic Stack 6.2.4日志系统搭建

Elasticsearch是一个高度可扩展的开源全文搜索和分析引擎。它允许您快速，近实时地存储，搜索和分析大量数据。它通常用作支持具有复杂搜索功能和需求的应用程序的底层引擎/技术。

        Logstash是一个开源的用于收集,分析和存储日志的工具。

        Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以汇总、分析和搜索重要数据日志。

        Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称，可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch，然后进行后续的数据分析活动。Beats由如下组成:

        1.Packetbeat：是一个网络数据包分析器，用于监控、收集网络流量信息，

                              Packetbeat嗅探服务器之间的流量，解析应用层协议，并关联到消息的处理，                                     其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、

                              MongoDB、Memcache等协议；

        2. Filebeat：用于监控、收集服务器日志文件，其已取代 logstash forwarder；

        3. Metricbeat：可定期获取外部系统的监控指标信息，其可以监控、收集

                    Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、

                               Redis、System、Zookeeper等服务；

        4. Winlogbeat：用于监控、收集Windows系统的日志信息；

        5. Create your own Beat：自定义beat ，如果上面的指标不能满足需求，elasticsarch鼓励开发者          使用go语言，扩展实现自定义的beats，只需要按照模板，实现监控的输入，日志，输出等即可。

 

        Beats 将搜集到的数据发送到 Logstash，经 Logstash 解析、过滤后，将其发送到 Elasticsearch 存储，并由 Kibana 呈现给用户。

        Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源，解决了 Logstash 在各服务器节点上占用系统资源高的问题。

一、环境

# dmidecode|grep "System Information" -A9|egrep "Manufacturer|Product"

Manufacturer: Dell Inc.

Product Name: PowerEdge R630

# uname -a

Linux linux-node2 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release

CentOS Linux release 7.4.1708 (Core)

关闭firewalld,selinux

二、下载Elasticsearch软件包

使用最新Elasticsearch6.2.4包，需要先安装Java version 1.8.0_131以上版本：

软件支持信息参考：https://www.elastic.co/support/matrix#matrix_os

安装官方文档：https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

本次基于Centos7 使用RPM安装

官网下载地址：https://www.elastic.co/downloads/elasticsearch

软件包安装如下：

2.1使用ZIP包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.zip wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.zip.sha512 shasum -a 512 -c elasticsearch-6.2.4.zip.sha512 unzip elasticsearch-6.2.4.zip cd elasticsearch-6.2.4/

2.2使用TAR包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz.sha512 shasum -a 512 -c elasticsearch-6.2.4.tar.gz.sha512 tar -xzf elasticsearch-6.2.4.tar.gz cd elasticsearch-6.2.4/

使用包安装报错：

# ./elasticsearch

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

##########/etc/elasticsearch/jvm.options添加参数

[2018-05-23T15:08:06,797][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]

at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]

▽ at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]

Caused by: java.lang.RuntimeException: can not run elasticsearch as root

at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]

... 6 more

2.3使用YUM安装

#rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch 导入PGP KEY

#vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md

#yum install elasticsearch

[root@linux-node1 yum.repos.d]# yum install elasticsearch

Loaded plugins: fastestmirror

elasticsearch-6.x | 1.3 kB 00:00:00

elasticsearch-6.x/primary | 67 kB 00:00:05

Loading mirror speeds from cached hostfile

* base: mirrors.aliyun.com

* extras: mirrors.aliyun.com

* updates: mirrors.aliyun.com

elasticsearch-6.x 180/180

Resolving Dependencies

--> Running transaction check

---> Package elasticsearch.noarch 0:6.2.4-1 will be installed

--> Finished Dependency Resolution

 

Dependencies Resolved

 

====================================================================================================================================================================

Package Arch Version Repository Size

====================================================================================================================================================================

Installing:

elasticsearch noarch 6.2.4-1 elasticsearch-6.x 28 M

 

Transaction Summary

====================================================================================================================================================================

Install 1 Package

 

Total download size: 28 M

Installed size: 31 M

Is this ok [y/d/N]: y

Downloading packages:

elasticsearch-6.2.4.rpm | 28 MB 00:02:10

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Creating elasticsearch group... OK

Creating elasticsearch user... OK

Installing : elasticsearch-6.2.4-1.noarch 1/1

### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd

sudo systemctl daemon-reload

sudo systemctl enable elasticsearch.service

### You can start elasticsearch service by executing

sudo systemctl start elasticsearch.service

Verifying : elasticsearch-6.2.4-1.noarch 1/1

 

Installed:

elasticsearch.noarch 0:6.2.4-1

 

Complete!

elasticsearch 安装完成后

# systemctl start elasticsearch 默认不记录LOG需要进行设置

取消/usr/lib/systemd/system/elasticsearch.service文件中--quiet

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# systemctl daemon-reload

# systemctl restart elasticsearch

@@@@@@@@@@@

# journalctl --unit elasticsearch

-- Logs begin at Wed 2018-05-23 14:32:54 CST, end at Wed 2018-05-23 15:53:11 CST. --

May 23 15:34:02 linux-node1 systemd[1]: Started Elasticsearch.

May 23 15:34:02 linux-node1 systemd[1]: Starting Elasticsearch...

May 23 15:34:04 linux-node1 elasticsearch[11511]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should co

...skipping...

#journalctl --unit elasticsearch --since "2016-10-30 18:17:16"可查看指定时间后的LOG

@@@@@@@@@@@

# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)

Active: active (running) since Wed 2018-05-23 15:34:02 CST; 11s ago

Docs: http://www.elastic.co

Main PID: 11511 (java)

Tasks: 14

Memory: 1.1G

CGroup: /system.slice/elasticsearch.service

└─11511 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -...

 

May 23 15:34:02 linux-node1 systemd[1]: Started Elasticsearch.

May 23 15:34:02 linux-node1 systemd[1]: Starting Elasticsearch...

May 23 15:34:04 linux-node1 elasticsearch[11511]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then y...Threads=N

Hint: Some lines were ellipsized, use -l to show in full.

# ss -tlnp |grep -E '9200|9300' ###ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。

LISTEN 0 128 ::ffff:127.0.0.1:9200 :::* users:(("java",pid=11511,fd=121))

LISTEN 0 128 ::1:9200 :::* users:(("java",pid=11511,fd=120))

LISTEN 0 128 ::ffff:127.0.0.1:9300 :::* users:(("java",pid=11511,fd=113))

LISTEN 0 128 ::1:9300 :::* users:(("java",pid=11511,fd=111))

# /usr/share/elasticsearch/bin/elasticsearch -V

#ln -s /usr/share/elasticsearch/bin/elasticsearch /bin/elasticsearch

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_171

#curl -X GET http://localhost:9200

{

"name" : "IGgk_NL",

"cluster_name" : "elasticsearch",

"cluster_uuid" : "8u-EnhG8QsatgT3r6BDWrA",

"version" : {

"number" : "6.2.4",

"build_hash" : "ccec39f",

"build_date" : "2018-04-12T20:37:28.497551Z",

"build_snapshot" : false,

"lucene_version" : "7.2.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

"tagline" : "You Know, for Search"

}

 

三、配置elasticsearch

YUM安装默认配置目录/etc/elasticsearch

# ls -lh

total 16K

-rw-rw----. 1 root elasticsearch 2.9K Apr 13 04:39 elasticsearch.yml

-rw-rw----. 1 root elasticsearch 2.8K Apr 13 04:39 jvm.options

-rw-rw----. 1 root elasticsearch 5.0K Apr 13 04:39 log4j2.properties

# chown -R elasticsearch:elasticsearch /etc/elasticsearch

ZIP和TAR安装配置目录在$ES_HOME/config目录下 （ES_HOME为解压的目录位置）

可以修改为：

ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch

 

对于包分发版，配置目录位置默认为/etc/elasticsearch。配置目录的位置也可以通过ES_PATH_CONF环境变量进行更改，但是请注意，在外壳中设置这个位置是不够的。相反，这个变量是从/etc/default/elasticsearch（用于Debian软件包）和/etc/sysconfig/elasticsearch（用于RPM包）的。您将需要在其中一个文件中编辑espathconf=/etc/elasticsearch条目，以更改配置目录位置。

config/elasticsearch.ymal中配置项说明：

请参考下面网址和官网：https://blog.csdn.net/gamer_gyt/article/details/59077189

http://www.cnblogs.com/zhaijunming5/p/7932213.html

• cluster_name 集群名称，默认为elasticsearch，这里我们设置为es5.2.1Cluster
• node.name配置节点名，用来区分节点
• network.host 是配置可以访问本节点的路由地址
• http.port 路由地址端口
• transport.tcp.port TCP协议转发地址端口
• node.master 是否作为集群的主结点 ，值为true或true
• node.data 是否存储数据，值为true或true
• discovery.zen.ping.unicast.hosts 用来配置所有用来组建集群的机器的IP地址，由于新版本是不支持多播的，因此这个值需要提前设定好，当集群需要扩展的时候，该值都要做改变，增加新机器的IP地址，如果是在一个ip上，要把TCP协议转发端口写上
• discovery.zen.minimum_master_nodes 用来配置主节点数量的最少值，如果主节点数量低于该值，闭包范围内的集群将会停止服务,之所以加粗体，是因为暂时尚未认证，下面配置为1方便集群更容易形成，即使只有一个主节点，也可以构建集群
• gateway.* 网关的相关配置
• script.* indices.* 根据需求添加的配置（可选）

# 配置文件中给出了三种配置高性能集群拓扑结构的模式,如下： # 1. 如果你想让节点从不选举为主节点,只用来存储数据,可作为负载器 # node.master: false # node.data: true # 2. 如果想让节点成为主节点,且不存储任何数据,并保有空闲资源,可作为协调器 # node.master: true # node.data: false # 3. 如果想让节点既不称为主节点,又不成为数据节点,那么可将他作为搜索器,从节点中获取数据,生成搜索结果等 # node.master: false # node.data: false

配置elasticsearch.yml

path: data: /var/lib/elasticsearch logs: /var/log/elasticsearch

或

path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch

node.name: ${HOSTNAME} network.host: ${ES_NETWORK_HOST}

四、重要配置参数参考：

https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html

转载于:https://www.cnblogs.com/icerain0/p/9115746.html