ELK快速搭建教程

一、 Elasticsearch安装

环境准备
系统： Centos7
防火墙： 关闭
Sellinux: 关闭
JDK 1.8（备注至少是1.8版本的JDK）

image.png

1、下载

https://www.elastic.co/downloads/elasticsearch

image.png

[elk@localhost elasticsearch]$ pwd
/usr/local/src/elasticsearch
[elk@localhost elasticsearch]$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.tar.gz
--2018-10-05 23:26:43-- https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 54.225.221.128, 54.225.214.74, 184.73.245.233, ...
Connecting to artifacts.elastic.co (artifacts.elastic.co)|54.225.221.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 97914519 (93M) [application/x-gzip]
Saving to: ‘elasticsearch-6.4.2.tar.gz’
100%[====================================================================================================================================================================================================================================>] 97,914,519 2.55MB/s in 24s   
2018-10-05 23:27:09 (3.84 MB/s) - ‘elasticsearch-6.4.2.tar.gz’ saved [97914519/97914519]
[elk@localhost elasticsearch]$ ll
total 95620
-rw-rw-r--. 1 elk elk 97914519 Oct 2 06:58 elasticsearch-6.4.2.tar.gz

2、解压

[elk@localhost elasticsearch]$ tar -zxvf elasticsearch-6.4.2.tar.gz

3、配置elasticsearch.yml配置文件

在config目录下的elasticsearch.yml文件中你可以设置es服务的端口号以及网络ip等

image.png

4、启动es

切换到bin目录下

[elk@localhost bin]$ ./elasticsearch

启动的过程可能有点缓慢，请耐心等待
启动报错：
问题一：max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决方案：
切换到root用户
编辑 /etc/sysctl.conf，追加以下内容：
vm.max_map_count=655360
保存后，执行：
sysctl -p

问题二：ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]
解决：切换到root用户，编辑limits.conf 添加类似如下内容
vi /etc/security/limits.conf
添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

问题三：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
解决：切换到root用户，进入limits.d目录下修改配置文件。
vi /etc/security/limits.d/90-nproc.conf
修改如下内容：

* soft nproc 1024

修改为

* soft nproc 2048

重新启动，成功。

image.png

5、验证es是否启动成功

[elk@localhost ~]$ curl localhost:9200
{
    "name": "bWZVh7O",
    "cluster_name": "elasticsearch",
    "cluster_uuid": "9Rvwbh7bQs6SsRALTwefNQ",
    "version": {
        "number": "6.4.2",
        "build_flavor": "default",
        "build_type": "tar",
        "build_hash": "04711c2",
        "build_date": "2018-09-26T13:34:09.098244Z",
        "build_snapshot": false,
        "lucene_version": "7.4.0",
        "minimum_wire_compatibility_version": "5.6.0",
        "minimum_index_compatibility_version": "5.0.0"
    },
    "tagline": "You Know, for Search"
}

或者在浏览器输入：

image.png

二、 Logstash的安装

1、下载

[elk@localhost elasticsearch]$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz
--2018-10-05 23:41:42-- https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 54.235.82.130, 107.21.237.188, 23.21.67.46, ...
Connecting to artifacts.elastic.co (artifacts.elastic.co)|54.235.82.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 153924169 (147M) [application/x-gzip]
Saving to: ‘logstash-6.4.2.tar.gz’
100%[====================================================================================================================================================================================================================================>] 153,924,169 2.27MB/s in 56s   
2018-10-05 23:42:40 (2.62 MB/s) - ‘logstash-6.4.2.tar.gz’ saved [153924169/153924169]
[elk@localhost elasticsearch]$ ll
total 245940
drwxr-xr-x. 9 elk elk 155 Oct 5 23:33 elasticsearch-6.4.2
-rw-rw-r--. 1 elk elk 97914519 Oct 2 06:58 elasticsearch-6.4.2.tar.gz
-rw-rw-r--. 1 elk elk 153924169 Oct 2 06:59 logstash-6.4.2.tar.gz
[elk@localhost elasticsearch]$

2、解压

[elk@localhost elasticsearch]$ tar -zxvf logstash-6.4.2.tar.gz

3、配置文件

在config目录下的logstash.yml我这里也都采用的是默认的,大家可以根据自己的需求自行设置

[elk@localhost config]$ vim logstash.yml

4、设置tomcat的serer.xml

[elk@localhost conf]$ pwd
/usr/local/src/tomcat/apache-tomcat-8.5.34/conf
[elk@localhost conf]$ vim server.xml

将最后的pattern后面的参数更改如下

pattern="%h %l %u %t "%r" %s %b %D "%{Referer}i" "%{User-Agent}i""

image.png

5、创建root-tomcat.conf文件

[elk@localhost logstash-6.4.2]$ pwd
/usr/local/src/elasticsearch/logstash-6.4.2
[elk@localhost logstash-6.4.2]$ vim root-tomcat.conf

新增内容

input {
    file {  
        path => "/usr/local/src/tomcat/apache-tomcat-8.5.34/logs/localhost_access_log*.txt"  
        sincedb_path => "/usr/local/src/elasticsearch/logstash-6.4.2/config/sincedb_apache_access_log.txt"
        type => "apache_access_log"
        add_field => {"tomcatip" => "192.168.1.23"}
    }
}
filter{
    if [type] == "apache_access_log" {
        grok{
            match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:responsetime} \"(?:%{URI:referrer}|-)\" %{QS:agent}" }
        }
        ruby {
            code => "event.timestamp.time.localtime"
        }
        date{
            match => [ "timestamp", "dd/MM/yyyy:HH:mm:ss Z" ]
            target => ["writetime"]
        }
        mutate {
            convert => { 
                "response" => "integer" 
                "bytes" => "integer" 
                "responsetime" => "integer" 
            }
        }
    }
    
}
output {
    if [type] == "apache_access_log" {
        elasticsearch {
            hosts => ["192.168.1.23:9200"]
            index => "logstash-apacheaccesslog-%{+YYYY.MM.dd}"
        }
    }
}

6、启动logstash

[elk@localhost logstash-6.4.2]$ ./bin/logstash -f root-tomcat.conf

image.png

三、 安装kibana

1、下载

[elk@localhost elasticsearch]$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz
--2018-10-06 00:21:55-- https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 54.225.221.128, 107.21.237.188, 107.21.202.15, ...
Connecting to artifacts.elastic.co (artifacts.elastic.co)|54.225.221.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 188077286 (179M) [application/x-gzip]
Saving to: ‘kibana-6.4.2-linux-x86_64.tar.gz’
100%[====================================================================================================================================================================================================================================>] 188,077,286 4.75MB/s in 41s   
2018-10-06 00:22:39 (4.34 MB/s) - ‘kibana-6.4.2-linux-x86_64.tar.gz’ saved [188077286/188077286]
[elk@localhost elasticsearch]$ ll
total 429616
drwxr-xr-x. 9 elk elk 155 Oct 5 23:33 elasticsearch-6.4.2
-rw-rw-r--. 1 elk elk 97914519 Oct 2 06:58 elasticsearch-6.4.2.tar.gz
-rw-rw-r--. 1 elk elk 188077286 Oct 2 06:58 kibana-6.4.2-linux-x86_64.tar.gz
drwxrwxr-x. 13 elk elk 4096 Oct 6 00:15 logstash-6.4.2
-rw-rw-r--. 1 elk elk 153924169 Oct 2 06:59 logstash-6.4.2.tar.gz

2、解压

[elk@localhost elasticsearch]$ tar -zxvf kibana-6.4.2-linux-x86_64.tar.gz

3、修改配置文件

[elk@localhost kibana-6.4.2-linux-x86_64]$ cd config/
[elk@localhost config]$ vim kibana.yml

image.png

4、启动

[elk@localhost kibana-6.4.2-linux-x86_64]$ ./bin/kibana

通过提示，无任何错误表示启动完成。此时通过访问http://192.168.1.23:5601即可；
在访问这个页面之前别忘记启动tomcat

四、 成功的喜悦

如下图，大家可以开始elk之旅了

image.png