基于CentOS6.5或Ubuntu14.04下Suricata里搭配安装 ELK (elasticsearch, logstash, kibana)（图文详解）...

 

 

　　前期博客

基于CentOS6.5下Suricata（一款高性能的网络IDS、IPS和网络安全监控引擎）的搭建（图文详解）（博主推荐）

 

基于Ubuntu14.04下Suricata（一款高性能的网络IDS、IPS和网络安全监控引擎）的搭建（图文详解）（博主推荐）

 

 

 

 

 

 

 　　参考官网

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

　　注意，这官网，默认是适合Ubuntu系统的安装手法。

 

 

 

 

　　

　　首先，

　　Make sure your Suricata is compiled/installed with libjansson support enabled:

[root@suricata ~]# suricata --build-info
This is Suricata version 3.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON TLS 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-18), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                no
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no
  Hyperscan support:                       no
  Libnet support:                          yes

  Suricatasc install:                      yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /usr/local/var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /usr/local/var

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native
  PCAP_CFLAGS                               
  SECCFLAGS                                
[root@suricata ~]# 

 

 

 

 

 

 

　　然后，需要配置

Configure suricata

[root@suricata suricata-3.1]# pwd
/root/suricata-3.1
[root@suricata suricata-3.1]# ls
aclocal.m4             compile       config.h.in   config.status  configure.ac  depcomp     libhtp   ltmain.sh  Makefile.am  qa                scripts   suricata.yaml
ChangeLog              config.guess  config.log    config.sub     contrib       doc         libtool  m4         Makefile.in  reference.config  src       suricata.yaml.in
classification.config  config.h      config.rpath  configure      COPYING       install-sh  LICENSE  Makefile   missing      rules             stamp-h1  threshold.config
[root@suricata suricata-3.1]# sudo vim suricata.yaml

 

 

 

 # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
      #prefix: "@cee: " # prefix to prepend to each log entry
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      #redis:
      #  server: 127.0.0.1
      #  port: 6379
      #  mode: list ## possible values: list (default), channel
      #  key: suricata ## key or channel to use (default to suricata)
      # Redis pipelining set up. This will enable to only do a query every
      # 'batch-size' events. This should lower the latency induced by network
      # connection at the cost of some memory. There is no flushing implemented
      # so this setting as to be reserved to high traffic suricata.
      #  pipelining:
      #    enabled: yes ## set enable to yes to enable query pipelining
      #    batch-size: 10 ## number of entry to keep in buffer
 types:
        - alert:
            # payload: yes             # enable dumping payload in Base64
            # payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
            # payload-printable: yes   # enable dumping payload in printable (lossy) format
            # packet: yes              # enable dumping of packet (without stream segments)
            http: yes                # enable dumping of http fields
            tls: yes                 # enable dumping of tls fields
            ssh: yes                 # enable dumping of ssh fields
            smtp: yes                # enable dumping of smtp fields

            # HTTP X-Forwarded-For support by adding an extra field or overwriting
            # the source or destination IP address (depending on flow direction)
            # with the one reported in the X-Forwarded-For HTTP header. This is
            # helpful when reviewing alerts for traffic that is being reverse
            # or forward proxied.
            xff:
              enabled: no
              # Two operation modes are available, "extra-data" and "overwrite".
              mode: extra-data
              # Two proxy deployments are supported, "reverse" and "forward". In
              # a "reverse" deployment the IP address used is the last one, in a
              # "forward" deployment the first IP address is used.
              deployment: reverse
              # Header name where the actual IP address will be reported, if more
              # than one IP address is present, the last IP address will be the
              # one taken into consideration.
              header: X-Forwarded-For
 - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional http fields to be included in eve-log
            # the example below adds three additional fields when uncommented
            #custom: [Accept-Encoding, Accept-Language, Authorization]
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: no   # force logging magic on all logged files
            force-md5: no     # force logging of md5 checksums
        #- drop:
        #    alerts: no       # log alerts that caused drops
 - smtp:
            #extended: yes # enable this for extended logging information
            # this includes: bcc, message-id, subject, x_mailer, user-agent
            # custom fields logging from the list:
            #  reply-to, bcc, message-id, subject, x-mailer, user-agent, received,
            #  x-originating-ip, in-reply-to, references, importance, priority,
            #  sensitivity, organization, content-md5, date
            #custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            # output md5 of fields: body, subject
            # for the body you need to set app-layer.protocols.smtp.mime.body-md5
            # to yes
            #md5: [body, subject]

        - ssh
        - stats:
            totals: yes       # stats for all threads merged together
            threads: no       # per thread stats
            deltas: no        # include delta values
        # bi-directional flows
        - flow
        # uni-directional flows
        #- netflow

 

 

 

 　　改为

# "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata" 
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh
        - smtp

 

 

 

 　　然后，接下是安装

Install ELK (elasticsearch, logstash, kibana)

　　先卸载自带的openjdk

rpm -qa|grep java    

rpm -e --nodeps tzdata-java-2013g-1.el6.noarch
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

 

　　注意：这里一定要是jdk1.8，不然会报如下错误

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: Elasticsearch requires at least Java 8 but your Java version from /usr/local/jdk/jdk1.7.0_79/bin/java does not meet this requirement
                                                           [FAILED]
[root@suricata ~]# 

 

 

 

 　　官网提示了，建议安装orcal 的 jdk。

　　我这里安装jdk18。

[root@suricata suricata-3.1]# cd /usr/local/
[root@suricata local]# ls
bin  etc  games  include  lib  lib64  libexec  sbin  share  src  var
[root@suricata local]# mkdir jdk
[root@suricata local]# cd jdk/
[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total 0
[root@suricata jdk]# rz

 

 

 

 

[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total 176992
-rw-r--r--. 1 root root 181238643 Sep  3  2016 jdk-8u60-linux-x64.tar.gz
[root@suricata jdk]# tar -zxvf jdk-8u60-linux-x64.tar.gz 

 

 

 

 

　　然后，再vim /etc/profile

#java
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

 

 

 

 

 

　　文件生效

source /etc/profile

 

 

 

 

 

 

 

 

 　　然后，接下来就是安装ELk和配置JDK。（注意以下只是个例子）

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.1.deb
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb

tar -C /var/www/ -xzf kibana-3.0.0.tar.gz
dpkg -i elasticsearch-1.6.1.deb
dpkg -i logstash_1.5.3-1_all.deb

 

 

 

 

 

 

 

 

 

　　我这里啊，Elasticsearch选取5.5.0

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 　　所以，总的是

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.deb

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.deb

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz

 

　　最后下载，得到

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]# 

 

 

 

 

 

　　然后，进行对kibana的安装

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]# tar -C /var/www/ -xzf kibana-5.5.0-linux-x86_64.tar.gz

 

 

 　　得到

[root@suricata www]# pwd
/var/www
[root@suricata www]# ls
cgi-bin  error  html  icons  kibana-5.5.0-linux-x86_64
[root@suricata www]# 

 

 

 

 　　若大家没安装过这个命令，则安装下即可。

[root@suricata ~]# yum -y install dpkg 

 

 

 

 

 　　然后，进行对elasticsearch的安装

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]# dpkg -i elasticsearch-5.5.0.deb

 

 

　　这里啊，我建议大家还是去安装tar.gz包，这种方式比较好。

 

 

 　　所以，后面，我还是下载了，rpm包，以这种方式来进行安装。

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm

 

 

 

 　　

 　　　进行elasticsearch-5.5.0的安装

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                elasticsearch-5.5.0.rpm  install.log.syslog                logstash-5.5.0.deb  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log              kibana-5.5.0-linux-x86_64.tar.gz  logstash-5.5.0.rpm  Pictures  suricata-3.1  Templates

[root@suricata ~]# rpm -ivh elasticsearch-5.5.0.rpm
warning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
1:elasticsearch ########################################### [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start

 

　安装完，默认即有

 

 

 

 

 

 

 　　然后，进行logstash的安装

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                elasticsearch-5.5.0.rpm  install.log.syslog                logstash-5.5.0.deb  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log              kibana-5.5.0-linux-x86_64.tar.gz  logstash-5.5.0.rpm  Pictures  suricata-3.1  Templates
[root@suricata ~]# rpm -ivh logstash-5.5.0.rpm
warning: logstash-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                ########################################### [100%]
   1:logstash               ########################################### [100%]
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: /usr/bin/java: No such file or directory
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: exec: /usr/bin/java: cannot execute: No such file or directory
Unable to install system startup script for Logstash.
[root@suricata ~]# 

 

　　安装完，默认即有

 

 

 

 

 

 　　然后，下面是

Logstash configuration

[root@suricata conf.d]# pwd
/etc/logstash/conf.d
[root@suricata conf.d]# ll
total 0
[root@suricata conf.d]# touch /etc/logstash/conf.d/logstash.conf

 

 

 

　　插入如下内容

input {
  file { 
    path => ["/var/log/suricata/eve.json"]
    sincedb_path => ["/var/lib/logstash/"]
    codec =>   json 
    type => "SuricataIDPS" 
  }

}

filter {
  if [type] == "SuricataIDPS" {
    date {
      match => [ "timestamp", "ISO8601" ]
    }
    ruby {
      code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;" 
    }
  }

  if [src_ip]  {
    geoip {
      source => "src_ip" 
      target => "geoip" 
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip" 
          target => "geoip" 
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float" ]
        }
      }
    }
  }
}

output { 
  elasticsearch {
    host => localhost
    #protocol => http
  }
}

 

 

 

 

　　然后，再

Configure the start-up services

update-rc.d elasticsearch defaults 95 10 （这两个是ubuntu系统里的做法，我这里是centos系统，不需要）
update-rc.d logstash defaults

 

service httpd start                    启动apache
service elasticsearch start
service logstash start

或者

启动服务sudo /etc/init.d/elasticsearch start
停止服务sudo /etc/init.d/elasticsearch stop

启动服务sudo /etc/init.d/logstash/start
停止服务sudo /etc/init.d/logstashstop

 

 

　　如果大家在启动过程中，出现如下问题。

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
[root@suricata ~]# 

 

　　则见

ElasticSearch里JAVA环境丢失的问题

 

　　-rwxr-x---. 1 root root  5113 Jul  1 07:20 elasticsearch   ，经过

[root@suricata init.d]# chmod 755  elasticsearch
[root@suricata init.d]# pwd
/etc/init.d
[root@suricata init.d]# 

 

　　改为

-rwxr-xr-x. 1 root root  5113 Jul  1 07:20 elasticsearch

　　

　　大家，自行按照自己的jdk位置。

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60

　　然后，就可以成功了。

 

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch:                                    [  OK  ]
[root@suricata ~]# 

 

 

 

　　然后，大家也许在启动，会遇到这样的问题：

Logstash默认的配置文件位置为./config和/etc/logstash/，后者已经存在，但直接运行依然会报错：

　　WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash.

 

　　见

Logstash使用grok解析IIS日志

 

　　简单起见，直接在Logstash根目录创建软链接即可：

cd /usr/share/logstash

[root@suricata logstash]# pwd
/usr/share/logstash
[root@suricata logstash]# ln -s /etc/logstash ./config
[root@suricata logstash]#

　　设置配置文件实时生效，从而不用频繁地启停Logstash。

 

　　修改/etc/logstash/logstash.yml

config.reload.automatic: true

 

 

 

 

　　Logstash的运行

　　与Elasticsearch、Kibana不同，Logstash默认并不作为系统服务安装，我们也不建议作为服务启动。主要原因为：大多数情况下，Elasticsearch、Kibana在每台服务器上只有一个实例，可以作为服务启动；而一个Logstash实例一般只服务于一个数据流，因此一台服务器上经常运行多个实例 。编写一个配置文件helloworld.conf进行测试。注意，不要把该文件放在/etc/logstash/conf.d下，以避免不同实例间产生混淆，因为该文件夹是Logstash默认读取的文件夹。简单起见，可直接放在Logstash根目录下。

input { 
    stdin {} 
}
output { 
    stdout {}
}

　　这是一个命令行输入、命令行输出的Logstash实例，运行成功则说明已配置正确： 

bin/logstash -f helloworld.conf 

 

　　虽然如此，很多时候我们还是希望各个Logstash实例能随系统而启动并在后台运行。将其加入计划任务即可： 

 nohup /usr/share/logstash/bin/logstash -f /usr/share/logstash/helloworld.conf > /dev/null &

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

　　然后，

　　Now make sure Suricata is running and you have logs written in your JSON log files

http://localhost/kibana-5.5.0

 

 

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

 

 

 

　　继续