Logstash+Elasticsearch+Kibana+Nginx set up our private log query system

Logstash+Elasticsearch+Kibana+S3+Nginx

build our private log query system

• System structure

• How to setup

  • Logstash-index (Logstash server)

    yum -y install java-1.7.0-openjdk

  • Install and configure Elasticsearch ( Logstash 1.4.2 recommends Elasticsearch 1.1.1 )

    rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

    vi /etc/yum.repos.d/elasticsearch.repo  ( Add the following repository configuration )

    [elasticsearch-1.1]

    name=Elasticsearch repository for 1.1.x packages

    baseurl=http://packages.elasticsearch.org/elasticsearch/1.1/centos

    gpgcheck=1

    gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

    enabled=1

  • yum -y install elasticsearch-1.1.1

  • vi /etc/elasticsearch/elasticsearch.ymlAdd the following line somewhere in the file, to disable dynamic scripts      script.disable_dynamic: true

  • You will also want to restrict outside access to your Elasticsearch instance, so outsiders can't read your data or shutdown your Elasticseach cluster through the HTTP API. Find the line that specifies network.host and uncomment it so it looks like this:

        network.host: localhost

    
    

  • Then disable multicast by finding the discovery.zen.ping.multicast.enabled item and uncommenting so it looks like this:    discovery.zen.ping.multicast.enabled: false

  • chkconfig --add elasticsearch

  • service elasticsearch restart

• Install Kibana (  Logstash 1.4.2 recommends Kibana 3.0.1 )

  • cd ~; curl -O https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gztar xvf kibana-3.0.1.tar.gzvi ~/kibana-3.0.1/config.js  ( replace the port number (9200 by default) with 80 )    elasticsearch: "http://"+window.location.hostname+":80"

• 
  

  • mkdir -p /usr/share/nginx/kibana3

  • cp -R ~/kibana-3.0.1/* /usr/share/nginx/kibana3/

• Install Nginx

  • rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

  • yum -y install nginx

  • cd ~; curl -OL https://github.com/elasticsearch/kibana/raw/master/sample/nginx.conf

  • vi nginx.conf  ( Find and change the values of the server_name to your FQDN (or localhost if you aren't using a domain name) and root to where we installed Kibana, so they look like the following entries )

server_name FQDN;
root  /usr/share/nginx/kibana3;

• cp ~/nginx.conf /etc/nginx/conf.d/default.conf

• htpasswd -c /etc/nginx/conf.d/kibana.myhost.org.htpasswd linuxblind

• Password: linuxblind

• chkconfig --levels 235 nginx on

• service nginx restart

• Install and configure Logstash

• vi /etc/yum.repos.d/logstash.repo (Add the following repository configuration)

[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

• yum -y install logstash-1.4.2

• vi /etc/logstash/conf.d/logstash.conf ( see below)

• service logstash start

• 
  

  • vi /etc/yum.repos.d/logstash.repo (Add the following repository configuration)

  • yum -y install java-1.7.0-openjdk

  • cd /etc/pki/tls; openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 -subj "/C=CN/ST=Beijing/L=Beijing/O=IT/CN=logstash.linuxblind.com"  -keyout logstash.linuxblind.com.key -out logstash.linuxblind.com.certLogstash agent

  • Generate SSL Certificate

  • Install JDK1.7

  • Install and configure Logstash

[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

• yum -y install logstash-1.4.2

• service logstash start

=============== Logstash Collect maillog (sendmail) =====================

Step1:

• Copy SSL certificate from logstash-indexer to logstash-agent

  • scp /etc/pki/tls/certs/logstash.linuxblind.com.cert user@logstash_agent_IP:/tmp

Step2:

• In logstash-agent

• Install and Configure logstash-forwarder

  • Change content like below:

  • cd ~; curl -O http://packages.elasticsearch.org/logstashforwarder/centos/logstash-forwarder-0.3.1-1.x86_64.rpm

  • rpm -ivh ~/logstash-forwarder-0.3.1-1.x86_64.rpm

  • cd /etc/init.d/; curl -o logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_init

  • chmod +x logstash-forwarder

  • curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig

  • vi /etc/sysconfig/logstash-forwarder

LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100"

• cp /tmp/logstash.linuxblind.com.cert /etc/pki/tls/certs/

• vi /etc/logstash-forwarder

{

"network": {

    "servers": [ "logstash.linuxblind.com:5000" ],

   "timeout": 15,

   "ssl certificate": "/etc/pki/tls/certs/logstash.linuxblind.com.cert",

   "ssl key": "/etc/pki/tls/private/logstash.linuxblind.com.key",

   "ssl ca": "/etc/pki/tls/certs/logstash.linuxblind.com.cert"

 },

 "files": [

   {

     "paths": [

       "/var/log/message"

      ],

     "fields": { "type": "syslog" }

   },

   {

     "paths": [

       "/var/log/maillog"

      ],

     "fields": { "type": "maillog" }

   }

  ]

}

• chkconfig logstash-forwarder on

• /etc/init.d/logstash-forwarder start

• crtl+c  # exit

Step3:

• In logstash-indexer

  • Modified /opt/logstash/patterns/grok-patterns and append following content

# Match for sendmail

#EMAIL [.a-zA-Z0-9_-]+@%{IPORHOST}

LOGIN [.a-zA-Z0-9_-]+

EMAIL %{LOGIN}@%{IPORHOST}

DSN [0-9][.][0-9][.][0-9]

QID [A-za-z0-9]{14}

# Match a relay that gives us a QID in the return status.

SENDMAIL_TO_1 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\], dsn=%{DSN:dsn}, stat=%{DATA:status} \(%{QID:qid} %{GREEDYDATA:status_message}\)

# Match a relay that does NOT give us a QID in the return status.

SENDMAIL_TO_2 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\], dsn=%{DSN:dsn}, stat=%{DATA:status} \(%{GREEDYDATA:status_message}\)

# Match a message with no relay IP address or status message.

SENDMAIL_TO_3 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay}, dsn=%{DSN:dsn}, stat=%{GREEDYDATA:status}

# Match a message with no relay info at all.

SENDMAIL_TO_4 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ stat=%{GREEDYDATA:status}

### TODO - match multiple recipients in To: field.

SENDMAIL_TO_5 %{SYSLOGBASE} %{QID:qid}: to=(<%{EMAIL:to}>,)+ (%{WORD}=%{DATA},)+ %{GREEDYDATA:status}

SENDMAIL_TO (%{SENDMAIL_TO_1}|%{SENDMAIL_TO_2}|%{SENDMAIL_TO_3}|%{SENDMAIL_TO_4})

SENDMAIL_FROM %{SYSLOGBASE} %{QID:qid}: from=<%{EMAIL:from}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\]

SENDMAIL_OTHER_1 %{SYSLOGBASE} %{QID:qid}: %{GREEDYDATA:message}

SENDMAIL_OTHER_2 %{SYSLOGBASE} STARTTLS=(client|server), relay=(\[)?%{IPORHOST:relay}(\])?%{GREEDYDATA:message}

SENDMAIL_OTHER_3 %{SYSLOGBASE} STARTTLS: %{GREEDYDATA:message}

SENDMAIL_OTHER_4 %{SYSLOGBASE} ruleset=tls_server, arg1=SOFTWARE, relay=%{IPORHOST:relay}, %{GREEDYDATA:message}

SENDMAIL_OTHER_5 %{SYSLOGBASE} STARTTLS=client, error: %{GREEDYDATA:message}

SENDMAIL_RELAY %{SYSLOGBASE} ruleset=check_relay, arg1=(\[)?%{IPORHOST}(\])?, arg2=%{IP:ip}, relay=(\[)?%{IPORHOST:relay}(\])??%{GREEDYDATA:message}

SENDMAIL_AUTH_1 %{SYSLOGBASE} AUTH=server, relay=%{IPORHOST:relay} \[%{IP}\]( \(may be forged\))?, authid=%{LOGIN:user}(@%{IPORHOST})?, %{GREEDYDATA:message}

SENDMAIL_AUTH_2 %{SYSLOGBASE} AUTH=server, relay=\[%{IP}\], authid=%{LOGIN:user}(@%{IPORHOST})?, %{GREEDYDATA:message}

SENDMAIL_AUTH (%{SENDMAIL_AUTH_1}|%{SENDMAIL_AUTH_2})

SENDMAIL_OTHER (%{SENDMAIL_OTHER_1}|%{SENDMAIL_OTHER_2}|%{SENDMAIL_OTHER_3}|%{SENDMAIL_OTHER_4}|%{SENDMAIL_OTHER_5})

SENDMAIL (%{SENDMAIL_TO}|%{SENDMAIL_FROM}|%{SENDMAIL_OTHER}|%{SENDMAIL_AUTH}|%{SENDMAIL_RELAY})

• Modified /etc/logstash/conf.d/10-syslog.conf and Add following content in right place.

if [type] == "maillog" {

               grok {  

                        match => [ "message", "%{SENDMAIL_TO_1}",

                                   "message", "%{SENDMAIL_TO_2}",

                                   "message", "%{SENDMAIL_TO_3}",

                                   "message", "%{SENDMAIL_TO_4}",

                                   "message", "%{SENDMAIL_TO_5}",

                                   "message", "%{SENDMAIL_TO}",

                                   "message", "%{SENDMAIL_FROM}",

                                   "message", "%{SENDMAIL_OTHER_1}",

                                   "message", "%{SENDMAIL_OTHER_2}",

                                   "message", "%{SENDMAIL_OTHER_3}",

                                   "message", "%{SENDMAIL_OTHER_4}",

                                   "message", "%{SENDMAIL_OTHER_5}",

                                   "message", "%{SENDMAIL_RELAY}",

                                   "message", "%{SENDMAIL_AUTH_1}",

                                   "message", "%{SENDMAIL_AUTH_2}",

                                   "message", "%{SENDMAIL_AUTH}",

                                   "message", "%{SENDMAIL_OTHER}",

                                   "message", "%{SENDMAIL}"

                                   ]

               }

       }

• service logstash restart

============= Configuration For Logstash with Logstash-forwarder =============

• On Logstash master

  • cat 01-lumberjack-input.conf

input {

lumberjack {

port => 5000

type => "logs"

ssl_certificate => "/etc/pki/tls/certs/logstash.linuxblind.com.cert"

ssl_key => "/etc/pki/tls/private//logstash.linuxblind.com.key"

}

}

• cat 10-syslog.conf

filter {

if [type] == "syslog" {

      grok {

          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

          add_field => [ "received_at", "%{@timestamp}" ]

           add_field => [ "received_from", "%{host}" ]

      }

      syslog_pri { }

      date {

          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

      }

 }

if [type] == "maillog" {

      grok {  

          match => [ "message", "%{SENDMAIL_TO_1}",

                      "message", "%{SENDMAIL_TO_2}",

                      "message", "%{SENDMAIL_TO_3}",

                     "message", "%{SENDMAIL_TO_4}",

                     "message", "%{SENDMAIL_TO_5}",

                      "message", "%{SENDMAIL_TO}",

                      "message", "%{SENDMAIL_FROM}",

                     "message", "%{SENDMAIL_OTHER_1}",

                     "message", "%{SENDMAIL_OTHER_2}",

                    "message", "%{SENDMAIL_OTHER_3}",

                     "message", "%{SENDMAIL_OTHER_4}",

                     "message", "%{SENDMAIL_OTHER_5}",

                      "message", "%{SENDMAIL_RELAY}",

                     "message", "%{SENDMAIL_AUTH_1}",

                      "message", "%{SENDMAIL_AUTH_2}",

                     "message", "%{SENDMAIL_AUTH}",

                     "message", "%{SENDMAIL_OTHER}",

                     "message", "%{SENDMAIL}"

           ]

      }

  }

  if [type] == "apache" {

      grok {

          pattern => "%{COMBINEDAPACHELOG}"

      }

  }

}

• cat 30-lumberjack-output.conf

  output {

  elasticsearch { host => localhost }

  stdout { codec => rubydebug }

  }

• On Logstash agent

  • cat /etc/logstash-forwarder

{

 "network": {

   "servers": [ "logstash.linuxblind.com:5000" ],

 "timeout": 15,

   "ssl certificate": "/etc/pki/tls/certs/logstash.linuxblind.com.cert",

   "ssl key": "/etc/pki/tls/private/logstash.linuxblind.com.key",

   "ssl ca": "/etc/pki/tls/certs/logstash.linuxblind.com.cert"

 },

 "files": [

   {

     "paths": [

       "/var/log/message"

         ],

        "fields": { "type": "syslog" }

      },

      {

             "paths": [

            "/var/log/maillog"

              ],

             "fields": { "type": "maillog" }

     }

   ]

}

================================================================

• How to package rpm for logstash-forwarder(32bit) and then Install logstash-forwarder on Amazon Linux (32bit) - The first time, you build environment need release is 32bit (i686/i386)

  • make rpm  (choose this)

  • make deb

  • yum -y install gcc ruby-devel golang git rpm-build

  • gem install fpm

  • git clone git://github.com/elasticsearch/logstash-forwarder.git

  • cd logstash-forwarder

  • go build

  • Make packages, either:

  • rpm -ivh logstash-forwarder-0.3.1-1.i686.rpm

  • wget https://raw.githubusercontent.com/jamtur01/logstashbook-code/master/code/4/logstash_forwarder_redhat_init --no-check-certificate

  • mv logstash_forwarder_redhat_init logstash-forwarder

  • chmod +x logstash-forwarder

  • cp -f logstash-forwarder /etc/init.d/logstash-forwarder

• The Second Time

  • Change content like below:

  • cd ~;

  • rpm -ivh logstash-forwarder-0.3.1-1.i686.rpm  # you can download from below

  • curl -o /etc/pki/tls/certs/logstash..crt http://logstash.linuxblind.com/logstash/logstash-forwarder.crt    

  • curl -o /etc/init.d/logstash-forwarder http://logstash.linuxblind.com/logstash/logstash-forwarder  

  • chkconfig logstash-forwarder on

  • curl -o /etc/logstash-forwarder http://logstash.linuxblind.com/logstash/logstash-forwarder.conf  

  • curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig

  • vi /etc/sysconfig/logstash-forwarder

LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100"

• Modified /etc/logstash-forwarder to match log paths

• cd /var/log/; /etc/init.d/logstash-forwarder start

作者注：写的比较粗，有点不堪入目，有时间再仔细整理下, 另外上面你下载不了，可以找我索取。

QQ：1037447289