ELK监控交换机日志

一、首先部署logstash监控UDP514端口，新建一个配置文件cisco.conf

交换机是通过配置rsyslog服务器来将日志发送到日志服务器的，所以需要在logstash上配置rsyslog监听端口既514端口

[root@server-1 conf.d]# cd /etc/logstash/conf.d/
[root@server-1 conf.d]# vim cisco.conf

input{
 syslog{
   port => 514
 }
}

output{

  stdout{
   codec => rubydebug
  }
}

二、加载配置文件 

[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/cisco.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-05 14:18:11.331 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-05 14:18:11.339 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-05 14:18:12.069 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-05 14:18:12.376 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-05 14:18:12.621 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-05 14:18:13.955 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-05 14:18:14.536 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[INFO ] 2019-11-05 14:18:14.545 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-05 14:18:14.578 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:109] syslog - Starting syslog udp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:14.595 [Ruby-0-Thread-16: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:113] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:514"}

三、测试UDP

 首先看看514端口是否被监听

[root@server-1 conf.d]# netstat -tunlp|grep java
tcp6       0      0 :::5002                 :::*                    LISTEN      16102/java          
tcp6       0      0 172.28.18.69:9200       :::*                    LISTEN      18608/java          
tcp6       0      0 :::10001                :::*                    LISTEN      16102/java          
tcp6       0      0 172.28.18.69:9300       :::*                    LISTEN      18608/java          
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      19444/java          
tcp6       0      0 172.28.18.69:9600       :::*                    LISTEN      16102/java          
udp        0      0 0.0.0.0:514             0.0.0.0:*                           19444/java

然后，使用tcpdump命令在514端口从抓包，确认有数据包发送过来

[root@server-1 conf.d]# tcpdump -i em1 udp  port 514 -c 100 -n -vvv
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes

，在另外一台172.28.18.71服务器上，设置好rssyslog服务器地址，就可以发送日志到514端口了

[root@localhost ~]# vim /etc/rsyslog.conf

在"rule"下增加如下语句“*.*  @@172.28.18.69“

#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console
*.*                                                     @@172.28.18.69

重启rsyslog服务

[root@localhost ~]# systemctl restart rsyslog

然后重启登录172.28.18.71，此时在172.28.18.69上tcpdump监听的514端口显示抓包的数据

[root@server-1 conf.d]# tcpdump -i em1 udp  port 514 -c 100 -n -vvv
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:14:09.093962 IP (tos 0x0, ttl 64, id 40767, offset 0, flags [DF], proto UDP (17), length 132)
    172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 104
        Facility authpriv (10), Severity info (6)
        Msg: Nov  5 10:14:09 localhost sshd[6587]: Accepted password for root from 172.28.146.109 port 59567 ssh2
        0x0000:  3c38 363e 4e6f 7620 2035 2031 303a 3134
        0x0010:  3a30 3920 6c6f 6361 6c68 6f73 7420 7373
        0x0020:  6864 5b36 3538 375d 3a20 4163 6365 7074
        0x0030:  6564 2070 6173 7377 6f72 6420 666f 7220
        0x0040:  726f 6f74 2066 726f 6d20 3137 322e 3238
        0x0050:  2e31 3436 2e31 3039 2070 6f72 7420 3539
        0x0060:  3536 3720 7373 6832
10:14:09.101472 IP (tos 0x0, ttl 64, id 40769, offset 0, flags [DF], proto UDP (17), length 104)
    172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 76
        Facility auth (4), Severity info (6)
        Msg: Nov  5 10:14:09 localhost systemd-logind: New session 4231 of user root.
        0x0000:  3c33 383e 4e6f 7620 2035 2031 303a 3134
        0x0010:  3a30 3920 6c6f 6361 6c68 6f73 7420 7379
        0x0020:  7374 656d 642d 6c6f 6769 6e64 3a20 4e65
        0x0030:  7720 7365 7373 696f 6e20 3432 3331 206f
        0x0040:  6620 7573 6572 2072 6f6f 742e
10:14:09.101738 IP (tos 0x0, ttl 64, id 40770, offset 0, flags [DF], proto UDP (17), length 101)
    172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 73
        Facility daemon (3), Severity info (6)
        Msg: Nov  5 10:14:09 localhost systemd: Started Session 4231 of user root.
        0x0000:  3c33 303e 4e6f 7620 2035 2031 303a 3134
        0x0010:  3a30 3920 6c6f 6361 6c68 6f73 7420 7379
        0x0020:  7374 656d 643a 2053 7461 7274 6564 2053
        0x0030:  6573 7369 6f6e 2034 3233 3120 6f66 2075
        0x0040:  7365 7220 726f 6f74 2e
10:14:09.102645 IP (tos 0x0, ttl 64, id 40771, offset 0, flags [DF], proto UDP (17), length 133)
    172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 105
        Facility authpriv (10), Severity info (6)
        Msg: Nov  5 10:14:09 localhost sshd[6587]: pam_unix(sshd:session): session opened for user root by (uid=0)
        0x0000:  3c38 363e 4e6f 7620 2035 2031 303a 3134
        0x0010:  3a30 3920 6c6f 6361 6c68 6f73 7420 7373
        0x0020:  6864 5b36 3538 375d 3a20 7061 6d5f 756e
        0x0030:  6978 2873 7368 643a 7365 7373 696f 6e29
        0x0040:  3a20 7365 7373 696f 6e20 6f70 656e 6564
        0x0050:  2066 6f72 2075 7365 7220 726f 6f74 2062
        0x0060:  7920 2875 6964 3d30 29

但是查看logstash，却没有数据输出到控制台上，此时查看logstash日志

[root@server-1 log]# tail -f /home/logstash/log/logstash-plain.log

ck in start_input'"]}
[2019-11-05T10:21:56,087][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2019-11-05T10:21:56,088][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:101:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]}
[2019-11-05T10:22:01,088][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2019-11-05T10:22:01,090][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:101:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]}

从日志看出报:exception=>#错误，百度一下，发现是由于514端口，必须要root用户才能启动，而logstash默认是logstash用户启动服务，所以修改logstash服务的用户

停止logstash服务

[root@server-1 conf.d]# systemctl stop logstash

修改服务配置

[root@server-1 conf.d]# vim /etc/systemd/system/logstash.service

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

将User Group改为root

[Unit]
Description=logstash

[Service]
Type=simple
#User=logstash
#Group=logstash
User=root
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

保存，重启logstash服务

[root@server-1 conf.d]# systemctl start logstash

关闭514端口监听进程，重新加载UDP监听配置文件

[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/cisco.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-05 14:18:11.331 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-05 14:18:11.339 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-05 14:18:12.069 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-05 14:18:12.376 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-05 14:18:12.621 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-05 14:18:13.955 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-05 14:18:14.536 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[INFO ] 2019-11-05 14:18:14.545 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-05 14:18:14.578 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:109] syslog - Starting syslog udp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:14.595 [Ruby-0-Thread-16: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:113] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:53.164 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:174] syslog - new connection {:client=>"172.28.18.71:40654"}
[INFO ] 2019-11-05 14:18:53.183 [Ruby-0-Thread-18: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:174] syslog - new connection {:client=>"172.28.18.71:40656"}

再次测试发送数据

 

{
          "@version" => "1",
         "logsource" => "localhost",
          "priority" => 30,
          "facility" => 3,
              "host" => "172.28.18.71",
        "@timestamp" => 2019-11-05T06:18:53.000Z,
         "timestamp" => "Nov  5 14:18:53",
           "program" => "systemd",
    "facility_label" => "system",
          "severity" => 6,
           "message" => "Stopping System Logging Service...\n",
    "severity_label" => "Informational"
}
{
          "@version" => "1",
         "logsource" => "localhost",
          "priority" => 46,
          "facility" => 5,
              "host" => "172.28.18.71",
        "@timestamp" => 2019-11-05T06:18:53.000Z,
         "timestamp" => "Nov  5 14:18:53",
           "program" => "rsyslogd",
    "facility_label" => "syslogd",
          "severity" => 6,
           "message" => "action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ]\n",
    "severity_label" => "Informational"
}

logstash显示日志数据了，修改配置文件将日志输出到elastcisearch

output{
input{
   syslog{
    port => 514
   }
}

#输出到elastcisearch
output{  
    elasticsearch{
      hosts => ["172.28.18.69:9200"]         #elasticsearch服务地址
      index => "system-cisco-log-%{+YYYY.MM}"   #创建的索引 
    }
}

重启加载配置文件，在elastcisearch服务器上查看索引

[root@server-1 conf.d]# curl http://172.28.18.69:9200/_cat/indices
yellow open nginx-172.28.18.75-2019.11.05 WK6Zr5guQ7KSoCLPd8JjqQ 5 1 12086 0   4.5mb   4.5mb
yellow open system-cisco-log-2019.11      IR__HXPvTfe3HNtQ1HOwFw 5 1    16 0 101.7kb 101.7kb
green  open .kibana                       QkF9i3nXSAKlNLMLNROM1A 1 0     4 1  23.5kb  23.5kb

已经生成了system-cisco-log-2019.11文件

四、配置交换机

 

 

 

 这样，logstash就可以接收到交换机日志了