Windows系统日志收集

方案

采用nxlog+fluentd+elasticsearch组件。nxlog 是用 C 语言写的一个跨平台日志收集处理软件，其内部支持使用 Perl 正则和语法来进行数据结构化和逻辑判断操作。nxlog负责收集Windows系统日志，并将日志传给fluentd，fluentd将这些日志信息抛给elasticsearch。fluentd官方文档Collecting Log Data from Windows中也是推荐使用该方案的。

预先准备

1. Windows服务器（本次使用的是Win10）。
2. Linux服务器（本次使用的CentOS7）。
3. 下载elasticsearch并安装在Linux服务器上。

环境搭建步骤

Windows上安装nxlog

1. 下载Windows版本的nxlog。
2. 直接点击next安装即可。

补充：nxlog默认是安装在C:\Program Files (x86)\nxlog目录中。

Linux上安装fluentd

1. 若从rpm库安装，执行如下命令：

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-  agent2.sh | sh

1. 常用命令：

$ /etc/init.d/td-agent start
$ /etc/init.d/td-agent stop
$ /etc/init.d/td-agent restart
$ /etc/init.d/td-agent status

测试

修改配置文件

1. 修改td-agent配置文件/etc/td-agent/td-agent.conf

   $ vim /etc/td-agent/td-agent.conf
   

   增添如下配置并保存：

   
      @type tcp
      format json
      port 5140
      tag windowslog
    
    
      type elasticsearch
    
   

2. 修改nxlog配置文件C:\Program Files (x86)\nxlog\conf
   改后的文件如下所示：

   define ROOT C:\Program Files (x86)\nxlog
   
   Moduledir %ROOT%\modules
   CacheDir %ROOT%\data
   Pidfile %ROOT%\data\nxlog.pid
   SpoolDir %ROOT%\data
   LogFile %ROOT%\data\nxlog.log
   
   
       Module      xm_syslog
   
   
   
       Module xm_json
   
   #日志输入端
   
       Module im_msvistalog
    #For windows 2003 and earlier use the following:
    #Module      im_mseventlog
       Exec to_json(); 
   
   #日志输出端
   
       Module om_tcp
       Host 192.168.159.133    #安装有td-agent的服务器地址
       Port 5140               #此端口要与td-agent监听端口一致
    
   
       Module om_file
       File   "D:\logs\log"    #为了便于观察，将收集到的系统日志存放在该位置下
   
   #绑定某输入到具体某输出
   
       Path in => out
   
   
       Path in => outfile
   
   

启动服务

1. 启动td-agent：

   $ /etc/init.d/td-agent start
   

2. 启动nxlog，使用powershell进入到C:\Program Files (x86)\nxlog目录下，执行命令：

   C:\Program Files (x86)\nxlog> .\\nxlog.exe -f -c ./conf/nxlog.conf
   

   部分输出如下：

   2017-07-13 14:43:30 INFO connecting to 192.168.159.133:5140
   2017-07-13 14:43:30 INFO nxlog-ce-2.9.1716 started
   

结果

在Linux控制台查看elasticsearch中的索引：

[root@localhost ~]# curl 'localhost:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   fluentd             c05Ykvp_RPebnOppfsa1_w   5   1          2            0     67.7kb         67.7kb
yellow open   .kibana             Zldk9n-BSweLtaeVNMsNtg   1   1          8            0       54kb           54kb

查看fluentd索引中的数据，部分输出如下：

[root@localhost ~]# curl -XGET 'localhost:9200/fluentd/_search?pretty'
{
  "took" : 363,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 68,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "fluentd",
        "_type" : "fluentd",
        "_id" : "AV06sthgdjHN6maqugXM",
        "_score" : 1.0,
        "_source" : {
          "EventTime" : "2017-07-13 14:45:59",
          "Hostname" : "LAPTOP-AQ1VQEMH",
          "Keywords" : 576460752303423488,
          "EventType" : "INFO",
          "SeverityValue" : 2,
          "Severity" : "INFO",
          "EventID" : 20,
          "SourceName" : "Microsoft-Windows-Audio",
          "ProviderGuid" : "{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}",
          "Version" : 0,
          "Task" : 116,
          "OpcodeValue" : 0,
          "RecordNumber" : 113490,
          "ProcessID" : 6804,
          "ThreadID" : 13988,
          "Channel" : "Microsoft-Windows-Audio/PlaybackManager",
          "Domain" : "LAPTOP-AQ1VQEMH",
          "AccountName" : "WHL",
          "UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
          "AccountType" : "User",
          "Message" : "格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false",
          "Opcode" : "信息",
          "Format" : "1",
          "SamplingRate" : "22050",
          "bAudioSrvStreamResourceType" : "false",
          "EventReceivedTime" : "2017-07-13 14:46:00",
          "SourceModuleName" : "in",
          "SourceModuleType" : "im_msvistalog"
        }
      },
      {
        "_index" : "fluentd",
        "_type" : "fluentd",
        "_id" : "AV06uizFdjHN6maqugXh",
        "_score" : 1.0,
        "_source" : {
          "EventTime" : "2017-07-13 14:53:26",
          "Hostname" : "LAPTOP-AQ1VQEMH",
          "Keywords" : 576460752303423488,
          "EventType" : "INFO",
          "SeverityValue" : 2,
          "Severity" : "INFO",
          "EventID" : 505,
          "SourceName" : "Microsoft-Windows-Application-Experience",
          "ProviderGuid" : "{EEF54E71-0661-422D-9A98-82FD4940B820}",
          "Version" : 0,
          "Task" : 0,
          "OpcodeValue" : 0,
          "RecordNumber" : 31916,
          "ProcessID" : 14276,
          "ThreadID" : 13872,
          "Channel" : "Microsoft-Windows-Application-Experience/Program-Telemetry",
          "Domain" : "LAPTOP-AQ1VQEMH",
          "AccountName" : "WHL",
          "UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
          "AccountType" : "User",
          "Message" : "兼容性修复程序已应用于 C:\\Users\\WHL\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe。\r\n修补程序信息: Google Chrome，{18A8B5B2-9D2F-4DB2-8307-196B5CC0CE6B}，0x80010101。",
          "Opcode" : "信息",
          "EventReceivedTime" : "2017-07-13 14:53:28",
          "SourceModuleName" : "in",
          "SourceModuleType" : "im_msvistalog"
        }
      },

打开D:\logs下的log文件，部分输出如下：

{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11792,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n           Volume Id: {4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}\r\n           Volume name: C:\r\n           Is boot volume: true\r\n\r\n           Interval duration: 3627921184 us\r\n\r\n           Non-cached reads:\r\n                     IO count: 7387\r\n                     Total bytes: 466456052\r\n                     Avg latency: 8210412 ns\r\n\r\n           Non-cached writes: \r\n                     IO count: 24179\r\n                     Total bytes: 412844919\r\n                     Avg latency: 3188872 ns\r\n\r\n           File flushes: \r\n                     IO count: 1622\r\n                     Avg latency: 17203859 ns\r\n\r\n           Volume flushes: \r\n                     IO count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           File level trims: \r\n                     IO count: 1\r\n                     Total bytes: 147374080\r\n                     Extents count: 1\r\n                     Avg latency: 1947000 ns\r\n\r\n           Volume trims: \r\n                     IO count: 324\r\n                     Total bytes: 1420537856\r\n                     Extents count: 1548\r\n                     Avg latency: 1463830 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}","VolumeNameLength":"2","VolumeName":"C:","IsBootVolume":"true","HighIoLatencyCount":"0","IntervalDurationUs":"3627921184","NCReadIOCount":"7387","NCReadTotalBytes":"466456052","NCReadAvgLatencyNs":"8210412","NCWriteIOCount":"24179","NCWriteTotalBytes":"412844919","NCWriteAvgLatencyNs":"3188872","FileFlushCount":"1622","FileFlushAvgLatencyNs":"17203859","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"1","FileLevelTrimTotalBytes":"147374080","FileLevelTrimExtentsCount":"1","FileLevelTrimAvgLatencyNs":"1947000","VolumeTrimCount":"324","VolumeTrimTotalBytes":"1420537856","VolumeTrimExtentsCount":"1548","VolumeTrimAvgLatencyNs":"1463830","IoBucketsCount":"48","TotalBytesBucketsCount":"44","ExtentsBucketsCount":"4","IoCount":"35","TotalLatencyUs":"51080","TotalBytes":"1208320","TrimExtentsCount":"1537","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11793,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n           Volume Id: {EFA1B98E-1BD3-45F1-A956-882127C35622}\r\n           Volume name: D:\r\n           Is boot volume: false\r\n\r\n           Interval duration: 3627921030 us\r\n\r\n           Non-cached reads:\r\n                     IO count: 11248\r\n                     Total bytes: 820325912\r\n                     Avg latency: 4644197 ns\r\n\r\n           Non-cached writes: \r\n                     IO count: 7992\r\n                     Total bytes: 242775864\r\n                     Avg latency: 15979689 ns\r\n\r\n           File flushes: \r\n                     IO count: 49\r\n                     Avg latency: 10135571 ns\r\n\r\n           Volume flushes: \r\n                     IO count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           File level trims: \r\n                     IO count: 0\r\n                     Total bytes: 0\r\n                     Extents count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           Volume trims: \r\n                     IO count: 2\r\n                     Total bytes: 16384\r\n                     Extents count: 4\r\n                     Avg latency: 893000 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{EFA1B98E-1BD3-45F1-A956-882127C35622}","VolumeNameLength":"2","VolumeName":"D:","IsBootVolume":"false","HighIoLatencyCount":"0","IntervalDurationUs":"3627921030","NCReadIOCount":"11248","NCReadTotalBytes":"820325912","NCReadAvgLatencyNs":"4644197","NCWriteIOCount":"7992","NCWriteTotalBytes":"242775864","NCWriteAvgLatencyNs":"15979689","FileFlushCount":"49","FileFlushAvgLatencyNs":"10135571","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"0","FileLevelTrimTotalBytes":"0","FileLevelTrimExtentsCount":"0","FileLevelTrimAvgLatencyNs":"0","VolumeTrimCount":"2","VolumeTrimTotalBytes":"16384","VolumeTrimExtentsCount":"4","VolumeTrimAvgLatencyNs":"893000","IoBucketsCount":"35","TotalBytesBucketsCount":"32","ExtentsBucketsCount":"1","IoCount":"2902","TotalLatencyUs":"1952578","TotalBytes":"41503232","TrimExtentsCount":"4","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:52","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113487,"ProcessID":6804,"ThreadID":13716,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:45:54","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113488,"ProcessID":6804,"ThreadID":12172,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113489,"ProcessID":6804,"ThreadID":9620,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113490,"ProcessID":6804,"ThreadID":13988,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113491,"ProcessID":6804,"ThreadID":11340,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}