fluent将日志采集到opentsdb的笔记（自定义插件）

GitHub上有唯一一个实现好了的插件 fluent-plugin-opentsdb

https://github.com/emurphy/fluent-plugin-opentsdb

经过一天的研究，没使用起来，或者说没有得到我们想要的数据，只有打算自己写一个插件了

官网也有如何自定义插件的方法：http://docs.fluentd.org/v0.12/articles/plugin-development#writing-buffered-output-plugins

由于使用的是 td-agent 所以插件放置路径改为：.../td-agent/plugin

config配置文件如下：

  # in_tail 输入插件允许fluentd从文本文件的尾部读事件。它的行为类似于tail-f 命令。
  @type tail
  #read_from_head true
  format json
  path /test/logs/nginx-logs/test-*.log
  pos_file /var/log/td-agent/nginx.log.pos
  tag es.nginx



    @type opentsdb1
    host 132.12.1.23
    port 4242

使用tail 输入插件，日志文件改动以后，将新增加的数据反馈到自己实现的输出插件

将官网的输出插件拷贝过来，填充自己的内容

require 'fluent/output'  #添加基础模块和自己需要用到的模块

module Fluent
  class SomeOutput < BufferedOutput    #自己写类继承输出模块，我是继承自 Output
    # First, register the plugin. NAME is the name of this plugin
    # and identifies the plugin in the configuration file.
    Fluent::Plugin.register_output('NAME', self)   # 写入自己的插件名

    # config_param defines a parameter. You can refer a parameter via @path instance variable
    # Without :default, a parameter is required.
    config_param :path, :string  定义一些自己需要的输出插件参数

    # This method is called before starting.
    # 'conf' is a Hash that includes configuration parameters.
    # If the configuration is invalid, raise Fluent::ConfigError.
    def configure(conf)  #启动前自动调用
      super

      # You can also refer raw parameter via conf[name].
      @path = conf['path']  #将配置文件的参数获取出来
      ...
    end

    # This method is called when starting.
    # Open sockets or files here.
    def start  #启动时调用，可写一些自己需要的实现在这里，笔者是这里写入测试与opentsdb的连接状态
      super
      ...
    end

    # This method is called when shutting down.
    # Shutdown the thread and close sockets or files here.
    def shutdown #关闭是调用，笔者在这里写的关闭所有socket连接
      super
      ...
    end

    # This method is called when an event reaches to Fluentd.
    # Convert the event to a raw string.
    def format(tag, time, record) #事件触发函数，将事件转换成原始字符串，笔者没有使用这个，用 emit（）代替
      [tag, time, record].to_json + "\n"
      ## Alternatively, use msgpack to serialize the object.
      # [tag, time, record].to_msgpack
    end
    
    def emit(tag, es, chain)
    es.each do |time,record|
        取出事件的事件和记录
        record 为hash数据，利用tail输入插件，这里的record记录了日志新增的内容
        通过自己定义函数解析这里的内容，获取opentsdb需要的数据
        笔者是用put 方法向 opentsdb写入数据
        @socket = TCPSocket.new(@host, @port)
     @socket.puts(message)
     @socket.close
        message 格式
        message = ['put', name, time, value, tag].join(' ')
        通过解析上面的 record 获取或组装成 message 里面的 name，time，value，tag
        到这里，一条日志数据就用put方式写入opentsdb了，如果解析比较繁琐，也能自定义解析函数调用，使代码更清晰
      end
    # This method is called every flush interval. Write the buffer chunk
    # to files or databases here.
    # 'chunk' is a buffer chunk that includes multiple formatted
    # events. You can use 'data = chunk.read' to get all events and
    # 'chunk.open {|io| ... }' to get IO objects.
    #
    # NOTE! This method is called by internal thread, not Fluentd's main thread. So IO wait doesn't affect other plugins.
    def write(chunk) #未使用
      data = chunk.read
      print data
    end

    ## Optionally, you can use chunk.msgpack_each to deserialize objects.
    #def write(chunk) #未使用
    #  chunk.msgpack_each {|(tag,time,record)|
    #  }
    #end
  end
end

笔者自己写好的一个输出到opentsdb的插件 ：http://download.csdn.net/detail/qq_21398167/9858721