rsyslog omkafka集群配置 后感

        这几天为了rsyslog日志检测搞得痛苦难言，将文档中的内容导入kafka中，听起来简单，过程却是十分复杂，走的弯路多，所以就把这几天的辛苦历程记录下来，给需要的同学们。

        kafka+zookeeper集群的配置我就不想多说了，配通知后自己新建topic，启动producter输入数据查看consumer有没有数据到处即可；

        具体的kafka那几条命令：

        创建topic：./bin/kafka-topics.sh --create --topic test --replication-factor 1 --partitions 32  --zookeeper  localhost:2181
        删除topic：./bin/kafka-topics.sh --delete --topic test --zookeeper localhost:2181
        查看topic列表：./bin/kafka-topics.sh  --list  --zookeeper localhost:2181
        查看某个topic详细：./bin/kafka-topics.sh --describe --topic test --zookeeper localhosts:2181
        监控某个topic的消费：./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
        指定消费组查看消费情况：/usr/local/kafka/kafka_2.11-0.8.2.1/bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181  --group test

        查看数据：/usr/local/kafka/kafka_2.11-0.8.2.1/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --from-beginning --topic test（这条命令打开后，只要rsyslog有数据写入，会自动显示导入的内容）

   注意：--zookeeper后面的内容一定是主机名+端口号（/etc/hosts里面有主机名和IP的映射），集群方式的zookeeper安装形式逗号隔开：

  例如：

    ./kafka-topics --zookeeper LetvHM2PXD2:2181,LetvHM1VXD2:2181,LetvHM1PXD2:2181 -alter --partitions 10 --topic AppAd （修改partitions命令）

   我们现在只讲rsyslo的配置：

   楼主这几天仔细看了rsyslog的官网文档：http://www.rsyslog.com/doc/v8-stable/configuration/index.html（官网网址）确实有点多，牵扯到syslog方面的东西，没有基础的看起来有点费劲，我们姑且不管，具体的语法，可以自己慢慢看，我们先搭起来基本的配置，能够将数据导入kafka作为我们的基本任务，废话不多说，我们进入正题：

  rsyslog安装：

我用的系统是Centos6.6，yum安装的方式，所用的yum源：

[rsyslog_v8]
name=Adiscon CentOS-$releasever - local packages for $basearch
baseurl=http://rpms.adiscon.com/v8-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon
protect=1

导入之后，运行以下命令安装rsyslog和rsyslog-kafka：

 【命令】yum install -y rsyslog rsyslog-kafka   
 【命令】rpm -ql rsyslog-kafka
                 /lib64/rsyslog/omkafka.so   #rsyslog  连接  kafka的插件
  【命令】 rsyslogd -N1  #检测rsyslog配置文件是否对错的命令（重要）    

切记：每次配置rsyslog配置文档时一定要重启rsyslogrestart（重要）

我的配置文档放在/etc/rsyslog.d/kafka.conf中，内容如图所示：

################### egrep -v "^$|^#" /etc/rsyslog.d/kafka.conf
module(load="imfile"  PollingInterval="1")
module(load="omkafka")
input(type="imfile"
      File="/data/ark/tracking"
      Tag="tracking"
      PersistStateInterval="1"
      Facility="*"
      Severity="*"
      ruleset="tracking"
)
$template tracking_template, "%msg%\n"
ruleset(name="tracking"){
action (type="omkafka"
        topic="rsyslog"
        broker="HMHRXD2:9092,HM9LXD2:9092,HMJSXD2:9092,HMMQXD2:9092,HMKLXD2:9092,HMPPXD2:9092,HMBVXD2:9092,HMKPXD2:9092,HMNNXD2:9092"
        template="tracking_template"
        partitions.auto="on"
         confParam=["socket.keepalive.enable=true"]
        )
}

楼主检测的是/data/ark下的tracking文件，只要tracking中有内容追加，kafka中就会有数据写入。

主机名换成自己集群kafka主机的主机名；

切记：

broker中的书写格式一定要是主机名：端口号，楼主就是在这不出了问题直接用IP：端口号的方式栽了大跟头，就在这一步。

另附/etc/rsysog.conf的内容：

# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced 
# things, RainerScript configuration is suggested.


# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html


#### MODULES ####


#module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
#module(load="imklog")   # provides kernel logging support (previously done by rklogd)
#module(load"immark")  # provides --MARK-- message capability


# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="1514")
#module(load="imfile")
#module(load="omhiredis") # lets you send to Redis
#module(load="omkafka")   # lets you send to Kafka
#action(type="omkafka" topic="test_nginx" broker="localhost:9092")
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")
#action(type="omkafka" topic="test_nginx" broker="localhost:9092")


#### GLOBAL DIRECTIVES ####


# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on


# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf




#### RULES ####


# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console


# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages


# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure


# Log all the mail messages in one place.
mail.*                                                  /var/log/maillog




# Log cron stuff
cron.*                                                  /var/log/cron


# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*


# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler


# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log




# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

以上内容如果小白的话可以直接复制粘贴，楼主亲测，成功。