weixin_34293246

Flume自定义拦截器（Interceptors）或自带拦截器时的一些经验技巧总结（图文详解）...

　　不多说，直接上干货！

一、自定义拦截器类型必须是：类全名$内部类名，其实就是内部类名称
　　如：zhouls.bigdata.MySearchAndReplaceInterceptor$Builder

二、为什么这样写
　　至于为什么这样写：是因为Interceptor接口还有一个公共的内部接口（Builder），所以自定义拦截器要是实现 Builder接口，
　　也就是实现一个内部类（该内部类的主要作用是：获取flume-conf.properties 自定义的参数，并将参数传递给自定义拦截器）
三、
　　本人知识有限,可能描述的不太清楚，可自行了解 java接口与内部类

　　由于有时候内置的拦截器不够用，所以需要针对特殊的业务需求自定义拦截器。
官方文档中没有发现自定义interceptor的步骤，但是可以根据flume源码参考内置的拦截器的代码
flume-1.7/flume-ng-core/src/main/java/org/apache/flume/interceptor/***Iterceptor.java

　　无论，是flume的自带拦截器，还是，flume的自定义拦截器，我这篇博文呢，是想给大家，去规范和方便化！！！

[hadoop@master app]$ rm -rf flume
[hadoop@master app]$ ln -s flume-1.7.0/ flume
[hadoop@master app]$ ll
lrwxrwxrwx   1 hadoop hadoop   12 Jul 27 11:42 flume -> flume-1.7.0/
drwxrwxr-x   7 hadoop hadoop 4096 Apr 20 12:17 flume-1.6.0
drwxrwxr-x   7 hadoop hadoop 4096 Apr 20 12:00 flume-1.7.0

　　Host Interceptor的应用场景是，将同一主机或服务器上的数据flume在一起。

　　 Regex Extractor Iterceptor的应用场景是，

这里，教大家一个非常实用的技巧，

[hadoop@master flume-1.7.0]$ pwd
/home/hadoop/app/flume-1.7.0
[hadoop@master flume-1.7.0]$ ll
total 148
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$ cp -r conf conf_HostInterceptor
[hadoop@master flume-1.7.0]$ ll
total 152
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 11:59 conf_HostInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$

[hadoop@master flume-1.7.0]$ ll
total 152
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:01 conf_HostInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$ cp -r conf conf_RegexExtractorInterceptor
[hadoop@master flume-1.7.0]$ ll
total 156
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:01 conf_HostInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:03 conf_RegexExtractorInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$

drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:01 conf_HostInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:03 conf_RegexExtractorInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$ cp -r conf conf_SearchandReplaceInterceptor
[hadoop@master flume-1.7.0]$ ll
total 160
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:01 conf_HostInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:03 conf_RegexExtractorInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 12:04 conf_SearchandReplaceInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$

　　大家，想必，很想问，为什么要这么cp复制出来呢？如flume的以下3种重要的自带拦截器？？？

cp -r conf conf_HostInterceptor
cp -r conf conf_SearchandReplaceInterceptor
cp -r conf conf_RegexExtractorInterceptor

　　你想啊，若不复制的话，则我们在使用时，则会不方便管理。尤其是，见如下，共用同一个log4j.properties，日志排查起来一点都不方便！！！

　　而，现在是

　　这样做下来，就是非常的方便和正规。

　　同时，大家，还要如下更改下

[hadoop@master conf_HostInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_HostInterceptor
[hadoop@master conf_HostInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 12:01 flume-conf.properties.template
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 12:01 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 12:01 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3107 Jul 27 12:01 log4j.properties
[hadoop@master conf_HostInterceptor]$ mv flume-conf.properties.template flume-conf.properties
[hadoop@master conf_HostInterceptor]$ vim log4j.properties

#flume.root.logger=DEBUG,console
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume_HostInterceptor.log

　　同理

[hadoop@master conf_RegexExtractorInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_RegexExtractorInterceptor
[hadoop@master conf_RegexExtractorInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 12:03 flume-conf.properties.template
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 12:03 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 12:03 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3107 Jul 27 12:03 log4j.properties
[hadoop@master conf_RegexExtractorInterceptor]$ mv flume-conf.properties.template flume-conf.properties
[hadoop@master conf_RegexExtractorInterceptor]$ vim log4j.properties

#flume.root.logger=DEBUG,console
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume_RegexExtractorInterceptor.log

　　同理

[hadoop@master conf_SearchandReplaceInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_SearchandReplaceInterceptor
[hadoop@master conf_SearchandReplaceInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 12:04 flume-conf.properties.template
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 12:04 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 12:04 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3107 Jul 27 12:04 log4j.properties
[hadoop@master conf_SearchandReplaceInterceptor]$ mv flume-conf.properties.template flume-conf.properties
[hadoop@master conf_SearchandReplaceInterceptor]$ vim log4j.properties

#flume.root.logger=DEBUG,console
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume_SearchandReplaceInterceptor.log

Host Interceptor

　　conf_HostInterceptor的flume-conf.properties

agent1.sources = r1
agent1.sinks = k1
agent1.channels = c1

# Describe/configure the source
agent1.sources.r1.type = netcat
agent1.sources.r1.bind = localhost
agent1.sources.r1.port = 44444



agent1.sources.r1.interceptors = i1
agent1.sources.r1.interceptors.i1.type = host
agent1.sources.r1.interceptors.i1.hostHeader = hostname


# Use a channel which buffers events in memory
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1
agent1.channels.c1.transactionCapacity = 1

# Bind the source and sink to the channel
agent1.sources.r1.channels = c1
agent1.sinks.k1.channel = c1

# Describe the sink
agent1.sinks.k1.type = logger

则，注意，启动命令也要发生变化

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_HostInterceptor/  --conf-file conf_HostInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

SLF4J: Found binding in [jar:file:/home/hadoop/app/hbase-0.98.19/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hive-1.0.0/lib/hive-jdbc-1.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017-07-27 12:41:49,451 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2017-07-27 12:41:50,137 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:conf_HostInterceptor/flume-conf.properties
2017-07-27 12:41:50,188 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2017-07-27 12:41:50,189 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1 2017-07-27 12:41:50,189 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: agent1 2017-07-27 12:41:50,280 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent1] 2017-07-27 12:41:50,280 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels 2017-07-27 12:41:50,337 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory 2017-07-27 12:41:50,423 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel c1 2017-07-27 12:41:50,425 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat 2017-07-27 12:41:51,478 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger 2017-07-27 12:41:51,490 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel c1 connected to [r1, k1] 2017-07-27 12:41:52,050 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@13f948e counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} } 2017-07-27 12:41:52,052 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1 2017-07-27 12:41:53,484 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean. 2017-07-27 12:41:53,517 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started 2017-07-27 12:41:53,522 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1 2017-07-27 12:41:53,524 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1 2017-07-27 12:41:53,531 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting 2017-07-27 12:41:54,384 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:169)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

　　等待数据的采集

[hadoop@master ~]$ yum -y install telnet
Loaded plugins: fastestmirror, refresh-packagekit, security
You need to be root to perform this command.
[hadoop@master ~]$ su root
Password: 
[root@master hadoop]# yum -y install telnet
Loaded plugins: fastestmirror, refresh-packagekit, security Loading mirror speeds from cached hostfile * base: mirrors.cqu.edu.cn * extras: mirrors.sohu.com

　　成功地，然后，这边随便输入什么。比如hello

[root@master ~]# telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
OK

Event: { headers:{hostname=192.168.80.145} body: 68 65 6C 6C 6F 0D                               hello. }

　　这就是Host Interceptor的作用体现！

agent1.sources.r1.interceptors = i1
agent1.sources.r1.interceptors.i1.type = host
agent1.sources.r1.interceptors.i1.hostHeader = hostname

　　若想要如下的效果，则

Event: { headers:{hostname=master} body: 7A 68 6F 75 6C 73 0D                            zhouls. }

　　则

agent1.sources = r1
agent1.sinks = k1
agent1.channels = c1

# Describe/configure the source
agent1.sources.r1.type = netcat
agent1.sources.r1.bind = localhost
agent1.sources.r1.port = 44444



agent1.sources.r1.interceptors = i1
agent1.sources.r1.interceptors.i1.type = host
agent1.sources.r1.interceptors.i1.useIP = false
agent1.sources.r1.interceptors.i1.hostHeader = hostname


# Use a channel which buffers events in memory
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1
agent1.channels.c1.transactionCapacity = 1

# Bind the source and sink to the channel
agent1.sources.r1.channels = c1
agent1.sinks.k1.channel = c1

# Describe the sink
agent1.sinks.k1.type = logger

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_HostInterceptor/  --conf-file conf_HostInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

[root@master ~]# telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
zhouls
OK

Event: { headers:{hostname=master} body: 7A 68 6F 75 6C 73 0D                            zhouls. }

Regex Extractor Interceptor（正则抽取拦截器）

　　conf_RegexExtractorInterceptor的flume-conf.properties

[hadoop@master conf_RegexExtractorInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_RegexExtractorInterceptor
[hadoop@master conf_RegexExtractorInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 12:03 flume-conf.properties
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 12:03 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 12:03 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3133 Jul 27 12:31 log4j.properties
[hadoop@master conf_RegexExtractorInterceptor]$ vim flume-conf.properties

　　首先，我们来说说这个拦截器的应用场景　　

　　假设，有如下的flume测试数据

video_info

{"id":"14943445328940974601","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974602","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974603","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974604","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974605","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974606","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974607","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974608","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974609","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}
{"id":"14943445328940974610","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hots":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","replay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}



userinfo

{"uid":"861848974414839801","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839802","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839803","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839804","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839805","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839806","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839807","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839808","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839809","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}
{"uid":"861848974414839810","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_face":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"userinfo"}




gift_record

{"send_id":"834688818270961664","good_id":"223","video_id":"14943443045138661356","gold":"10","timestamp":1494344574,"type":"gift_record"}
{"send_id":"829622867955417088","good_id":"72","video_id":"14943429572096925829","gold":"4","timestamp":1494344574,"type":"gift_record"}
{"send_id":"827187230564286464","good_id":"193","video_id":"14943394752706070833","gold":"6","timestamp":1494344574,"type":"gift_record"}
{"send_id":"829622867955417088","good_id":"80","video_id":"14943429572096925829","gold":"6","timestamp":1494344574,"type":"gift_record"}
{"send_id":"799051982152663040","good_id":"72","video_id":"14943435528719800690","gold":"4","timestamp":1494344574,"type":"gift_record"}
{"send_id":"848799149716930560","good_id":"72","video_id":"14943435528719800690","gold":"4","timestamp":1494344574,"type":"gift_record"}
{"send_id":"775251729037262848","good_id":"777","video_id":"14943390379833490630","gold":"5","timestamp":1494344574,"type":"gift_record"}
{"send_id":"835670464000425984","good_id":"238","video_id":"14943428496217015696","gold":"2","timestamp":1494344574,"type":"gift_record"}
{"send_id":"834688818270961664","good_id":"223","video_id":"14943443045138661356","gold":"10","timestamp":1494344574,"type":"gift_record"}
{"send_id":"834688818270961664","good_id":"223","video_id":"14943443045138661356","gold":"10","timestamp":1494344574,"type":"gift_record"}

　　以上是flume采集后的数据。假设都是在这个flume测试数据.txt里，现在呢，我想按照type来存放到不同的目录下。

　　即video_info的存放到video_info目录下、userinfo的存放到userinfo目录下、gift_record的存放到gift_record目录下。

　　则，这样的应用场景，即根据数据里内容的type字段的值的不同，来分别存储。则需要Regex Extractor Interceptor派上用场了。

　　　怎么做呢，其实很简单，把type的值，放到

# 定义拦截器
agent1.sources.r1.interceptors = i1
# 设置拦截器类型
agent1.sources.r1.interceptors.i1.type = regex_extractor
# 设置正则表达式，匹配指定的数据，这样设置会在数据的header中增加log_type=”对应的值”
agent1.sources.r1.interceptors.i1.regex = "type":"(\\w+)"
agent1.sources.r1.interceptors.i1.serializers = s1
agent1.sources.r1.interceptors.i1.serializers.s1.name = log_type

　　为什么是这么来写？

agent1.sources.r1.interceptors.i1.regex = "type":"(\\w+)"

　　是因为数据的内容决定的。

"type":"video_info"


"type":"userinfo"


"type":"gift_record"

#source的名字
agent1.sources = fileSource
# channels的名字，建议按照type来命名
agent1.channels = memoryChannel
# sink的名字，建议按照目标来命名
agent1.sinks = hdfsSink

# 指定source使用的channel名字
agent1.sources.fileSource.channels = memoryChannel
# 指定sink需要使用的channel的名字,注意这里是channel
agent1.sinks.hdfsSink.channel = memoryChannel


agent1.sources.fileSource.type = exec
agent1.sources.fileSource.command = tail -F /usr/local/log/server.log



#------- fileChannel-1相关配置-------------------------
# channel类型

agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 1000
agent1.channels.memoryChannel.transactionCapacity = 1000
agent1.channels.memoryChannel.byteCapacityBufferPercentage = 20
agent1.channels.memoryChannel.byteCapacity = 800000


#---------拦截器相关配置------------------
# 定义拦截器
agent1.sources.fileSource.interceptors = i1
# 设置拦截器类型
agent1.sources.fileSource.interceptors.i1.type = regex_extractor
# 设置正则表达式，匹配指定的数据，这样设置会在数据的header中增加log_type="某个值"
agent1.sources.fileSource.interceptors.i1.regex = "type":"(\\w+)"
agent1.sources.fileSource.interceptors.i1.serializers = s1
agent1.sources.fileSource.interceptors.i1.serializers.s1.name = log_type




#---------hdfsSink 相关配置------------------
agent1.sinks.hdfsSink.type = hdfs
# 注意, 我们输出到下面一个子文件夹datax中
agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type}
agent1.sinks.hdfsSink.hdfs.writeFormat = Text
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.callTimeout = 3600000
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true

#当文件大小为52428800字节时，将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollSize = 52428800
#events数据达到该数量的时候，将临时文件滚动成目标文件
agent1.sinks.hdfsSink.hdfs.rollCount = 0
#每隔N s将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollInterval = 1200

#配置前缀和后缀
agent1.sinks.hdfsSink.hdfs.filePrefix=run
agent1.sinks.hdfsSink.hdfs.fileSuffix=.data

　　监控文件是在

/usr/local/log/server.log

[root@master local]# pwd
/usr/local
[root@master local]# ll
total 40
drwxr-xr-x. 2 root root 4096 Sep 23  2011 bin
drwxr-xr-x. 2 root root 4096 Sep 23  2011 etc
drwxr-xr-x. 2 root root 4096 Sep 23  2011 games
drwxr-xr-x. 2 root root 4096 May  1 19:40 include
drwxr-xr-x. 2 root root 4096 May  1 19:40 lib
drwxr-xr-x. 2 root root 4096 Sep 23  2011 lib64
drwxr-xr-x. 2 root root 4096 Sep 23  2011 libexec
drwxr-xr-x. 2 root root 4096 Sep 23  2011 sbin
drwxr-xr-x. 6 root root 4096 May  1 19:40 share
drwxr-xr-x. 2 root root 4096 Sep 23  2011 src
[root@master local]# mkdir log
[root@master local]# cd log
[root@master log]# pwd
/usr/local/log
[root@master log]# ll
total 0
[root@master log]#

　　然后，执行

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_RegexExtractorInterceptor/  --conf-file conf_RegexExtractorInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

　　然后，我这边，采用如下的一个shell脚本来模拟产生测试数据。

　　producerLog.sh

[root@master log]# pwd
/usr/local/log
[root@master log]# ll
total 0
[root@master log]# vim producerLog.sh

#!/bin/bash
log1='{"id":"14943445328940974610","uid":"840717325115457536","lat":"53.530598","lnt":"-2.5620373","hot
s":0,"title":"0","status":"1","topicId":"0","end_time":"1494344570","watch_num":0,"share_num":"1","repl
ay_url":null,"replay_num":0,"start_time":"1494344544","timestamp":1494344571,"type":"video_info"}'

log2='{"uid":"861848974414839810","nickname":"mick","usign":"","sex":1,"birthday":"","face":"","big_fac
e":"","email":"[email protected]","mobile":"","reg_type":"102","last_login_time":"1494344580","reg_time":"1494
344580","last_update_time":"1494344580","status":"5","is_verified":"0","verified_info":"","is_seller":"
0","level":1,"exp":0,"anchor_level":0,"anchor_exp":0,"os":"android","timestamp":1494344580,"type":"user_info"}'


log3='{"send_id":"834688818270961664","good_id":"223","video_id":"14943443045138661356","gold":"10","ti
mestamp":1494344574,"type":"gift_record"}'


declare -i count

count=0
while [ 'a' = 'a' ]
do
echo -e $log1 >> /usr/local/log/server.log
echo -e $log2 >> /usr/local/log/server.log
echo -e $log3 >> /usr/local/log/server.log
count+=1
    if [ ${count} -eq 500 ]
    then
    count=0
    echo "sleep..."
    sleep 3
    fi
done

　　这个shell脚本不太难哈。即log1会生成500条、log2会生成500条、log3会生成500条。每隔3秒。

　　然后，再来创建server.log文件

[root@master log]# pwd
/usr/local/log
[root@master log]# ll
total 4
-rw-r--r-- 1 root root 1157 Jul 27 14:39 producerLog.sh
[root@master log]# vim producerLog.sh 
[root@master log]# touch server.log
[root@master log]# ll
total 4
-rw-r--r-- 1 root root 1157 Jul 27 14:42 producerLog.sh
-rw-r--r-- 1 root root    0 Jul 27 14:43 server.log
[root@master log]# cat server.log 
[root@master log]#

　　然后，来执行这个脚本，以模拟产生数据。

[root@master log]# pwd
/usr/local/log
[root@master log]# ll
total 4
-rw-r--r-- 1 root root 1157 Jul 27 14:42 producerLog.sh
-rw-r--r-- 1 root root    0 Jul 27 14:43 server.log
[root@master log]# chmod 755 producerLog.sh 
[root@master log]# ll
total 4
-rwxr-xr-x 1 root root 1157 Jul 27 14:42 producerLog.sh
-rw-r--r-- 1 root root    0 Jul 27 14:43 server.log
[root@master log]# ./producerLog.sh

2017-07-27 14:46:42,275 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:521)] Block Under-replication detected. Rotating file.
2017-07-27 14:46:42,279 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:357)] Closing hdfs://master:9000/data/types/20170727//run.1501137914366.data.tmp
2017-07-27 14:46:43,117 (hdfs-hdfsSink-call-runner-9) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:618)] Renaming hdfs://master:9000/data/types/20170727/run.1501137914366.data.tmp to hdfs://master:9000/data/types/20170727/run.1501137914366.data
2017-07-27 14:46:43,429 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170727//run.1501137914367.data.tmp
2017-07-27 14:46:45,017 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:521)] Block Under-replication detected. Rotating file.
2017-07-27 14:46:45,017 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:357)] Closing hdfs://master:9000/data/types/20170727/video_info/run.1501137883920.data.tmp
2017-07-27 14:46:45,091 (hdfs-hdfsSink-call-runner-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:618)] Renaming hdfs://master:9000/data/types/20170727/video_info/run.1501137883920.data.tmp to hdfs://master:9000/data/types/20170727/video_info/run.1501137883920.data
2017-07-27 14:46:45,236 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170727/video_info/run.1501137883921.data.tmp
2017-07-27 14:46:45,412 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:521)] Block Under-replication detected. Rotating file.
2017-07-27 14:46:45,412 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:357)] Closing hdfs://master:9000/data/types/20170727//run.1501137914367.data.tmp
2017-07-27 14:46:45,455 (hdfs-hdfsSink-call-runner-7) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:618)] Renaming hdfs://master:9000/data/types/20170727/run.1501137914367.data.tmp to hdfs://master:9000/data/types/20170727/run.1501137914367.data
2017-07-27 14:46:45,585 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170727//run.1501137914368.data.tmp
2017-07-27 14:46:45,942 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:521)] Block Under-replication detected. Rotating file.
2017-07-27 14:46:45,942 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:357)] Closing hdfs://master:9000/data/types/20170727/gift_record/run.1501137916399.data.tmp
2017-07-27 14:46:46,074 (hdfs-hdfsSink-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:618)] Renaming hdfs://master:9000/data/types/20170727/gift_record/run.1501137916399.data.tmp to hdfs://master:9000/data/types/20170727/gift_record/run.1501137916399.data
2017-07-27 14:46:46,138 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170727/gift_record/run.1501137916400.data.tmp

Search and Replace Interceptor

　　以上存放，是在

　　模拟产生的gift_record是存放在 /data/types/20170727/gift_record

　　但是呢。我现在需求是

　　模拟产生的gift_record是存放在 /data/types/20170727/giftRecord

　　则改为

agent1.sources.r1.interceptors = i1 i2 i3 i4
agent1.sources.r1.interceptors.i1.type = search_replace
agent1.sources.r1.interceptors.i1.searchPattern = "type":"gift_record"
agent1.sources.r1.interceptors.i1.replaceString = "type":"giftRecord"


agent1.sources.r1.interceptors.i2.type = search_replace
agent1.sources.r1.interceptors.i2.searchPattern = "type":"video_info"
agent1.sources.r1.interceptors.i2.replaceString = "type":"videoInfo"


agent1.sources.r1.interceptors.i3.type = search_replace
agent1.sources.r1.interceptors.i3.searchPattern = "type":"user_info"
agent1.sources.r1.interceptors.i3.replaceString = "type":"userInfo"



agent1.sources.fileSource.interceptors.i4.type = regex_extractor
agent1.sources.fileSource.interceptors.i4.regex = "type":"(\\w+)"
agent1.sources.fileSource.interceptors.i4.serializers = s1
agent1.sources.fileSource.interceptors.i4.serializers.s1.name = log_type

[hadoop@master conf_SearchandReplaceInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_SearchandReplaceInterceptor
[hadoop@master conf_SearchandReplaceInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 12:04 flume-conf.properties
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 12:04 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 12:04 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3135 Jul 27 12:32 log4j.properties
[hadoop@master conf_SearchandReplaceInterceptor]$ vim flume-conf.properties

#source的名字
agent1.sources = fileSource
# channels的名字，建议按照type来命名
agent1.channels = memoryChannel
# sink的名字，建议按照目标来命名
agent1.sinks = hdfsSink

# 指定source使用的channel名字
agent1.sources.fileSource.channels = memoryChannel
# 指定sink需要使用的channel的名字,注意这里是channel
agent1.sinks.hdfsSink.channel = memoryChannel


agent1.sources.fileSource.type = exec
agent1.sources.fileSource.command = tail -F /usr/local/log/server.log



#------- fileChannel-1相关配置-------------------------
# channel类型

agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 1000
agent1.channels.memoryChannel.transactionCapacity = 1000
agent1.channels.memoryChannel.byteCapacityBufferPercentage = 20
agent1.channels.memoryChannel.byteCapacity = 800000


#---------拦截器相关配置------------------

agent1.sources.r1.interceptors = i1 i2 i3 i4
agent1.sources.r1.interceptors.i1.type = search_replace
agent1.sources.r1.interceptors.i1.searchPattern = "type":"gift_record"
agent1.sources.r1.interceptors.i1.replaceString = "type":"giftRecord"

agent1.sources.r1.interceptors.i2.type = search_replace
agent1.sources.r1.interceptors.i2.searchPattern = "type":"video_info"
agent1.sources.r1.interceptors.i2.replaceString = "type":"videoInfo"

agent1.sources.r1.interceptors.i3.type = search_replace
agent1.sources.r1.interceptors.i3.searchPattern = "type":"user_info"
agent1.sources.r1.interceptors.i3.replaceString = "type":"userInfo"


agent1.sources.fileSource.interceptors.i4.type = regex_extractor
agent1.sources.fileSource.interceptors.i4.regex = "type":"(\\w+)"
agent1.sources.fileSource.interceptors.i4.serializers = s1
agent1.sources.fileSource.interceptors.i4.serializers.s1.name = log_type




#---------hdfsSink 相关配置------------------
agent1.sinks.hdfsSink.type = hdfs
# 注意, 我们输出到下面一个子文件夹datax中
agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type}
agent1.sinks.hdfsSink.hdfs.writeFormat = Text
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.callTimeout = 3600000
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true

#当文件大小为52428800字节时，将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollSize = 52428800
#events数据达到该数量的时候，将临时文件滚动成目标文件
agent1.sinks.hdfsSink.hdfs.rollCount = 0
#每隔N s将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollInterval = 1200

#配置前缀和后缀
agent1.sinks.hdfsSink.hdfs.filePrefix=run
agent1.sinks.hdfsSink.hdfs.fileSuffix=.data

　　然后，执行

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_SearchandReplaceInterceptor/  --conf-file conf_SearchandReplaceInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

　　我这里，出现了这个错误

2017-07-29 10:17:51,006 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: fileSource started
2017-07-29 10:17:52,792 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)] Serializer = TEXT, UseRawLocalFileSystem = false
2017-07-29 10:17:55,094 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170729//run.1501294672792.data.tmp
2017-07-29 10:17:55,842 (hdfs-hdfsSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-07-29 10:18:00,495 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:352)] Failed while running command: tail -F /usr/local/log/server.log
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:127)
    at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
    at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
    at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:341)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:18:00,544 (timedFlushExecService21-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:327)] Exception occured when processing event batch
org.apache.flume.ChannelException: java.lang.InterruptedException
    at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
    at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
    at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:254)
    at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:323)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

　　然后，这边模拟产生数据。

[root@master log]# pwd
/usr/local/log
[root@master log]# ll
total 4
-rwxr-xr-x 1 root root 1157 Jul 29 10:01 producerLog.sh
-rw-r--r-- 1 root root    0 Jul 29 10:06 server.log
[root@master log]# ./producerLog.sh 
sleep...
sleep...
sleep...

Flume自定义拦截器（Interceptors）

一、自定义拦截器类型必须是：类全名$内部类名，其实就是内部类名称
　　如：zhouls.bigdata.MySearchAndReplaceInterceptor$Builder

　　由于有时候内置的拦截器不够用，所以需要针对特殊的业务需求自定义拦截器
　　官方文档中没有发现自定义interceptor的步骤，但是可以根据flume源码参考内置的拦截器的代码
　　flume-1.7/flume-ng-core/src/main/java/org/apache/flume/interceptor/HostInterceptor.java

　　大家，去https://github.com/找到，因为，我的flume是1.7.0的。所以如下

　　修改后的pom.xml为

"http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    4.0.0

    zhouls.bigdata
    flumeDemo
    0.0.1-SNAPSHOT
    jar

    flumeDemo
    http://maven.apache.org

    
        UTF-8
    

    
        
            junit
            junit
            4.12
            test
        
        
        
            org.apache.curator
            curator-framework
            2.10.0
         
    
    
        org.apache.flume
        flume-ng-core
        1.7.0

　　然后，我这里，参考github上的给定参考代码，来写出属于我们自己业务需求的flume自定义拦截器代码编程。

　　MySearchAndReplaceInterceptor.java.java

package zhouls.bigdata.flumeDemo;

import com.google.common.base.Preconditions;
import org.apache.commons.lang.StringUtils;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * Created by zhouls.
 *
 * 使用说明：
 * ======================================================
 * # 定义拦截器
 * agent.sources.kafkaSource.interceptors = i0
 * # 设置拦截器类型
 * # gift_record:giftRecord的意思是会把日志中的gift_record替换为giftRecord
 * agent.sources.kafkaSource.interceptors.i0.type = zhouls.MySearchAndReplaceInterceptor
 * agent.sources.kafkaSource.interceptors.i0.searchReplace = "gift_record:giftRecord,video_info:videoInfo"
 * ======================================================
 */
public class MySearchAndReplaceInterceptor implements Interceptor {

    private static final Logger logger = LoggerFactory
            .getLogger(MySearchAndReplaceInterceptor.class);

    /**
     * 需要替换的字符串信息
     * 格式："key:value,key:value"
     */
    private final String search_replace;
    private String[] splits;
    private String[] key_value;
    private String key;
    private String value;
    private HashMap hashMap = new HashMap();
    private Pattern compile = Pattern.compile("\"type\":\"(\\w+)\"");
    private Matcher matcher;
    private String group;

    private MySearchAndReplaceInterceptor(String search_replace) {
        this.search_replace = search_replace;
    }

    /**
     * 初始化放在，最开始执行一次
     * 把配置的数据初始化到map中，方便后面调用
     */
    public void initialize() {
        try{
            if(StringUtils.isNotBlank(search_replace)){
                splits = search_replace.split(",");
                for (String key_value_pair:splits) {
                    key_value = key_value_pair.split(":");
                    key = key_value[0];
                    value = key_value[1];
                    hashMap.put(key,value);
                }
            }
        }catch (Exception e){
            logger.error("数据格式错误，初始化失败。"+search_replace,e.getCause());
        }

    }
    public void close() {

    }


    /**
     * 具体的处理逻辑
     * @param event
     * @return
     */
    public Event intercept(Event event) {
        try{
            String origBody = new String(event.getBody());
            matcher = compile.matcher(origBody);
            if(matcher.find()){
                group = matcher.group(1);
                if(StringUtils.isNotBlank(group)){
                    String newBody = origBody.replaceAll("\"type\":\""+group+"\"", "\"type\":\""+hashMap.get(group)+"\"");
                    event.setBody(newBody.getBytes());
                }
            }
        }catch (Exception e){
            logger.error("拦截器处理失败！",e.getCause());
        }
        return event;
    }

    public List intercept(List events) {
        for (Event event : events) {
            intercept(event);
        }
        return events;
    }

    public static class Builder implements Interceptor.Builder {
        private static final String SEARCH_REPLACE_KEY = "searchReplace";

        private String searchReplace;

        public void configure(Context context) {
            searchReplace = context.getString(SEARCH_REPLACE_KEY);
            Preconditions.checkArgument(!StringUtils.isEmpty(searchReplace),
                    "Must supply a valid search pattern " + SEARCH_REPLACE_KEY +
                            " (may not be empty)");
        }

        public Interceptor build() {
            Preconditions.checkNotNull(searchReplace,
                    "Regular expression searchReplace required");
            return new MySearchAndReplaceInterceptor(searchReplace);
        }

    }
}

　　然后把MySearchAndReplaceInterceptor这个类导出成一个jar包。

　　同时，大家也可以用maven来打jar包

　　把这个jar包上传到flume1.7.0的lib目录下

[hadoop@master lib]$ rz

[hadoop@master lib]$ ls
apache-log4j-extras-1.1.jar    flume-file-channel-1.7.0.jar              flume-taildir-source-1.7.0.jar  kite-data-core-1.0.0.jar             parquet-hive-bundle-1.4.1.jar
async-1.4.0.jar                flume-hdfs-sink-1.7.0.jar                 flume-thrift-source-1.7.0.jar   kite-data-hbase-1.0.0.jar            parquet-jackson-1.4.1.jar
asynchbase-1.7.0.jar           flume-hive-sink-1.7.0.jar                 flume-tools-1.7.0.jar           kite-data-hive-1.0.0.jar             protobuf-java-2.5.0.jar
avro-1.7.4.jar                 flume-irc-sink-1.7.0.jar                  flume-twitter-source-1.7.0.jar  kite-hadoop-compatibility-1.0.0.jar  scala-library-2.10.5.jar
avro-ipc-1.7.4.jar             flume-jdbc-channel-1.7.0.jar              gson-2.2.2.jar                  libthrift-0.9.0.jar                  serializer-2.7.2.jar
commons-cli-1.2.jar            flume-jms-source-1.7.0.jar                guava-11.0.2.jar                log4j-1.2.17.jar                     servlet-api-2.5-20110124.jar
commons-codec-1.8.jar          flume-kafka-channel-1.7.0.jar             httpclient-4.2.1.jar            lz4-1.2.0.jar                        slf4j-api-1.6.1.jar
commons-collections-3.2.2.jar  flume-kafka-source-1.7.0.jar              httpcore-4.1.3.jar              mapdb-0.9.9.jar                      slf4j-log4j12-1.6.1.jar
commons-compress-1.4.1.jar     flume-ng-auth-1.7.0.jar                   irclib-1.10.jar                 metrics-core-2.2.0.jar               snappy-java-1.1.0.jar
commons-dbcp-1.4.jar           flume-ng-configuration-1.7.0.jar          jackson-annotations-2.3.0.jar   mina-core-2.0.4.jar                  twitter4j-core-3.0.3.jar
commons-io-2.1.jar             flume-ng-core-1.7.0.jar                   jackson-core-2.3.1.jar          MySearchAndReplaceInterceptor.jar    twitter4j-media-support-3.0.3.jar
commons-jexl-2.1.1.jar         flume-ng-elasticsearch-sink-1.7.0.jar     jackson-core-asl-1.9.3.jar      netty-3.9.4.Final.jar                twitter4j-stream-3.0.3.jar
commons-lang-2.5.jar           flume-ng-embedded-agent-1.7.0.jar         jackson-databind-2.3.1.jar      opencsv-2.3.jar                      velocity-1.7.jar
commons-logging-1.1.1.jar      flume-ng-hbase-sink-1.7.0.jar             jackson-mapper-asl-1.9.3.jar    paranamer-2.3.jar                    xalan-2.7.2.jar
commons-pool-1.5.4.jar         flume-ng-kafka-sink-1.7.0.jar             jetty-6.1.26.jar                parquet-avro-1.4.1.jar               xercesImpl-2.9.1.jar
curator-client-2.6.0.jar       flume-ng-log4jappender-1.7.0.jar          jetty-util-6.1.26.jar           parquet-column-1.4.1.jar             xml-apis-1.3.04.jar
curator-framework-2.6.0.jar    flume-ng-morphline-solr-sink-1.7.0.jar    joda-time-2.1.jar               parquet-common-1.4.1.jar             xz-1.0.jar
curator-recipes-2.6.0.jar      flume-ng-node-1.7.0.jar                   jopt-simple-3.2.jar             parquet-encoding-1.4.1.jar           zkclient-0.7.jar
derby-10.11.1.1.jar            flume-ng-sdk-1.7.0.jar                    jsr305-1.3.9.jar                parquet-format-2.0.0.jar
flume-avro-source-1.7.0.jar    flume-scribe-source-1.7.0.jar             kafka_2.10-0.9.0.1.jar          parquet-generator-1.4.1.jar
flume-dataset-sink-1.7.0.jar   flume-spillable-memory-channel-1.7.0.jar  kafka-clients-0.9.0.1.jar       parquet-hadoop-1.4.1.jar
[hadoop@master lib]$ pwd
/home/hadoop/app/flume-1.7.0/lib
[hadoop@master lib]$

drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 13:40 conf_HostInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 14:31 conf_RegexExtractorInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 15:26 conf_SearchandReplaceInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Jul 27 16:26 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$ cp -r conf conf_MySearchAndReplaceInterceptor
[hadoop@master flume-1.7.0]$ ll
total 164
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 bin
-rw-r--r--  1 hadoop hadoop 77387 Oct 11  2016 CHANGELOG
drwxr-xr-x  2 hadoop hadoop  4096 Apr 20 12:00 conf
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 13:40 conf_HostInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 16:27 conf_MySearchAndReplaceInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 14:31 conf_RegexExtractorInterceptor
drwxr-xr-x  2 hadoop hadoop  4096 Jul 27 15:26 conf_SearchandReplaceInterceptor
-rw-r--r--  1 hadoop hadoop  6172 Sep 26  2016 DEVNOTES
-rw-r--r--  1 hadoop hadoop  2873 Sep 26  2016 doap_Flume.rdf
drwxr-xr-x 10 hadoop hadoop  4096 Oct 13  2016 docs
drwxrwxr-x  2 hadoop hadoop  4096 Jul 27 16:26 lib
-rw-r--r--  1 hadoop hadoop 27625 Oct 13  2016 LICENSE
-rw-r--r--  1 hadoop hadoop   249 Sep 26  2016 NOTICE
-rw-r--r--  1 hadoop hadoop  2520 Sep 26  2016 README.md
-rw-r--r--  1 hadoop hadoop  1585 Oct 11  2016 RELEASE-NOTES
drwxrwxr-x  2 hadoop hadoop  4096 Apr 20 12:00 tools
[hadoop@master flume-1.7.0]$

　　修改好log4j.properties ，为了方便管理查看日志

[hadoop@master conf_MySearchAndReplaceInterceptor]$ pwd
/home/hadoop/app/flume-1.7.0/conf_MySearchAndReplaceInterceptor
[hadoop@master conf_MySearchAndReplaceInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 16:27 flume-conf.properties.template
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 16:27 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 16:27 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3107 Jul 27 16:27 log4j.properties
[hadoop@master conf_MySearchAndReplaceInterceptor]$ mv flume-conf.properties.template flume-conf.properties
[hadoop@master conf_MySearchAndReplaceInterceptor]$ vim log4j.properties

#flume.root.logger=DEBUG,console
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume_MySearchAndReplaceInterceptor.log

[hadoop@master conf_MySearchAndReplaceInterceptor]$ ll
total 16
-rw-r--r-- 1 hadoop hadoop 1661 Jul 27 16:27 flume-conf.properties
-rw-r--r-- 1 hadoop hadoop 1455 Jul 27 16:27 flume-env.ps1.template
-rw-r--r-- 1 hadoop hadoop 1565 Jul 27 16:27 flume-env.sh.template
-rw-r--r-- 1 hadoop hadoop 3137 Jul 27 16:29 log4j.properties
[hadoop@master conf_MySearchAndReplaceInterceptor]$ vim flume-conf.properties

　　然后，修改flume的配置文件如下：

　　注意：不能为上面。

　　除非你的程序需要引号（“”），否则不要加引号（“”），本程序不需要引号，因此是错误的

#source的名字
agent1.sources = fileSource
# channels的名字，建议按照type来命名
agent1.channels = memoryChannel
# sink的名字，建议按照目标来命名
agent1.sinks = hdfsSink

# 指定source使用的channel名字
agent1.sources.fileSource.channels = memoryChannel
# 指定sink需要使用的channel的名字,注意这里是channel
agent1.sinks.hdfsSink.channel = memoryChannel


agent1.sources.fileSource.type = exec
agent1.sources.fileSource.command = tail -F /usr/local/log/server.log



#------- fileChannel-1相关配置-------------------------
# channel类型

agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 1000
agent1.channels.memoryChannel.transactionCapacity = 1000
agent1.channels.memoryChannel.byteCapacityBufferPercentage = 20
agent1.channels.memoryChannel.byteCapacity = 800000


#---------拦截器相关配置------------------
#定义拦截器
agent1.sources.r1.interceptors = i1 i2
# 设置拦截器类型
agent1.sources.r1.interceptors.i1.type = zhouls.bigdata.MySearchAndReplaceInterceptor
agent1.sources.r1.interceptors.i1.searchReplace = gift_record:giftRecord,video_info:videoInfo,user_info:userInfo

# 设置拦截器类型
agent1.sources.r1.interceptors.i2.type = regex_extractor
# 设置正则表达式，匹配指定的数据，这样设置会在数据的header中增加log_type="某个值"
agent1.sources.r1.interceptors.i2.regex = "type":"(\\w+)"
agent1.sources.r1.interceptors.i2.serializers = s1
agent1.sources.r1.interceptors.i2.serializers.s1.name = log_type



#---------hdfsSink 相关配置------------------
agent1.sinks.hdfsSink.type = hdfs
# 注意, 我们输出到下面一个子文件夹datax中
agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type}
agent1.sinks.hdfsSink.hdfs.writeFormat = Text
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.callTimeout = 3600000
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true

#当文件大小为52428800字节时，将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollSize = 52428800
#events数据达到该数量的时候，将临时文件滚动成目标文件
agent1.sinks.hdfsSink.hdfs.rollCount = 0
#每隔N s将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollInterval = 1200

#配置前缀和后缀
agent1.sinks.hdfsSink.hdfs.filePrefix=run
agent1.sinks.hdfsSink.hdfs.fileSuffix=.data

　　主要在里面添加拦截器的配置是如下

#---------拦截器相关配置------------------
#定义拦截器
agent1.sources.r1.interceptors = i1 i2
# 设置拦截器类型
agent1.sources.r1.interceptors.i1.type = zhouls.bigdata.MySearchAndReplaceInterceptor
agent1.sources.r1.interceptors.i1.searchReplace = "gift_record:giftRecord,video_info:videoInfo,user_info:userInfo"

# 设置拦截器类型
agent1.sources.r1.interceptors.i2.type = regex_extractor
# 设置正则表达式，匹配指定的数据，这样设置会在数据的header中增加log_type="某个值"
agent1.sources.r1.interceptors.i2.regex = "type":"(\\w+)"
agent1.sources.r1.interceptors.i2.serializers = s1
agent1.sources.r1.interceptors.i2.serializers.s1.name = log_type

　　意思就是，即把gift_record 换成giftRecord

　　　　　　　　　video_info转换成videoInfo

　　　　　　　　　user_info转换成userInfo

　　然后，启动agent服务即可。

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_MySearchAndReplaceInterceptor/  --conf-file conf_MySearchAndReplaceInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

　　我这里，出现了这个错误

2017-07-29 10:17:51,006 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: fileSource started
2017-07-29 10:17:52,792 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)] Serializer = TEXT, UseRawLocalFileSystem = false
2017-07-29 10:17:55,094 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170729//run.1501294672792.data.tmp
2017-07-29 10:17:55,842 (hdfs-hdfsSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-07-29 10:18:00,495 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:352)] Failed while running command: tail -F /usr/local/log/server.log
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:127)
    at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
    at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
    at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:341)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:18:00,544 (timedFlushExecService21-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:327)] Exception occured when processing event batch
org.apache.flume.ChannelException: java.lang.InterruptedException
    at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
    at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
    at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:254)
    at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:323)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

　　见博客

Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法（图文详解）

　　中间，我这里还出现下面这个错误

Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法（图文详解）

　　中间，我这里还出现下面这个错误

Flume启动报错[ERROR - org.apache.flume.sink.hdfs. Hit max consecutive under-replication rotations (30); will not continue rolling files under this path due to under-replication解决办法（图文详解）

[root@master log]# ll
total 4
-rwxr-xr-x 1 root root 1157 Jul 27 14:42 producerLog.sh
-rw-r--r-- 1 root root    0 Jul 27 15:30 server.log
[root@master log]# ./producerLog.sh

　　查看

对于目标文件的生成
我这里，貌似懂了
是要达到那么多的临时文件大小生成后
才会有一股目标目录出来
让它等吧

　　　还有资料说，

flume自定义拦截器实现多行读取日志

　　加了还是没用。

欢迎大家，加入我的微信公众号：大数据躺过的坑人工智能躺过的坑

同时，大家可以关注我的个人博客：

http://www.cnblogs.com/zlslch/ 和 http://www.cnblogs.com/lchzls/ http://www.cnblogs.com/sunnyDream/

详情请见：http://www.cnblogs.com/zlslch/p/7473861.html

　　人生苦短，我愿分享。本公众号将秉持活到老学到老学习无休止的交流分享开源精神，汇聚于互联网和个人学习工作的精华干货知识，一切来于互联网，反馈回互联网。
　　目前研究领域：大数据、机器学习、深度学习、人工智能、数据挖掘、数据分析。语言涉及：Java、Scala、Python、Shell、Linux等。同时还涉及平常所使用的手机、电脑和互联网上的使用技巧、问题和实用软件。只要你一直关注和呆在群里，每天必须有收获

对应本平台的讨论和答疑QQ群：大数据和人工智能躺过的坑（总群）（161156071）

你可能感兴趣的:(Flume自定义拦截器（Interceptors）或自带拦截器时的一些经验技巧总结（图文详解）...)

斤斤计较的婚姻到底有多难？白心之岂必有为
很多人私聊我会问到在哪个人群当中斤斤计较的人最多？我都会回答他，一般婚姻出现问题的斤斤计较的人士会非常多，以我多年经验，在婚姻落的一塌糊涂的人当中，斤斤计较的人数占比在20～30%以上，也就是说10个婚姻出现问题的斤斤计较的人有2-3个有多不减。在婚姻出问题当中，有大量的心理不平衡的、尖酸刻薄的怨妇。在婚姻中仅斤斤计较有两种类型：第一种是物质上的，另一种是精神上的。在物质与精神上抠门已经严重的影响
情绪觉察日记第37天露露_e800
今天是家庭关系规划师的第二阶最后一天，慧萍老师帮我做了个案，帮我处理了埋在心底好多年的一份恐惧，并给了我深深的力量！这几天出来学习，爸妈过来婆家帮我带小孩，妈妈出于爱帮我收拾东西，并跟我先生和婆婆产生矛盾，妈妈觉得他们没有照顾好我…。今晚回家见到妈妈，我很欣赏她并赞扬她，妈妈说今晚要跟我睡我说好，当我们俩躺在床上准备睡觉的时候，我握着妈妈的手对她说:妈妈这几天辛苦你了，你看你多利害把我们的家收拾得
芦花鞋一四许叶晗
又是在一个寒冷的夏日里，青铜和葵花决定今天一起去卖芦花鞋，奶奶亲手给他们做了一碗热乎乎的粥对他们说:“就靠你们两挣生活费了这碗粥赶紧趁热喝了吧！”于是青铜和葵花喝完了奶奶给她们做的粥，就准备去镇上卖卢花鞋，这回青铜和葵花穿着新的芦花鞋来到了镇上。青铜这回看到了很多人都在卖，用手势表达对葵花说:“这回有好多人在抢我们生意呢！我们必须得吆喝起来。”葵花点了点头。可是谁知他们也大声的叫，卖芦花喽！卖芦花
QQ群采集助手，精准引流必备神器 2401_87347160 其他经验分享
功能概述微信群查找与筛选工具是一款专为微信用户设计的辅助工具，它通过关键词搜索功能，帮助用户快速找到相关的微信群，并提供筛选是否需要验证的群组的功能。主要功能关键词搜索：用户可以输入关键词，工具将自动查找包含该关键词的微信群。筛选功能：工具提供筛选机制，用户可以选择是否只显示需要验证或不需要验证的群组。精准引流：通过上述功能，用户可以更精准地找到目标群组，进行有效的引流操作。3.设备需求该工具可以
关于沟通这件事，项目经理不需要每次都面对面进行流程大师兄
很多项目经理都会遇到这样的问题，项目中由于事情太多，根本没有足够的时间去召开会议，那在这种情况下如何去有效地管理项目中的利益相关者？当然，不建议电子邮件也不需要开会的话，建议可以采取下面几种方式来形成有效的沟通，这几种方式可以帮助你努力的通过各种办法来保持和各方面的联系。项目经理首先要问自己几个问题，项目中哪些利益相关者是必须要进行沟通的？可以列出项目中所有的利益相关者清单，同时也整理出项目中哪些
机器学习与深度学习间关系与区别 ℒℴѵℯ心·动ꦿ໊ོ꫞ 人工智能学习深度学习 python
一、机器学习概述定义机器学习（MachineLearning,ML）是一种通过数据驱动的方法，利用统计学和计算算法来训练模型，使计算机能够从数据中学习并自动进行预测或决策。机器学习通过分析大量数据样本，识别其中的模式和规律，从而对新的数据进行判断。其核心在于通过训练过程，让模型不断优化和提升其预测准确性。主要类型1.监督学习（SupervisedLearning）监督学习是指在训练数据集中包含输入
铭刻于星（四十二）随风至
69夜晚，绍敏同学做完功课后，看了眼房外，没听到动静才敢从书包的夹层里拿出那个心形纸团。折痕压得很深，都有些旧了，想来是已经写好很久了。绍敏同学慢慢地、轻轻地捏开折叠处，待到全部拆开后，又反复抚平纸张，然后仔细地一字字默看。只是开头的三个字是第一次看到，让她心漏跳了几拍。“亲爱的绍敏：从四年级的时候，我就喜欢你了，但是我一直不敢说，怕影响你学习。六年级的时候听说有人跟你表白，你接受了，我很难过，但
底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说造命者说
底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说我叫吴起，生于公元前440年的战国初期，正是群雄并起、天下纷争不断的时候。后人说我是军事家、政治家、改革家，是兵家代表人物。评价我一生历仕鲁、魏、楚三国，通晓兵家、法家、儒家三家思想，在内政军事上都有极高的成就。周安王二十一年（公元前381年），因变法得罪守旧贵族，被人乱箭射死。我出生在卫国一个“家累万金”的富有家庭，从年轻时候起就不甘平凡
2020-01-25 晴岚85
郑海燕坚持分享590天2020.1.24在生活中只存在两个问题。一个问题是：你知道想要达成的目标是什么，但却不知道如何才能达成；另一个问题是：你不知道你的目标是什么。前一个是行动的问题，后一个是结果的问题。通过制定具体的下一步行动，可以解决不知道如何开始行动的问题。而通过去想象结果，对结果做预估，可以解决找不着目标的问题。对于所有吸引我们注意力，想要完成的任务，你可以先想象一下，预期的结果究竟是什
随笔 | 仙一般的灵气海思沧海
仙岛今天，我看了你全部，似乎已经进入你的世界我不知道，这是否是梦幻，还是你仙一般的灵气吸引了我也许每一个人都要有一份属于自己的追求，这样才能够符合人生的梦想，生活才能够充满着阳光与快乐我不知道，我为什么会这样的感叹，是在感叹自己的人生，还是感叹自己一直没有孜孜不倦的追求只感觉虚度了光阴，每天活在自己的梦中，活在一个不真实的世界是在逃避自己，还是在逃避周围的一切有时候我嘲笑自己，嘲笑自己如此的虚无，
想家爆米花机
也许不同于大家对家乡的思念，我对家乡甚至是疯狂的不舍。还未踏出车站就感觉到幸福，我享受这里的夕阳、这里的浓烈柴火味、这里每一口家常菜。我是宅女，我贪恋家的安逸。刚刚踏出大学校门，初出茅庐，无法适应每年只能国庆和春节回家。我焦虑、失眠、无端发脾气，是无法适应工作的节奏，是无法接受我将一步步离开家乡的事实。我不想承认自己胸无大志，选择再次踏上征程。图片发自App
【iOS】MVC设计模式 Magnetic_h ios mvc 设计模式 objective-c 学习 ui
MVC前言如何设计一个程序的结构，这是一门专门的学问，叫做"架构模式"（architecturalpattern），属于编程的方法论。MVC模式就是架构模式的一种。它是Apple官方推荐的App开发架构，也是一般开发者最先遇到、最经典的架构。MVC各层controller层Controller/ViewController/VC（控制器）负责协调Model和View，处理大部分逻辑它将数据从Mod
OC语言多界面传值五大方式 Magnetic_h ios ui 学习 objective-c 开发语言
前言在完成暑假仿写项目时，遇到了许多需要用到多界面传值的地方，这篇博客来总结一下比较常用的五种多界面传值的方式。属性传值属性传值一般用前一个界面向后一个界面传值，简单地说就是通过访问后一个视图控制器的属性来为它赋值，通过这个属性来做到从前一个界面向后一个界面传值。首先在后一个界面中定义属性@interfaceBViewController:UIViewController@propertyNSSt
一百九十四章. 自相矛盾巨木擎天
唉！就这么一夜，林子感觉就像过了很多天似的，先是回了阳间家里，遇到了那么多不可思议的事情儿。特别是小伙伴们，第二次与自己见面时，僵硬的表情和恐怖的气氛，让自己如坐针毡，打从心眼里难受！还有东子，他现在还好吗？有没有被人欺负？护城河里的小鱼小虾们，还都在吗？水不会真的干枯了吧？那对相亲相爱漂亮的太平鸟儿，还好吧！春天了，到了做窝、下蛋、喂养小鸟宝宝的时候了，希望它们都能够平安啊！虽然没有看见家人，也
UI学习——cell的复用和自定义cell Magnetic_h ui 学习
目录cell的复用手动（非注册）自动（注册）自定义cellcell的复用在iOS开发中，单元格复用是一种提高表格（UITableView）和集合视图（UICollectionView）滚动性能的技术。当一个UITableViewCell或UICollectionViewCell首次需要显示时，如果没有可复用的单元格，则视图会创建一个新的单元格。一旦这个单元格滚动出屏幕，它就不会被销毁。相反，它被添
element实现动态路由+面包屑软件技术NINI vue案例 vue.js 前端
el-breadcrumb是ElementUI组件库中的一个面包屑导航组件，它用于显示当前页面的路径，帮助用户快速理解和导航到应用的各个部分。在Vue.js项目中，如果你已经安装了ElementUI，就可以很方便地使用el-breadcrumb组件。以下是一个基本的使用示例：安装ElementUI（如果你还没有安装的话）:你可以通过npm或yarn来安装ElementUI。bash复制代码npmi
10月|愿你的青春不负梦想-读书笔记-01 Tracy的小书斋
本书的作者是俞敏洪，大家都很熟悉他了吧。俞敏洪老师是我行业的领头羊吧，也是我事业上的偶像。本日摘录他书中第一章中的金句：『一个人如果什么目标都没有，就会浑浑噩噩，感觉生命中缺少能量。能给我们能量的，是对未来的期待。第一件事，我始终为了进步而努力。与其追寻全世界的骏马，不如种植丰美的草原，到时骏马自然会来。第二件事，我始终有阶段性的目标。什么东西能给我能量？答案是对未来的期待。』读到这里的时候，我便
C语言宏函数南林yan C语言 c语言
一、什么是宏函数？通过宏定义的函数是宏函数。如下，编译器在预处理阶段会将Add(x,y)替换为((x)*(y))#defineAdd(x,y)((x)*(y))#defineAdd(x,y)((x)*(y))intmain(){inta=10;intb=20;intd=10;intc=Add(a+d,b)*2;cout<
地推话术，如何应对地推过程中家长的拒绝校师学
相信校长们在做地推的时候经常遇到这种情况：市场专员反馈家长不接单，咨询师反馈难以邀约这些家长上门，校区地推疲软，招生难。为什么？仅从地推层面分析，一方面因为家长受到的信息轰炸越来越多，对信息越来越“免疫”；而另一方面地推人员的专业能力和营销话术没有提高，无法应对家长的拒绝，对有意向的家长也不知如何跟进，眼睁睁看着家长走远；对于家长的疑问，更不知道如何有技巧地回答，机会白白流失。由于回答没技巧和专业
谢谢你们，爱你们！鹿游儿
昨天家人去泡温泉，二个孩子也带着去，出发前一晚，匆匆下班，赶回家和孩子一起收拾。饭后，我拿出笔和本子（上次去澳门时做手帐的本子）写下了1\2\3\4\5\6\7\8\9,让后让小壹去思考，带什么出发去旅游呢？她在对应的数字旁边画上了，泳衣、泳圈、肖恩、内衣内裤、tapuy、拖鞋……画完后，就让她自己对着这个本子，将要带的，一一带上，没想到这次带的书还是这本《便便工厂》(晚上姑婆发照片过来，妹妹累得
C语言如何定义宏函数？小九格物 c语言
在C语言中，宏函数是通过预处理器定义的，它在编译之前替换代码中的宏调用。宏函数可以模拟函数的行为，但它们不是真正的函数，因为它们在编译时不会进行类型检查，也不会分配存储空间。宏函数的定义通常使用#define指令，后面跟着宏的名称和参数列表，以及宏展开后的代码。宏函数的定义方式：1.基本宏函数：这是最简单的宏函数形式，它直接定义一个表达式。#defineSQUARE(x)((x)*(x))2.带参
微服务下功能权限与数据权限的设计与实现 nbsaas-boot 微服务 java 架构
在微服务架构下，系统的功能权限和数据权限控制显得尤为重要。随着系统规模的扩大和微服务数量的增加，如何保证不同用户和服务之间的访问权限准确、细粒度地控制，成为设计安全策略的关键。本文将讨论如何在微服务体系中设计和实现功能权限与数据权限控制。1.功能权限与数据权限的定义功能权限：指用户或系统角色对特定功能的访问权限。通常是某个用户角色能否执行某个操作，比如查看订单、创建订单、修改用户资料等。数据权限：
理解Gunicorn：Python WSGI服务器的基石范范0825 ipython linux 运维
理解Gunicorn：PythonWSGI服务器的基石介绍Gunicorn，全称GreenUnicorn，是一个为PythonWSGI（WebServerGatewayInterface）应用设计的高效、轻量级HTTP服务器。作为PythonWeb应用部署的常用工具，Gunicorn以其高性能和易用性著称。本文将介绍Gunicorn的基本概念、安装和配置，帮助初学者快速上手。1.什么是Gunico
小丽成长记（四十三）玲玲54321
小丽发现，即使她好不容易调整好自己的心态下一秒总会有不确定的伤脑筋的事出现，一个接一个的问题，人生就没有停下的时候，小问题不断出现。不过她今天看的书，她接受了人生就是不确定的，厉害的人就是不断创造确定性，在Ta的领域比别人多的确定性就能让自己脱颖而出，显示价值从而获得的比别人多的利益。正是这样的原因，因为从前修炼自己太少，使得她现在在人生道路上打怪起来困难重重，她似乎永远摆脱不了那种无力感，有种习
学点心理知识，呵护孩子健康静候花开_7090
昨天听了华中师范大学教育管理学系副教授张玲老师的《哪里才是学生心理健康的最后庇护所，超越教育与技术的思考》的讲座。今天又重新学习了一遍，收获匪浅。张玲博士也注意到了当今社会上的孩子由于心理问题导致的自残、自杀及伤害他人等恶性事件。她向我们普及了一个重要的命题，她说心理健康的一些基本命题，我们与我们通常的一些教育命题是不同的，她还举了几个例子，让我们明白我们原来以为的健康并非心理学上的健康。比如如果
2021年12月19日，春蕾教育集团团建活动感受——黄晓丹黄错错加油
感受:1.从陌生到熟悉的过程。游戏环节让我们在轻松的氛围中得到了锻炼，也增长了不少知识。2.游戏过程中，我们贡献的是个人力量，展现的是团队的力量。它磨合的往往不止是工作的熟悉，更是观念上契合度的贴近。3.这和工作是一样的道理。在各自的岗位上，每个人摆正自己的位置、各司其职充分发挥才能，并团结一致劲往一处使，才能实现最大的成功。新知:1.团队精神需要不断地创新。过去，人们把创新看作是冒风险，现在人们
Cell Insight | 单细胞测序技术又一新发现，可用于HIV-1和Mtb共感染个体诊断尐尐呅
结核病是艾滋病合并其他疾病中导致患者死亡的主要原因。其中结核病由结核分枝杆菌（Mycobacteriumtuberculosis,Mtb）感染引起，获得性免疫缺陷综合症（艾滋病）由人免疫缺陷病毒（Humanimmunodeficiencyvirustype1,HIV-1）感染引起。国家感染性疾病临床医学研究中心/深圳市第三人民医院张国良团队携手深圳华大生命科学研究院吴靓团队，共同研究得出单细胞测序
c++ 的iostream 和 c++的stdio的区别和联系黄卷青灯77 c++算法开发语言 iostream stdio
在C++中，iostream和C语言的stdio.h都是用于处理输入输出的库，但它们在设计、用法和功能上有许多不同。以下是两者的区别和联系：区别1.编程风格iostream（C++风格）：C++标准库中的输入输出流类库，支持面向对象的输入输出操作。典型用法是cin（输入）和cout（输出），使用>操作符来处理数据。更加类型安全，支持用户自定义类型的输入输出。#includeintmain(){in
瑶池防线谜影梦蝶
冥华虽然逃过了影梦的军队，但他是一个忠臣，他选择上报战况。败给影梦后成逃兵，高层亡尔还活着，七重天失守......随便一条，即可处死冥华。冥华自然是知道以仙界高层的习性此信一发自己必死无疑，但他还选择上报实情，因为责任。同样此信送到仙宫后，知道此事的人，大多数人都认定冥华要完了，所以上到仙界高层，下到扫大街的，包括冥华自己，全都准备好迎接冥华之死。如果仙界现在还属于两方之争的话，冥华必死无疑。然而
爬山后遗症璃绛
爬山，攀登，一步一步走向制高点，是一种挑战。成功抵达是一种无法言语的快乐，在山顶吹吹风，看看风景，这是从未有过的体验。然而，爬山一时爽，下山腿打颤，颠簸的路，一路向下走，腿部力量不够，走起来抖到不行，停不下来了！第二天必定腿疼，浑身酸痛，坐立难安！
JAVA中的Enum 周凡杨 java enum 枚举
Enum是计算机编程语言中的一种数据类型---枚举类型。在实际问题中，有些变量的取值被限定在一个有限的范围内。例如，一个星期内只有七天我们通常这样实现上面的定义： public String monday; public String tuesday; public String wensday; public String thursday
赶集网mysql开发36条军规 Bill_chen mysql 业务架构设计 mysql调优 mysql性能优化
(一)核心军规 (1)不在数据库做运算 cpu计算务必移至业务层； (2)控制单表数据量 int型不超过1000w，含char则不超过500w；合理分表；限制单库表数量在300以内； (3)控制列数量字段少而精，字段数建议在20以内
Shell test命令 daizj shell 字符串 test 数字文件比较
Shell test命令 Shell中的 test 命令用于检查某个条件是否成立，它可以进行数值、字符和文件三个方面的测试。数值测试参数说明 -eq 等于则为真 -ne 不等于则为真 -gt 大于则为真 -ge 大于等于则为真 -lt 小于则为真 -le 小于等于则为真实例演示： num1=100 num2=100if test $[num1]
XFire框架实现WebService(二) 周凡杨 java webservice
有了XFire框架实现WebService(一)，就可以继续开发WebService的简单应用。 Webservice的服务端(WEB工程)：两个java bean类： Course.java package cn.com.bean; public class Course { private
重绘之画图板朱辉辉33 画图板
上次博客讲的五子棋重绘比较简单，因为只要在重写系统重绘方法paint（）时加入棋盘和棋子的绘制。这次我想说说画图板的重绘。画图板重绘难在需要重绘的类型很多，比如说里面有矩形，园，直线之类的，所以我们要想办法将里面的图形加入一个队列中，这样在重绘时就
Java的IO流西蜀石兰 java
刚学Java的IO流时，被各种inputStream流弄的很迷糊，看老罗视频时说想象成插在文件上的一根管道，当初听时觉得自己很明白，可到自己用时，有不知道怎么代码了。。。每当遇到这种问题时，我习惯性的从头开始理逻辑，会问自己一些很简单的问题，把这些简单的问题想明白了，再看代码时才不会迷糊。 IO流作用是什么？答：实现对文件的读写，这里的文件是广义的； Java如何实现程序到文件
No matching PlatformTransactionManager bean found for qualifier 'add' - neither 林鹤霄
java.lang.IllegalStateException: No matching PlatformTransactionManager bean found for qualifier 'add' - neither qualifier match nor bean name match! 网上找了好多的资料没能解决，后来发现：项目中使用的是xml配置的方式配置事务，但是
Row size too large (> 8126). Changing some columns to TEXT or BLOB aigo column
原文：http://stackoverflow.com/questions/15585602/change-limit-for-mysql-row-size-too-large 异常信息： Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAM
JS 格式化时间 alxw4616 JavaScript
/** * 格式化时间 2013/6/13 by 半仙 [email protected] * 需要 pad 函数 * 接收可用的时间值. * 返回替换时间占位符后的字符串 * * 时间占位符:年 Y 月 M 日 D 小时 h 分 m 秒 s 重复次数表示占位数 * 如 YYYY 4占4位 YY 占2位<p></p> * MM DD hh mm
队列中数据的移除问题百合不是茶队列移除
队列的移除一般都是使用的remov();都可以移除的,但是在昨天做线程移除的时候出现了点问题,没有将遍历出来的全部移除, 代码如下; // package com.Thread0715.com; import java.util.ArrayList; public class Threa
Runnable接口使用实例 bijian1013 java thread Runnable java多线程
Runnable接口 a. 该接口只有一个方法：public void run(); b. 实现该接口的类必须覆盖该run方法 c. 实现了Runnable接口的类并不具有任何天
oracle里的extend详解 bijian1013 oracle 数据库 extend
扩展已知的数组空间，例： DECLARE TYPE CourseList IS TABLE OF VARCHAR2(10); courses CourseList; BEGIN -- 初始化数组元素，大小为3 courses := CourseList('Biol 4412 ', 'Psyc 3112 ', 'Anth 3001 '); --
【httpclient】httpclient发送表单POST请求 bit1129 httpclient
浏览器Form Post请求浏览器可以通过提交表单的方式向服务器发起POST请求，这种形式的POST请求不同于一般的POST请求 1. 一般的POST请求，将请求数据放置于请求体中，服务器端以二进制流的方式读取数据，HttpServletRequest.getInputStream()。这种方式的请求可以处理任意数据形式的POST请求，比如请求数据是字符串或者是二进制数据 2. Form
【Hive十三】Hive读写Avro格式的数据 bit1129 hive
1. 原始数据 hive> select * from word; OK 1 MSN 10 QQ 100 Gtalk 1000 Skype 2. 创建avro格式的数据表 hive> CREATE TABLE avro_table(age INT, name STRING)STORE
nginx+lua+redis自动识别封解禁频繁访问IP ronin47
在站点遇到攻击且无明显攻击特征，造成站点访问慢，nginx不断返回502等错误时，可利用nginx+lua+redis实现在指定的时间段内，若单IP的请求量达到指定的数量后对该IP进行封禁，nginx返回403禁止访问。利用redis的expire命令设置封禁IP的过期时间达到在指定的封禁时间后实行自动解封的目的。一、安装环境： CentOS x64 release 6.4(Fin
java-二叉树的遍历-先序、中序、后序（递归和非递归）、层次遍历 bylijinnan java
import java.util.LinkedList; import java.util.List; import java.util.Stack; public class BinTreeTraverse { //private int[] array={ 1, 2, 3, 4, 5, 6, 7, 8, 9 }; private int[] array={ 10,6,
Spring源码学习-XML 配置方式的IoC容器启动过程分析 bylijinnan java spring IOC
以FileSystemXmlApplicationContext为例，把Spring IoC容器的初始化流程走一遍： ApplicationContext context = new FileSystemXmlApplicationContext ("C:/Users/ZARA/workspace/HelloSpring/src/Beans.xml&q
[科研与项目]民营企业请慎重参与军事科技工程 comsci 企业
军事科研工程和项目并非要用最先进，最时髦的技术，而是要做到“万无一失” 而民营科技企业在搞科技创新工程的时候，往往考虑的是技术的先进性，而对先进技术带来的风险考虑得不够，在今天提倡军民融合发展的大环境下，这种“万无一失”和“时髦性”的矛盾会日益凸显。。。。。。所以请大家在参与任何重大的军事和政府项目之前，对
spring 定时器-两种方式 cuityang spring quartz 定时器
方式一：间隔一定时间运行 <bean id="updateSessionIdTask" class="com.yang.iprms.common.UpdateSessionTask" autowire="byName" /> <bean id="updateSessionIdSchedule
简述一下关于BroadView站点的相关设计 damoqiongqiu view
终于弄上线了，累趴，戳这里http://www.broadview.com.cn 简述一下相关的技术点前端：jQuery+BootStrap3.2+HandleBars，全站Ajax（貌似对SEO的影响很大啊！怎么破？），用Grunt对全部JS做了压缩处理，对部分JS和CSS做了合并（模块间存在很多依赖，全部合并比较繁琐，待完善）。后端：U
运维 PHP问题汇总 dcj3sjt126com windows2003
1、Dede(织梦)发表文章时,内容自动添加关键字显示空白页解决方法：后台>系统>系统基本参数>核心设置>关键字替换（是/否），这里选择“是”。后台>系统>系统基本参数>其他选项>自动提取关键字，这里选择“是”。 2、解决PHP168超级管理员上传图片提示你的空间不足网站是用PHP168做的，反映使用管理员在后台无法
mac 下安装php扩展 - mcrypt dcj3sjt126com PHP
MCrypt是一个功能强大的加密算法扩展库，它包括有22种算法，phpMyAdmin依赖这个PHP扩展，具体如下：下载并解压libmcrypt-2.5.8.tar.gz。在终端执行如下命令： tar zxvf libmcrypt-2.5.8.tar.gz cd libmcrypt-2.5.8/ ./configure --disable-posix-threads --
MongoDB更新文档 [四] eksliang mongodb Mongodb更新文档
MongoDB更新文档转载请出自出处：http://eksliang.iteye.com/blog/2174104 MongoDB对文档的CURD，前面的博客简单介绍了，但是对文档更新篇幅比较大，所以这里单独拿出来。语法结构如下： db.collection.update( criteria, objNew, upsert, multi) 参数含义参数
Linux下的解压，移除，复制，查看tomcat命令 y806839048 tomcat
重复myeclipse生成webservice有问题删除以前的，干净 1、先切换到：cd usr/local/tomcat5/logs 2、tail -f catalina.out 3、这样运行时就可以实时查看运行日志了 Ctrl+c 是退出tail命令。有问题不明的先注掉 cp /opt/tomcat-6.0.44/webapps/g
Spring之使用事务缘由(3-XML实现) ihuning spring
用事务通知声明式地管理事务事务管理是一种横切关注点。为了在 Spring 2.x 中启用声明式事务管理，可以通过 tx Schema 中定义的 <tx:advice> 元素声明事务通知，为此必须事先将这个 Schema 定义添加到 <beans> 根元素中去。声明了事务通知后，就需要将它与切入点关联起来。由于事务通知是在 <aop:
GCD使用经验与技巧浅谈啸笑天 GC
前言 GCD(Grand Central Dispatch)可以说是Mac、iOS开发中的一大“利器”，本文就总结一些有关使用GCD的经验与技巧。 dispatch_once_t必须是全局或static变量这一条算是“老生常谈”了，但我认为还是有必要强调一次，毕竟非全局或非static的dispatch_once_t变量在使用时会导致非常不好排查的bug，正确的如下： 1
linux（Ubuntu）下常用命令备忘录1 macroli linux 工作 ubuntu
在使用下面的命令是可以通过--help来获取更多的信息1,查询当前目录文件列表：ls ls命令默认状态下将按首字母升序列出你当前文件夹下面的所有内容，但这样直接运行所得到的信息也是比较少的，通常它可以结合以下这些参数运行以查询更多的信息： ls / 显示/.下的所有文件和目录 ls -l 给出文件或者文件夹的详细信息 ls -a 显示所有文件，包括隐藏文
nodejs同步操作mysql qiaolevip 学习永无止境每天进步一点点 mysql nodejs
// db-util.js var mysql = require('mysql'); var pool = mysql.createPool({ connectionLimit : 10, host: 'localhost', user: 'root', password: '', database: 'test', port: 3306 });
一起学Hive系列文章 superlxw1234 hive Hive入门
[一起学Hive]系列文章目录贴，入门Hive，持续更新中。 [一起学Hive]之一—Hive概述，Hive是什么 [一起学Hive]之二—Hive函数大全-完整版 [一起学Hive]之三—Hive中的数据库(Database)和表(Table) [一起学Hive]之四-Hive的安装配置 [一起学Hive]之五-Hive的视图和分区 [一起学Hive
Spring开发利器：Spring Tool Suite 3.7.0 发布 wiselyman spring
Spring Tool Suite(简称STS)是基于Eclipse，专门针对Spring开发者提供大量的便捷功能的优秀开发工具。在3.7.0版本主要做了如下的更新：将eclipse版本更新至Eclipse Mars 4.5 GA Spring Boot(JavaEE开发的颠覆者集大成者，推荐大家学习)的配置语言YAML编辑器的支持(包含自动提示，