flume搭建调试
flume搭建调试
Installing CDH3
https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation
流水账，备忘。
wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo -O /etc/yum.repos.d/cloudera.repo yum search hadoop yum -y install hadoop-0.20 yum -y install hadoop-0.20-namenode yum -y install hadoop-0.20-datanode #yum -y install hadoop-0.20-secondarynamenode yum -y install hadoop-0.20-jobtracker yum -y install hadoop-0.20-tasktracker
Installing CDH3 Components
https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3Components
yum install
install/Use
---------------------------------
Flume flume
Sqoop sqoop
Hue hue
Pig hadoop-pig
Hive hadoop-hive
HBase hadoop-hbase
ZooKeeper hadoop-zookeeper
Oozie server oozie
Oozie client oozie-client
Whirr whirr
Snappy hadoop-0.20-native
Mahout mahout
flume分为：
flume 核心
flume.node 作为节点的服务自启动脚本
flume.master 作为maaster的服务自启动脚本
yum install flume*
[root@flume-hadoop-node-1 ~]# flume usage: flume command [args...] commands include: dump Takes a specified source and dumps to console source Takes a specified source and dumps to console node Start a Flume node/agent (with watchdog) master Start a Flume Master server (with watchdog) version Dump flume build version information node_nowatch Start a flume node/agent (no watchdog) master_nowatch Start a Flume Master server (no watchdog) class <class> Run specified fully qualified class using Flume environment (no watchdog) ex: flume com.cloudera.flume.agent.FlumeNode classpath Dump the classpath used by the java executables shell Start the flume shell killmaster Kill a running master dumplog Takes a specified WAL/DFO log file and dumps to console sink Start a one-shot flume node with console source and specified sink cd /etc/flume/conf mv flume-site.xml.template flume-site.xml vi flume-site.xml #修改masterhost为你的host /etc/init.d/flume-master start /etc/init.d/flume-node start
flume文档
http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html
flume总的来说，是面向流的设计，“source“和”sink"分别代表产生和消费，push、pull都支持，可以扩展支持各种数据源，及数据的处理，非常灵活。
先停掉服务，以前台模式运行，方便查看各种输出，直观的了解一把
/etc/init.d/flume-master stop && /etc/init.d/flume-node stop
启动flume
flume dump console
启动之后，你可以在输入任何字符，然后会有来自flume的回显，因为我们参数指定了console，这个其实是配置flume的source为console的输入，默认sink也是console
source为文件的情况
flume dump 'text("/etc/services")'
tail文件末尾信息的方法
flume dump 'tail("testfile")'
testfile可以不存在，没有问题，我们在另外的console里面创建这个文件，并添加些内容
[root@flume-hadoop-node-1 tmp]# echo "test flume">testfile [root@flume-hadoop-node-1 tmp]# echo "test flume 123">testfile [root@flume-hadoop-node-1 tmp]# echo "test flume 123">>testfile [root@flume-hadoop-node-1 tmp]# echo "test flume 1234">>testfile [root@flume-hadoop-node-1 tmp]# echo "test flume 12345\r\n123456">>testfile
在flume这边，就可以实时的看到反馈
2012-01-06 20:42:55,818 [main] INFO agent.LogicalNodeManager: Loading node name with FlumeConfigData: {srcVer:'Thu Jan 01 08:00:00 CST 1970' snkVer:'Thu Jan 01 08:00:00 CST 1970' ts='Thu Jan 01 08:00:00 CST 1970' flowId:'null' source:'tail( "testfile" )' sink:'console' } 2012-01-06 20:42:55,836 [main] INFO agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Thu Jan 01 08:00:00 CST 1970' snkVer:'Thu Jan 01 08:00:00 CST 1970' ts='Thu Jan 01 08:00:00 CST 1970' flowId:'null' source:'tail( "testfile" )' sink:'console' } 2012-01-06 20:42:55,920 [logicalNode dump-10] INFO debug.ConsoleEventSink: ConsoleEventSink( debug ) opened 2012-01-06 20:42:55,973 [main] INFO agent.FlumeNode: Hadoop Security enabled: false flume-hadoop-node-1 [INFO Fri Jan 06 20:43:21 CST 2012] { tailSrcFile : (long)8387236824819002469 (string) 'testfile' (double)4.914663849160389E252 } test flume flume-hadoop-node-1 [INFO Fri Jan 06 20:43:36 CST 2012] { tailSrcFile : (long)8387236824819002469 (string) 'testfile' (double)4.914663849160389E252 } 123 flume-hadoop-node-1 [INFO Fri Jan 06 20:43:48 CST 2012] { tailSrcFile : (long)8387236824819002469 (string) 'testfile' (double)4.914663849160389E252 } test flume 123 flume-hadoop-node-1 [INFO Fri Jan 06 20:43:56 CST 2012] { tailSrcFile : (long)8387236824819002469 (string) 'testfile' (double)4.914663849160389E252 } test flume 1234 flume-hadoop-node-1 [INFO Fri Jan 06 20:44:11 CST 2012] { tailSrcFile : (long)8387236824819002469 (string) 'testfile' (double)4.914663849160389E252 } test flume 12345\\r\\n123456
多个文件，也是可以的
flume dump 'multitail("test1", "test2")'
默认情况下，tail会处理文件的每一行，并分别生成event，默认分隔符是“\n”，并且不会排除分隔符本身，如果你需要自定义分隔符（采用正则表达式），也是可以的，支持
”prev":分隔符属于前一个event
"next"：分隔符属于下一个event
"exclude"：分隔符丢弃
tail("file", delim="\n\n+", delimMode="exclude") tail("file", delim="</a>", delimMode="prev")
开启一个UDP服务，并监听5140端口
flume dump 'syslogUdp(5140)'
flume web console
http://10.129.8.125:35871/flumemaster.jsp
Cloudera Manager Free Edition
https://ccp.cloudera.com/display/express37/Cloudera+Manager+Free+Edition+Documentation
wget http://archive.cloudera.com/cloudera-manager/installer/latest/cloudera-manager-installer.bin chmod a+x cloudera-manager-installer.bin ./cloudera-manager-installer.bin
安装之前，先禁用Selinux
vi /etc/selinux/config -- SELINUX=disabled -- setenforce 0
./cloudera-manager-installer.bin
安装失败，查看日志，发现安装包下载不下来，只能手动下载安装了。
手动安装JDK
wget http://archive.cloudera.com/cloudera-manager/redhat/5/x86_64/cloudera-manager/3/RPMS/jdk-6u21-linux-amd64.rpm rpm -Uhv jdk-6u21-linux-amd64.rpm
http://archive.cloudera.com/cloudera-manager/redhat/5/x86_64/cloudera-manager/3/RPMS/cloudera-manager-daemons-3.7.2.143-1.noarch.rpm
----------------------------华丽的不行了的分割线-----------------------------------------
2台机器：125 126
125上配置：
vi /etc/flume/conf/flume-site.xml <property> <name>flume.collector.event.host</name> <value>collector</value> <description>This is the host name of the default "remote" collector. </description> </property> <property> <name>flume.collector.port</name> <value>35853</value> <description>This default tcp port that the collector listens to in order to receive events it is collecting. </description> </property>
启动flume各节点
flume node_watch -n collector
HDFS服务器设置（新配）
hdfs://10.129.8.126/
cp /usr/lib/hadoop/example-confs/conf.pseudo/* /etc/hadoop/conf/ mkdir /var/lib/hadoop-0.20/cache/hadoop/dfs/name -p chmod 777 -R /var/lib/hadoop-0.20/ sudo -u hdfs hadoop namenode -format （注意大写的：Y） [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://127.0.0.1/ ls: Wrong FS: hdfs://127.0.0.1/, expected: hdfs://cloudera-node-1 Usage: java FsShell [-ls <path>] [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1 ls: Pathname from hdfs://cloudera-node-1 is not a valid DFS filename. Usage: java FsShell [-ls <path>] [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1/ [root@cloudera-node-1 logs]# hadoop fs -mkdir hdfs://cloudera-node-1/test [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1/ Found 1 items drwxr-xr-x - root supergroup 0 2012-02-03 00:54 /test [root@cloudera-node-1 logs]#
修改hadoop配置，使用外部ip
vi /etc/hadoop/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://10.129.8.126:8020</value> </property> /etc/init.d/hadoop-0.20-namenode restart [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://10.129.8.126/ Found 1 items drwxr-xr-x - root supergroup 0 2012-02-03 00:54 /test
设置访问权限：
hadoop dfs -chmod 777 hdfs://10.129.8.126/flume/ hadoop dfs -chmod 777 hdfs://10.129.8.126/flume/*
126节点，启动flume
flume node_nowatch
打开flume master
http://10.129.8.125:35871/flumemaster.jsp
cloudera-node-1 : text("/etc/services") | agentSink("10.129.8.125",35853); collector : collectorSource(35853) | collectorSink("hdfs://10.129.8.126/flume/","srcdata");
？？
Flume’s Tiered Event Sources
collectorSource[(port)]
Collector source. Listens for data from agentSinks forwarding to port port. If port is not specified, the node default collector TCP port, 35853.
!!
hadoop dfs -ls hdfs://10.129.8.126/flume/
125上报错：
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /flume/srcdata20120203-013616957+0800.2438481505068540.00000021.tmp could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy6.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy6.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3178) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3047) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClient.java:2305) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2500)
vi /etc/hadoop/conf/hdfs-site.xml
设置replica为0.也不行
<delete> 设置 vi /etc/hadoop/conf/hdfs-site.xml <property> <name>dfs.thrift.address</name> <value>10.129.8.126:10090</value> </property> </delete>
vi /etc/hadoop/conf/masters
替换localhost为ip：10.129.8.126
还是不行，在125上手动执行upload操作
vi a.txt hadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.1
报一样的错误，
在126上执行如上操作，报同样错误，MD
看来是datanode挂了，但是服务显示启动，重启试试。
[root@cloudera-node-1 ~]# /etc/init.d/hadoop-0.20-datanode status datanode (pid 4866) is running... [root@cloudera-node-1 ~]# /etc/init.d/hadoop-0.20-datanode restart Stopping Hadoop datanode daemon (hadoop-datanode): stopping datanode datanode is stopped [ OK ] Starting Hadoop datanode daemon (hadoop-datanode): starting datanode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-cloudera-node-1.out datanode (pid 8570) is running... [ OK ] [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# hadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.12 put: Target hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.12 already exists [root@cloudera-node-1 ~]# hadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.123 [root@cloudera-node-1 ~]#
ok了。
如果报safemode了
2012-02-03 01:42:17,467 [logicalNode collector-19] INFO rolling.RollSink: closing RollSink 'escapedCustomDfs("hdfs://10.129.8.126/flume/","srcdata%{rolltag}" )' 2012-02-03 01:42:17,467 [logicalNode collector-19] INFO rolling.RollSink: opening RollSink 'escapedCustomDfs("hdfs://10.129.8.126/flume/","srcdata%{rolltag}" )' 2012-02-03 01:42:17,468 [logicalNode collector-19] INFO debug.InsistentOpenDecorator: Opened MaskDecorator on try 0 2012-02-03 01:42:17,469 [pool-7-thread-1] INFO hdfs.EscapedCustomDfsSink: Opening hdfs://10.129.8.126/flume/srcdata20120203-014217467+0800.2438842015436540.00000019 2012-02-03 01:42:17,476 [logicalNode collector-19] INFO debug.InsistentAppendDecorator: append attempt 3 failed, backoff (8000ms): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/flume/srcdata20120203-014217467+0800.2438842015436540.00000019.tmp. Name node is in safe mode. The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1. Safe mode will be turned off automatically. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1182) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1150) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:597) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:576) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
执行
hadoop dfsadmin -safemode leave
ok，再来一遍
125上面；
flume node_nowatch -n collector
126上面：
flume node_nowatch
ok搞定
[root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 2 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 0 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023.tmp [root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 2 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 [root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 2 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 [root@flume-hadoop-node-1 log]# hadoop fs -tail hdfs://10.129.8.126/flume/srcdata20120203-021531413+0800.2440835961446540.00000021 \t\t3881/udp\t\t\t# Data Acquisition and Control","timestamp":1328205987177,"pri":"INFO","nanos":100778763829457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000陋赂\u0010娄","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}} {"body":"msdts1\t\t3882/tcp\t\t\t# DTS Service Port","timestamp":1328205987177,"pri":"INFO","nanos":100778763863457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000?隆w?","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}} {"body":"msdts1\t\t3882/udp\t\t\t# DTS Service Port","timestamp":1328205987177,"pri":"INFO","nanos":100778763897457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u00005=?\u0002","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}}
新加flume node
126上面：
flume node_nowatch -n agentAB
flume-master页面上面添加配置
agentAB : text("/var/log/dmesg") | agentSink("10.129.8.125",35853);
OK，没有问题，下面试试默认配置
flume node_nowatch -n agentABC
agentABC : text("/tmp/medcl") | agentSink("10.129.8.125");
这个时候，
node status里面
agentABC agentABC flume-hadoop-node-1 OPENING Fri Feb 03 02:31:11 CST 2012 3 Fri Feb 03 02:32:49 CST 2012
console端报错：
2012-02-03 02:31:14,823 [logicalNode agentABC-22] INFO connector.DirectDriver: Connector logicalNode agentABC-22 exited with error: /tmp/medcl (No such file or directory) java.io.FileNotFoundException: /tmp/medcl (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98) at com.cloudera.flume.handlers.debug.TextFileSource.open(TextFileSource.java:75) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87) Exception in thread "logicalNode agentABC-22" java.lang.NullPointerException at com.cloudera.flume.handlers.debug.TextFileSource.close(TextFileSource.java:69) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.ensureClosed(DirectDriver.java:183) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.errorCleanup(DirectDriver.java:204) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:92)
创建文件
echo "hello world" > /tmp/medcl
继续失败着，不能自动恢复，只能重启node
[root@cloudera-node-1 log]# hadoop dfs -tail hdfs://10.129.8.126/flume/srcdata20120203-023644240+0800.2442108787815540.00000021 {"body":"hello world","timestamp":1328207806233,"pri":"INFO","nanos":2442110780978540,"host":"flume-hadoop-node-1","fields":{"AckTag":"20120203-023646196+0800.2442110743929540.00000022","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000\rJ\u0011?","rolltag":"20120203-023644240+0800.2442108787815540.00000021"}}
flume node_nowatch -n agentABCD agentABCD : text("/tmp/medcl") | agentSink("10.129.8.125");
text sink只能执行一次，后续文件有变化，并不处理
tail就可以实现监听
flume node_nowatch -n collector #如果collector已经关闭，需要重新打开，配置文件在前面 flume node_nowatch -n agentABCDE agentABCDE : tail("/tmp/medcl") | agentSink("10.129.8.125");
collector每30秒写一次hadoop，hadoop文件每次新建一个
[root@flume-hadoop-node-1 tmp]# echo "happy new year">>medcl [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 7 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 -rw-r--r-- 3 root supergroup 197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021 -rw-r--r-- 3 root supergroup 318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021 -rw-r--r-- 3 root supergroup 761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021 -rw-r--r-- 3 root supergroup 336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021 -rw-r--r-- 3 root supergroup 329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021 [root@flume-hadoop-node-1 tmp]# [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 8 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 -rw-r--r-- 3 root supergroup 197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021 -rw-r--r-- 3 root supergroup 318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021 -rw-r--r-- 3 root supergroup 761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021 -rw-r--r-- 3 root supergroup 336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021 -rw-r--r-- 3 root supergroup 329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021 -rw-r--r-- 3 root supergroup 337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021 [root@flume-hadoop-node-1 tmp]# hadoop fs -get hdfs://10.129.8.126/flume/srcdata20120207-190929343+0800.2847273890577540.00000021 /tmp/lo2
如果是替换文件内容，不是追加，第一条记录会造成丢失，此处应该特别注意（bug？)
[root@flume-hadoop-node-1 tmp]# echo "who is your daddy?">medcl [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 8 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 -rw-r--r-- 3 root supergroup 197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021 -rw-r--r-- 3 root supergroup 318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021 -rw-r--r-- 3 root supergroup 761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021 -rw-r--r-- 3 root supergroup 336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021 -rw-r--r-- 3 root supergroup 329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021 -rw-r--r-- 3 root supergroup 337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021
再追加一条数据
[root@flume-hadoop-node-1 tmp]# echo "here is a new line">>medcl [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 9 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 -rw-r--r-- 3 root supergroup 197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021 -rw-r--r-- 3 root supergroup 318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021 -rw-r--r-- 3 root supergroup 761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021 -rw-r--r-- 3 root supergroup 336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021 -rw-r--r-- 3 root supergroup 329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021 -rw-r--r-- 3 root supergroup 337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021 -rw-r--r-- 3 root supergroup 0 2012-02-07 19:19 /flume/srcdata20120207-191702865+0800.2847727413000540.00000021.tmp [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/ Found 9 items -rw-r--r-- 3 root supergroup 11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021 -rw-r--r-- 3 root supergroup 7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023 -rw-r--r-- 3 root supergroup 197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021 -rw-r--r-- 3 root supergroup 318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021 -rw-r--r-- 3 root supergroup 761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021 -rw-r--r-- 3 root supergroup 336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021 -rw-r--r-- 3 root supergroup 329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021 -rw-r--r-- 3 root supergroup 337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021 -rw-r--r-- 3 root supergroup 341 2012-02-07 19:19 /flume/srcdata20120207-191702865+0800.2847727413000540.00000021 [root@flume-hadoop-node-1 tmp]# hadoop fs -tail hdfs://10.129.8.126/flume/srcdata20120207-191702865+0800.2847727413000540.00000021 {"body":"here is a new line","timestamp":1328613446703,"pri":"INFO","nanos":2847751251273540,"host":"flume-hadoop-node-1","fields":{"AckTag":"20120207-191720960+0800.2847745508415540.00000025","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000/rN?","tailSrcFile":"medcl","rolltag":"20120207-191702865+0800.2847727413000540.00000021"}}
果然，数据丢了一条了。
ok，前面提到了flume使用3种工作模式来保证数据的可靠性与可用性：
1.End2End，2端确认，失败会自动重试（重试次数多少，重试失败之后怎样处理，还要继续研究）
agentE2ESink[("machine"[,port])]
2.DiskFailover，失败写本地磁盘，周期性检查，collector可用的时候，自动重做任务。
agentDFOSink[("machine"[,port])]
3.高效模式，collector失败就丢弃日志，够狠够绝
agentBESink[("machine"[,port])]
前面使用到的agentSink，是第一种End2End的别名，效果和End2End一样。
多收集器的配置
多个collector能够提高吞吐量，因为日志收集都是平行，前面提到过，为保证可靠性，如果collector挂了，agent需要写本地磁盘，然后周期性的去重新连接collector，另外，日志收集停止了，后面的日志处理与分析也歇菜了，这个可不行的。
多个collector就可以解决这个问题，汗！
另外多个collector中，如果其中一个挂了，agent应该是能够自动切换的，怎么配呢？
使用failover chains，
agentA : src | agentE2EChain("collectorA:35853","collectorB:35853"); agentB : src | agentE2EChain("collectorA:35853","collectorC:35853"); agentC : src | agentE2EChain("collectorB:35853","collectorA:35853"); agentD : src | agentE2EChain("collectorB:35853","collectorC:35853"); agentE : src | agentE2EChain("collectorC:35853","collectorA:35853"); agentF : src | agentE2EChain("collectorC:35853","collectorB:35853"); collectorA : collectorSource(35853) | collectorSink("hdfs://...","src"); collectorB : collectorSource(35853) | collectorSink("hdfs://...","src"); collectorC : collectorSource(35853) | collectorSink("hdfs://...","src");
如上配置，chain指定了2个，第一个collector失败了之后，自动切换使用第二个。
自动FailoverChain，主要是通过使用特殊的source和sink名字（多master下不适用）
source使用：
autoCollectorSource
sink使用：
autoE2EChain, autoDFOChain, or autoBEChain
配置为：
agentA : src | autoE2EChain ;
agentB : src | autoE2EChain ;
agentC : src | autoE2EChain ;
agentD : src | autoE2EChain ;
agentE : src | autoE2EChain ;
agentF : src | autoE2EChain ;
collectorA : autoCollectorSource | collectorSink("hdfs://...", "src");
collectorB : autoCollectorSource | collectorSink("hdfs://...", "src");
collectorC : autoCollectorSource | collectorSink("hdfs://...", "src");
Logical Configurations
一个physical node包含若干个logical node，logical node又分为：logical sources 和logical sinks ，使用flow来隔离nodes和分组
logical node允许一个JVM实例包含多个logical nodes，实现在一个JVM上跑多个Source和Sink的线程。
每个logical node的名称必须唯一，包括physical node 名称或者 host名称都不能相同
logical定义分两步，
1.定义node类型
agent1 : _source_ | autoBEChain ;
collector1 : autoCollectorSource | collectorSink("hdfs://....") ;
2.mapping logical node和 physical node
map host1 agent1
map host2 collector1
3.解除一个logical节点
decommission agent1
试试
125上 1004 cd /tmp/ 1005 ls 1006 rm -rif flume-* 1007 /etc/init.d/flume-master restart 1008 /etc/init.d/flume-node star
126上
/etc/init.d/flume-node star
flume master页面
config：
agent1 : tail("/tmp/medcl") | autoBEChain ; collector1 : autoCollectorSource | collectorSink("hdfs://10.129.8.126/flume/","medcl") ;
注：主机名-ip
cloudera-node-1：10.129.8.126
flume-hadoop-node-1：10.129.8.125
raw command:
command: map arguments:10.129.8.125 agent1 #flume-hadoop-node-1 agent1 command: map arguments: 10.129.8.126 collector1 #cloudera-node-1 collector1 试试解除 map 10.129.8.125 agent2 decommission agent2
（注意空格，decommission两端不能有空格）
或者unmap和map操作来移动logicalnode
unmap host2 collector1 map host3 collector1
（
抓包得到请求为：
curl -XPOST http://10.129.8.125:35871/mastersubmit.jsp -d'cmd=unmap&args=10.129.8.125+agent1'
）
注：logical sources和logical sinks在多master下不适用
通过logical source和logical sink可以在不知道具体物理节点的时候就进行流程的配置，flume有一种翻译的机制，会自动将logical节点名称替换成实际的主机名和端口
事实上，autoSinks和auto-Chain也是这样来实现的。
Flow 隔离，（注，多master下也不适用，悲催啊）
假设你需要收集一个物理机的多种数据，并存放到不同的地方，一种方式是对所有的数据打上tag，通过同一个管道来传数据，然后通过后处理来分离数据
另一种是在整个传输过程中通过将两两种数据隔离，避免后处理的产生
Flume两种都支持，并且延时很低，通过引入flow的概念，将节点进行分组，配置方式如下:
flume master页面：
raw commands
命令：config
参数：[logincal node] [flow name] fooSrc autoBEChain
实际例子：
config AgentC myflow tail("/tmp/medcl") autoBEChain config CollectorC myflow autoCollectorSource collectorSink("hdfs://10.129.8.126/flume/","medcl_flow") map 10.129.8.125 AgentC map 10.129.8.126 CollectorC
!!!!
------------
1.问题：
fail( "logical node not mapped to physical node yet" )
1.使用主机名来做map，node status显示的是什么名称，map的时候就用什么名称
2.先map好logical node，然后再更新config配置
正常工作的配置，
map cloudera-node-1 agent1 map flume-hadoop-node-1 collector1 agent1 : tail("/tmp/medcl") | agentSink("10.129.8.125",35853); collector1 : collectorSource(35853) | collectorSink("hdfs://10.129.8.126/flume/","medcl");
!!!!
多master配置
多master之间自动同步，一个master挂了，其下node会自动转移到其他master上去。
flume master有两种工作模式：standalone和distributed
如何配置呢？
<property> <name>flume.master.servers</name> <value>hostA,hostB</value> </property>
一个Host则是standalone模式，多个host即distributed模式【分布式模式下，每个master的配置文件必须一样】
另外，每个master必须要配置不同的serverid，如下：
MaserA： <property> <name>flume.master.serverid</name> <value>0</value> </property> MasterB： <property> <name>flume.master.serverid</name> <value>1</value> </property>
【数字和前面配置的服务器列表的下标保持一致即可】
分布式环境下，至少需要3台服务器来保证允许一台失败，如果要允许同时两台挂掉，则至少需要5台服务器
，如果master节点存活率不能超过总数的一半，整个flume master 集群就会block住，无法读写配置信息
flume master存放配置信息的地方叫做：configuration store，支持插拔，本身支持两种实现：
基于内存的：MBCS和基于ZooKeeper的：ZBCS
默认ZBCS，flume内置zookeeper，支持配置到现有的zookeeper集群去
<property> <name>flume.master.store</name> <value>zookeeper</value> </property>
【value值可选：zookeeper或者memory】
ZBCS配置
flume.master.zk.logdir：存储配置文件信息，更新日志，失败信息等
flume.master.zk.server.quorum.port：默认3182，zookeeper server本地监听
flume.master.zk.server.election.port：默认3183，zookeeper server用来寻找其它节点
flume.master.zk.client.port：默认3181，用来与zookeeper server通讯
FlumeMaster的gossip协议支持：
<property> <name>flume.master.gossip.port</name> <value>57890</value> </property>
分布式模型下，flume node的配置也需要调整，从连一个改成连接多个master
<property> <name>flume.master.servers</name> <value>masterA,masterB,masterC</value> </property>
flume node通过定期与master的端口做心跳检测，一旦master 连接失败，自动随机切换到剩下的可以连上的master上去。【master节点通过配置flume.master.heartbeat.port来配置心跳端口】
如果要使用外部的zookeeper，配置如下
conf/flume-site.xml.
<property> <name>flume.master.zk.use.external</name> <value>true</value> </property> <property> <name>flume.master.zk.servers</name> <value>zkServerA:2181,zkServerB:2181,zkServerC:2181</value> </property>
Flume与数据源集成
Flume强大就在于灵活，支持各种数据源，结构化的，非结构化的,半结构化等等
三种方式：
pushing、polling、embedding（嵌入flume组件到你的应用程序中）
Push Sources:
syslogTcp,syslogUdp:syslog,syslog-ng日志协议
scribe：scribe日志系统的协议
Polling：
tail,mulitail:监视文件内容的追加信息
exec：适合从现有系统抽取数据
poller：收集来着flume node本身的信息
Flume Event的数据模型
6个主要的字段;
Unix timestamp
Nanosecond timestamp 【纳秒级别的时间戳】
Priority
Source host
Body
Metadata table with an arbitrary number of attribute value pairs.
所有的event都有这几个字段，不过body长度可能为0，metadata表可能为空。
priority :TRACE, DEBUG, INFO, WARN, ERROR, or FATAL，这几种
body：raw格式，默认最大32KB，多余的截掉，通过参数flume.event.max.size.bytes来进行配置
使用event的字段来自定义输出位置
collectorSink("hdfs://namenode/flume/webdata/%H00/", "%{host}-")
%H 为时间timestamp字段里的小时，host为field里面的主机名
快速参考：
[horizontal] %{host}
host
%{nanos}
nanos
%{priority}
priority string
%{body}
body
%%
a % character.
%t
Unix time in millis
时间比较特殊，直接使用，不需要{}
collectorSink("hdfs://namenode/flume/webdata/%Y-%m-%d/%H00/", "web-")
快速参考：
%a
locale’s short weekday name (Mon, Tue, …)
%A
locale’s full weekday name (Monday, Tuesday, …)
%b
locale’s short month name (Jan, Feb,…)
%B
locale’s long month name (January, February,…)
%c
locale’s date and time (Thu Mar 3 23:05:25 2005)
%d
day of month (01)
%D
date; same as %m/%d/%y
%H
hour (00..23)
%I
hour (01..12)
%j
day of year (001..366)
%k
hour ( 0..23)
%l
hour ( 1..12)
%m
month (01..12)
%M
minute (00..59)
%P
locale’s equivalent of am or pm
%s
seconds since 1970-01-01 00:00:00 UTC
%S
second (00..60)
%y
last two digits of year (00..99)
%Y
year (2010)
%z
+hhmm numeric timezone (for example, -0400)
输出文件格式
两种方式：
一直是在 flume-site.xml里面设置默认值，另外是由特定的sink来决定
1.flume-site.xml
flume.collector.output.format
格式快速参考
avro
Avro Native file format. Default currently is uncompressed.
avrodata
Binary encoded data written in the avro binary format.
avrojson
JSON encoded data generated by avro.
default
a debugging format.
json
JSON encoded data.
log4j
a log4j pattern similar to that used by CDH output pattern.
raw
Event body only. This is most similar to copying a file but does not preserve any uniqifying metadata like host/timestamp/nanos.
syslog
a syslog like text output format.
seqfile
the binary hadoop Sequence file format with WritableEventKeys keys, and WritableEvent as values.
2.分别配置
collectorSink( "dfsdir","prefix"[, rollmillis[, format]]) text("file"[,format]) formatDfs("hdfs://nn/file" [, format]) escapedFormatDfs("hdfs://nn/file" [, format])
压缩seqfile
formatDfs("hdfs://nn/dir/file", seqfile("bzip2"))
HDFS大量小文件与高延迟的处理
Flume两种策略来处理
1.合并小文件到大的文件
2.使用CombinedFileInputFormat
<property> <name>flume.collector.dfs.compress.codec</name> <value>None</value> <description>Writes formatted data compressed in specified codec to dfs. Value is None, GzipCodec, DefaultCodec (deflate), BZip2Codec, or any other Codec Hadoop is aware of </description> </property>
seqfile和avrodata支持内部的压缩，具体再研究
DataFlow定义语言
Fan out，往所有sinks写：
[ console, collectorSink ]
Fail over，当前失败，转移到下一个，尝试候选sink：
< logicalSink("collector1") ? logicalSink("collector2") >
配置样例：
agent1 : source | < logicalSink("collector1") ? logicalSink("collector2") > ;
Roll sink，每隔一段时间，关闭当前实例，创建新的实例，每次会创建新的独立的文件：
roll(millis) sink
配置样例：
roll(1000) [ console, escapedCustomDfs("hdfs://namenode/flume/file-%{rolltag}") ]
Sink Decorators，sink装饰器
Fan out和Failover影响messages去哪里，但不修改数据，如果要过滤数据什么的，使用sink decorator
sink decorator可以做很多事情，如可以给数据流添加属性，可以通过写ahead 日志来确保可靠性，或者通过批量、压缩来提供网络吞吐，抽样甚至轻量级的分析
flumenode: source | intervalSampler(10) sink;
flumenode: source | batch(100) sink;
flumenode: source | batch(100) gzip sink;
collector(15000) { escapedCustomDfs("xxx","yyy-%{rolltag}") }
collector(15000) { [ escapedCustomDfs("xxx","yyy-%{rolltag}"), hbase("aaa", "bbb-%{rolltag}"), elasticSearch("eeee","ffff") ] } 【同时往3个sink里面写数据，可能有些是持久化的，有些是瞬时的，都成功之后，才会确认成功】
node1 : tail("foo") | ackedWriteAhead batch(100) gzip lazyOpen stubbornAppend logicalSink("bar");【write ahead，批量100，gzip压缩】
Metadata支持正则来进行抽取
支持类似select语法来筛选
thriftSink and thriftSource
扩展与插件
http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_semantics_of_flume_extensions
附录真是好啊
http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_flume_source_catalog
map cloudera-node-1 agent2 agent2 : syslogTcp(2012) | agentSink("10.129.8.125",35853); flume node_nowatch -n medcl agent2 : syslogTcp(2012) | agentSink("10.129.8.125",35853);
测试syslog信息
1.NC连接 nc 10.129.8.126 2012 2.输入syslog消息（遵照格式：http://blog.csdn.net/xcj0535/article/details/4158624） <165>Aug 24 05:34:00 CST 1987 mymachine myproc[10]: %% It's time to make the do-nuts. %% Ingredients: Mix=OK, Jelly=OK # Devices: Mixer=OK, Jelly_Injector=OK, Frier=OK # Transport: Conveyer1=OK, Conveyer2=OK # %% <1> medcl is back syslog的格式 下面是一个syslog消息： <30>Oct 9 22:33:20 hlfedora auditd[1787]: The audit daemon is exiting. 其中“<30>”是PRI部分，“Oct 9 22:33:20 hlfedora”是HEADER部分，“auditd[1787]: The audit daemon is exiting.”是MSG部分。 [root@cloudera-node-1 ~]# hadoop fs -cat /flume/medcl20120209-221925655+0800.3031470203471540.00000026 {"body":"medcl is back","timestamp":1328797314800,"pri":"INFO","nanos":692106386851457,"host":"cloudera-node-1","fields":{"AckTag":"20120209-222148285+0800.692099872659457.00000037","syslogfacility":"\u0001","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000qu锚茫","syslogseverity":"\u0003","rolltag":"20120209-221925655+0800.3031470203471540.00000026"}} upload到HDFS的文件包含了太多内容 raw下 collector2 : syslogTcp( 2013) | collectorSink( "hdfs://10.129.8.126/flume/", "medcl_raw",3000,raw ); C:\Windows\system32>nc 10.129.8.125 2013 <1> i will be back <1> i will be back2 <1> i will be back3 <1> i will be back4 [root@cloudera-node-1 ~]# hadoop fs -cat /flume/medcl_raw20120209-235000888+0800.3036905435701540.00000069 i will be back i will be back2 i will be back3 i will be back4
.NET Agent 25个线程，结果压趴下了[另外后续测试发现经常无原因socket断开，服务端socket直接挂掉，flume显示error]。
2012-02-10 21:29:44,154 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode collector2-20 in error state SyslogTcpSourceThreads | Collector because null
syslogTcp不稳定，果断换thriftRpc作为Source，经测果然很稳定
thrift-0.6.0.exe -r -gen csharp flume.thrift 2012-02-13 23:36:30,574 [pool-4-thread-1] ERROR server.TSaneThreadPoolServer: Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client? at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213) at com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:224) at org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
此异常可能是因为服务端和客户端使用了不相同的transport，如framed和buffered不匹配
collector3 : thriftSource( 2014 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw ); collector4 : thriftSource( 2015 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw ); collector5 : thriftSource( 2016 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw ); collector6 : thriftSource( 2017 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw ); collector7 : thriftSource( 2018 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift2",30000); map cloudera-node-1 collector7
vi flume-site.xml,添加压缩和默认roll时间
<property> <name>flume.collector.dfs.compress.gzip</name> <value>true</value> <description>Writes compressed output in gzip format to dfs. value is boolean type, i.e. true/false</description> </property> <property> <name>flume.collector.roll.millis</name> <value>60000</value> <description>The time (in milliseconds) between when hdfs files are closed and a new file is opened (rolled). </description> </property>
测试文件模板
collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/app/%{host}/%Y-%m-%d/", "%H%M%S-test1-%t",5000); map cloudera-node-1 collector8 [root@flume-hadoop-node-1 ~]# hadoop fs -lsr hdfs://10.129.8.126/flume/app drwxr-xr-x - flume supergroup 0 2012-02-17 00:49 /flume/app/MEDCL-THINK drwxr-xr-x - flume supergroup 0 2012-02-17 00:49 /flume/app/MEDCL-THINK/4113221-02-12 -rw-r--r-- 1 flume supergroup 219 2012-02-17 00:49 /flume/app/MEDCL-THINK/4113221-02-12/203942-test1-12973855419598268720120217-004946767+0800.1305778353827457.00006891
更新
collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/%{catalog}/2012-%m/%d/", "%a-%{host}-",5000,raw());
结果：
/flume/FileTemplateRaw/2012-11/19/Fri-MEDCL-THINK-20120217-013005302+0800.1308196889416457.00007109
collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/%{catalog}/2012", "",5000,raw());