Flume传数据到Hadoop双namenode集群传数据可能出现的问题
现在的需求是在一台Flume采集机器上，往Hadoop集群上写HDFS，该机器没有安装Hadoop。
这里的Flume版本是1.6.0，Hadoop版本是2.7.1.
把Hadoop集群的hdfs-site.xml、core-site.xml两个配置文件复制到 flume安装目录的conf目录去，把hadoop-hdfs-2.7.1.jar复制到 Flume lib目录。
一、Flume配置文件：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = syslogtcp
a1.sources.r1.bind =
192.168
.
110.160
# 本机ip
a1.sources.r1.port =
23003
a1.sources.r1.workerThreads =
10
a1.channels.c1.type = memory
a1.channels.c1.capacity =
1000000
a1.channels.c1.transactionCapacity =
100000
a1.channels.c1.keep-alive =
6
a1.channels.c1.byteCapacityBufferPercentage =
20
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs:
//clusterpc/test/flume/%y-%m-%d
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round =
true
a1.sinks.k1.hdfs.roundValue =
10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.useLocalTimeStamp=
true
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
|
启动： bin/flume-ng agent --conf conf --conf-file conf/flume-tcp-memory-hdfs.conf --name a1 -Dflume.root.logger=info,console
二、错误集：
1、 找不到主机名
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
2016
-
09
-
19
16
:
15
:
48
,
518
(SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:
459
)] process failed
java.lang.IllegalArgumentException: java.net.UnknownHostException: cluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:
378
)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:
310
)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:
176
)
at org.apache.hadoop.hdfs.DFSClient.
678
)
at org.apache.hadoop.hdfs.DFSClient.
619
)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:
149
)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:
2653
)
at org.apache.hadoop.fs.FileSystem.access$
200
(FileSystem.java:
92
)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:
2687
)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:
2669
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
371
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
170
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
355
)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:
295
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
243
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
235
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
$
1
.run(BucketWriter.java:
679
)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:
50
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
.call(BucketWriter.java:
676
)
at java.util.concurrent.FutureTask.run(FutureTask.java:
262
)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1145
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615
)
at java.lang.Thread.run(Thread.java:
744
)
Caused by: java.net.UnknownHostException: cluster
|
cluster是公司Hadoop集群NameService的名字，这个错误是由于找不到Hadoop集群NameService造成的，所以需要把hdfs-site.xml复制到flume/conf目录。
2、
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
java.io.IOException: Mkdirs failed to create /test/flume/
16
-
09
-
19
(exists=
false
, cwd=file:/data/apache-flume-
1.6
.
0
-bin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:
450
)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:
435
)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
909
)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
890
)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
787
)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
776
)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:
96
)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:
78
)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:
69
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
246
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
235
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
$
1
.run(BucketWriter.java:
679
)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:
50
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
.call(BucketWriter.java:
676
)
at java.util.concurrent.FutureTask.run(FutureTask.java:
262
)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1145
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615
)
at java.lang.Thread.run(Thread.java:
744
)
|
把 core-site.xml复制到flume/conf目录
3、
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
java.io.IOException: No FileSystem
for
scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:
2644
)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:
2651
)
at org.apache.hadoop.fs.FileSystem.access$
200
(FileSystem.java:
92
)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:
2687
)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:
2669
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
371
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
170
)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:
355
)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:
295
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
243
)
at org.apache.flume.sink.hdfs.BucketWriter$
1
.call(BucketWriter.java:
235
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
$
1
.run(BucketWriter.java:
679
)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:
50
)
at org.apache.flume.sink.hdfs.BucketWriter$
9
.call(BucketWriter.java:
676
)
at java.util.concurrent.FutureTask.run(FutureTask.java:
262
)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1145
)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615
)
at java.lang.Thread.run(Thread.java:
744
)
|
把hadoop-hdfs-2.7.1.jar复制到flume/lib目录下
4、HDFS权限不足，这里往HDFS写文件的用户是登录Flume采集机器的用户。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
org.apache.hadoop.security.AccessControlException: Permission denied: user=kafka, access=WRITE, inode=
"/test/flume/16-09-19/events-.1474268726127.tmp"
:hadoop:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:
319
)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:
292
)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:
213
)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:
190
)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:
1698
)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:
1682
)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:
1665
)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:
2517
)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:
2452
)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:
2335
)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:
623
)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:
397
)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$
2
.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:
616
)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:
969
)
at org.apache.hadoop.ipc.Server$Handler$
1
.run(Server.java:
2049
)
at org.apache.hadoop.ipc.Server$Handler$
1
.run(Server.java:
2045
)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:
415
)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1657
)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:
2043
)
|
HDFS 权限不足，要授权。hadoop fs -chmod -R 777 /test/
5、时间戳
|
1
2
3
4
5
6
7
8
|
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was
null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:
204
)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:
228
)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:
432
)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:
380
)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:
68
)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:
147
)
at java.lang.Thread.run(Thread.java:
744
)
|
原因是Event对象headers没有设置timestamp造成的，解决办法：设置a1.sinks.k1.hdfs.useLocalTimeStamp=true，使用本地时间戳。