SparkFlumeEvent:spark streaming连接flume,从SparkFlumeEvent中获取记录内容

JavaReceiverInputDStream<SparkFlumeEvent> flumeStream =  FlumeUtils.createStream(jssc, args[0], Integer.parseInt(args[1]));
	    
		JavaDStream<String> words = flumeStream
				.flatMap(new FlatMapFunction<SparkFlumeEvent, String>() {
					@Override
					public Iterable<String> call(SparkFlumeEvent t)
							throws Exception {
						System.out.println("t.event():"+ t.event());
						System.out.println(new String(t.event().getBody().array()));
						FlumeVo vo = (FlumeVo) JSON.parseObject(t.event().toString(),FlumeVo.class);
						System.out.println(vo.getBody().getBytes());						
						return Arrays.asList(t.event().toString().split(" "));
					}
				});
		words.print();


spark streaming连接flume,得到的是SparkFlumeEvent对象,但是怎么能从SparkFlumeEvent对象里得到flume传过来的数据呢,开始我各种get函数去试,但是都没有get出来,t.event()输出的内容是:

{"headers": {}, "body": {"bytes": "[10.4.1.24:nginx]"UfaNode" "123.134.176.33" "10.4.2.30:31001" "[24/Jan/2015:13:53:54 +0800]" "0.034" "0.006" "-" "200" "-" "65536" "190" "xiaomi" "appback" "-" "POST /xiaomi/upload_block_chunk?chunk_pos=524288&body_sum=-39296692&upload_id=066af4a9_7mcrh73iSzu%28kzfGK%28p9Xho3qxtK58xo0lUldabQt1RuCMczzYM5YupnOVWFy133D0g%2959%29T5zWwUnrpyAhoNQ-0_2 HTTP/1.1" "MI 2S/MI 2S; MIUI/4.12.5 V5" "101.71.27.24""}}

结果我就建了一个对象FlumeVo来解析这个字符串,但是放到线上环境后又出问题了,因为flume传过来的真实数据中是有引号的,所以JSON解析异常,没办法又得重新尝试,在百度查了半天没结果,还得靠强大的google,给大家推荐一个网址:http://www.fqgoogle.com/,解决办法就是上面的:

new String(t.event().getBody().array())

你可能感兴趣的:(SparkFlumeEvent:spark streaming连接flume,从SparkFlumeEvent中获取记录内容)