Flume+kafka+Spark Steaming demo2

阅读更多

一，flume配置
# Name the components on this agent
a1.sources = tailsource-1 
a1.sinks = remotesink 
a1.channels = memoryChnanel-1 

# Describe/configure the source 
a1.sources.tailsource-1.type = exec 
a1.sources.tailsource-1.command = tail -F /var/log/test/raw_data.txt

a1.sources.tailsource-1.channels = memoryChnanel-1 

# Describe the sink 
a1.sinks.k1.type = logger 

# Use a channel which buffers events in memory 
a1.channels.memoryChnanel-1.type = memory 
a1.channels.memoryChnanel-1.keep-alive = 10 
a1.channels.memoryChnanel-1.capacity = 100000 
a1.channels.memoryChnanel-1.transactionCapacity = 100000 

# Bind the source and sink to the channel 
a1.sinks.remotesink.type = avro 
a1.sinks.remotesink.hostname = 172.18.203.137
a1.sinks.remotesink.port = 9999
a1.sinks.remotesink.channel = memoryChnanel-1


#agent section   
producer.sources = s   
producer.channels = c   
producer.sinks = r   

#source section   
producer.sources.s.type = avro 
producer.sources.s.bind = 172.18.203.137
producer.sources.s.port = 9999 

producer.sources.s.channels = c   

# Each sink's type must be defined   
producer.sinks.r.type = org.apache.flume.sink.kafka.KafkaSink 
producer.sinks.r.topic = mytopic 
producer.sinks.r.brokerList = master1:9092,master2:9092,slave2:9092 
producer.sinks.r.requiredAcks = 1 
producer.sinks.r.batchSize = 20 
producer.sinks.r.channel = c1            

#Specify the channel the sink should use   
producer.sinks.r.channel = c   

# Each channel's type is defined.   
producer.channels.c.type   = org.apache.flume.channel.kafka.KafkaChannel 
producer.channels.c.capacity = 10000 
producer.channels.c.transactionCapacity = 1000 
producer.channels.c.brokerList=master1:9092,master2:9092,slave2:9092 
producer.channels.c.topic=channel1 
producer.channels.c.zookeeperConnect=master2:2181,slave2:2181,slave4:2181

二， Spark代码

import kafka.serializer.StringDecoder
import org.apache.log4j.{Level, Logger}
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}

/**
  * Author: david
  * Date  : 3/7/17
  */
object StreamingDataTest {

  def main(args: Array[String]): Unit = {

    Logger.getLogger("org.apache.spark").setLevel(Level.WARN);
    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.ERROR);

    val conf = new SparkConf().setAppName("StreamingDataTest").setMaster("local[4]")

    val sc = new SparkContext(conf)

    val ssc = new StreamingContext(sc, Seconds(1))

    // Kafka的topic
    val topics = Set("mytopic")

    //kafka brokers列表
    val brokers = "master1:9092,master2:9092,slave3:9092"

    //kafka查询参数
    val kafkaParams = Map[String, String](
      "metadata.broker.list" -> brokers, "serializer.class" -> "kafka.serializer.StringEncoder")

    //创建direct stream
    val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)

    //kafkaStream这个ｔｕｐｌｅ的第二部分为接收kafka　topic里的文本流
    val rawDStream = kafkaStream.flatMap(_._2.split("\\s+")).map((_, 1))

    val resDStream = rawDStream.reduceByKeyAndWindow(
      (v1: Int, v2: Int) => {
        v1 + v2
      },
      Seconds(8),
      Seconds(4));

    resDStream.print();

    ssc.start()
    ssc.awaitTermination()
  }

}


三，注意事项
查看/var/log/flume-ng下面的日志报错信息
avro端口号绑定大于公共端口1024
注意linux防火墙service iptables stop
注意运行scala依赖的scope为 provided编译可以，但本机运行找不到class