Spark学习八：spark streaming与flume和kafka集成
Spark学习八：spark streaming与flume和kafka集成
标签（空格分隔）： Spark
- Spark学习八spark streaming与flume和kafka集成
- 一Kafka
- 二flume和kafka的集成
- 三kafka和spark streaming的集成方式一kafka推送
- 四kafka和spark streaming的集成方式一spark streaam主动获取
- 五spark stream的高级应用updateStateByKey实现累加功能
- 六spark stream的高级应用窗口函数
一，Kafka
1，概述
2,kafka的安装
解压安装文件
修改server.properties文件：
############################# Log Basics #############################
# A comma seperated list of directories under which to store log files
log.dirs=/opt/app/kafka_2.10-0.8.2.1/ka-logs
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=study.com.cn:2181启动zookeeper:
bin/zkServer.sh start启动kafka:
bin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh -daemon config/server.properties //运行在后台创建topic
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicTest
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181推送信息(生产者)：
bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicTest
eeeeeeeeeeeeeeeeeeeeeeeeeeee接受信息（消费者）：
bin/kafka-console-consumer.sh --zookeeper study.com.cn:2181 --topic topicTest --from-beginning二，flume和kafka的集成
创建topicFlume:
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicFlume
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181修改flume-kafka.xml的配置：
agent002.sources = sources002
agent002.channels = channels002
agent002.sinks = sinks002
## define sources
agent002.sources.sources002.type = exec
agent002.sources.sources002.command = tail -F /opt/app/apache-flume-1.5.0-bin/monitor/log.input
## define channels
agent002.channels.channels002.type = memory
agent002.channels.channels002.capacity = 10000
agent002.channels.channels002.transactionCapacity = 10000
agent002.channels.channels002.byteCapacityBufferPercentage = 20
agent002.channels.channels002.byteCapacity = 800000
##define sinks
agent002.sinks.sinks002.type =org.apache.flume.sink.kafka.KafkaSink
agent002.sinks.sinks002.brokerList=study.com.cn:9092
agent002.sinks.sinks002.topic=topicFlume
##relationship
agent002.sources.sources002.channels = channels002
agent002.sinks.sinks002.channel = channels002启动flume:
bin/flume-ng agent --conf conf --name agent002 --conf-file conf/flume-kafka001.properties -Dflume.root.logger=INFO,console
启动kafka消费者：
bin/kafka-console-consumer.sh --zookeeper study.com.cn:2181 --topic topicFlume --from-beginning测试：
echo "ddddddddddddddddddddd" >>log.input三，kafka和spark streaming的集成方式一（kafka推送）
创建topicFlume:
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181准备需要的jar包：
启动spark本地应用：
bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]spark源码:
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
val ssc = new StreamingContext(sc, Seconds(30))
val topicMap = Map("topicSpark" -> 1)
val kafkaStream = KafkaUtils.createStream(ssc, "study.com.cn:2181", "topicGroup", topicMap).map(_._2)
val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
wordCountDStream.print()
ssc.start()
ssc.awaitTermination()测试:
bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop四，kafka和spark streaming的集成方式一（spark streaam主动获取）
创建topicFlume:
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181准备需要的jar包：
启动spark本地应用：
bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]spark源码:
import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
val ssc = new StreamingContext(sc, Seconds(30))
// kafkaParams: Map[String, String]
val kafkaParams = Map("metadata.broker.list" -> "study.com.cn:9092")
// topics: Set[String]
val topics = Set("topicSpark")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)
val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
wordCountDStream.print()
ssc.start()
ssc.awaitTermination()测试:
bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop五，spark stream的高级应用updateStateByKey实现累加功能
创建topicFlume:
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181准备需要的jar包：
启动spark本地应用：
bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]spark源码:
import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
val ssc = new StreamingContext(sc, Seconds(30))
ssc.checkpoint(".")
val kafkaParams = Map("metadata.broker.list" -> "study.com.cn:9092")
// topics: Set[String]
val topics = Set("topicSpark")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)
val wordDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1))
// definition ,Option[Int]
// SessionInfo: sessionId, ip, count
val updateFunc = (values: Seq[Int], state: Option[Int]) => {
val currentCount = values.sum
val previousCount = state.getOrElse(0)
Some(currentCount + previousCount)
}
// wordDStream[(K, V)]
val wordCountDStream = wordDStream.updateStateByKey(updateFunc)
wordCountDStream.print()
ssc.start()
ssc.awaitTermination()测试:
bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop六，spark stream的高级应用窗口函数
创建topicFlume:
bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark
bin/kafka-topics.sh --list --zookeeper study.com.cn:2181准备需要的jar包：
启动spark本地应用：
bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]spark源码:
import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
val ssc = new StreamingContext(sc, Seconds(5))
// kafkaParams: Map[String, String]
val kafkaParams = Map("metadata.broker.list" -> "bigdata-senior01.ibeifeng.com:9092")
// topics: Set[String]
val topics = Set("sparkTopic")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)
// val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
// 每隔十五秒，统计最近五分钟之内的情况
val wordDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1))
//
val reduceFunc = (v1: Int, v1: Int) =>{
v1 + v2
}
val wordCountDStream = wordDStream.reduceByKeyAndWindow(reduceFunc, Seconds(5 * 12 * 5), Seconds(3 * 5))
wordCountDStream.print()
ssc.start()
ssc.awaitTermination()