flume整合spark实现监控目录下的数据

一、需求背景

​ 在做某项目时，遇到一个需求是这样的：每天产生的预演数据会存放在hdfs中某个目录，文件名假设为preview20200723，这个文件在当天可能会一直有数据在追加（间断性），也可能一次性写完（持续性），需要利用现有的技术监控这个目录中数据的变化，将获取到的json数据做解析再保留到数仓中（此部分为Spark编辑部分，本文不做测试）。

二、技术选型

​ flume + spark streaming，后期可以再添加kafka做个缓存机制，实现高可用性。

三、实现步骤

注：本文所做的代码实现，仅仅是测试，只实现整体的思路。实际应用可以根据需要修改配置和部分代码。

1、poll方式

1.1、安装flume

下载链接http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz

本文使用的是最新版本1.9（但其实已经有一年没有更新版本了）。

注：如果使用1.9以下版本，就需要在lib目录中添加scala-library-2.11.12.jar。

1.2、配置flume文件

flume-poll-spark.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1

#source
a1.sources.r1.channels = c1
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /root/logs
a1.sources.r1.fileHeader = true
#尝试使用端口发送信息来测试流程，不过需要另开一个命令窗口，打开44444端口
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444

#channel
a1.channels.c1.type =memory
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity=5000

#sinks
a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
a1.sinks.k1.hostname=localhost
a1.sinks.k1.port = 8888
a1.sinks.k1.batchSize= 2000  

值得注意的是，这里的sink是根据org.apache.spark.streaming.flume.sink.SparkSink这个类来创建的，而这个类是需要导入spark-sink的包的，本文使用的是spark-streaming-flume-sink_2.11-2.0.2.jar（一开始我导的时候没有看清楚是有加sink的，所以一直很纳闷为什么会报连接不上地址的错误，请各位小伙伴导包要看仔细）。

将下载好的jar包放在flume根目录下面的lib目录中，在flume启动的时候会去寻找SparkSink类，然后flume会自行创建，并根据配置文件传入hostname和port。有兴趣的可以看看spark-streaming-flume-sink_2.11-2.0.2.jar中的SparkSink源码。

1.3、启动flume

bin/flume-ng agent -n a1 -c conf -f conf/flume-poll-spark.conf -Dflume.root.logger=INFO,console

在windows上启动，需要将-Dflume.root.logger=INFO,console去掉，并修改斜杠

1.4、导入依赖

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0modelVersion>

    <groupId>com.stevengroupId>
    <artifactId>spark-demoartifactId>
    <version>1.0-SNAPSHOTversion>
    <properties>
        <scala.version>2.11.8scala.version>
        <hadoop.version>2.7.4hadoop.version>
        <spark.version>2.0.2spark.version>
    properties>
    <dependencies>
        
        <dependency>
            <groupId>org.scala-langgroupId>
            <artifactId>scala-libraryartifactId>
            <version>${scala.version}version>
        dependency>
        
        <dependency>
            <groupId>org.apache.sparkgroupId>
            <artifactId>spark-core_2.11artifactId>
            <version>${spark.version}version>
        dependency>
        
        <dependency>
            <groupId>org.apache.hadoopgroupId>
            <artifactId>hadoop-clientartifactId>
            <version>${hadoop.version}version>
        dependency>
        
        <dependency>
            <groupId>mysqlgroupId>
            <artifactId>mysql-connector-javaartifactId>
            <version>5.1.41version>
        dependency>
        
        <dependency>
            <groupId>org.apache.sparkgroupId>
            <artifactId>spark-sql_2.11artifactId>
            <version>${spark.version}version>
        dependency>

        <dependency>
            <groupId>org.apache.sparkgroupId>
            <artifactId>spark-hive_2.11artifactId>
            <version>${spark.version}version>
        dependency>
        <dependency>
            <groupId>org.apache.sparkgroupId>
            <artifactId>spark-streaming_2.11artifactId>
            <version>${spark.version}version>
        dependency>

        
        <dependency>
            <groupId>org.apache.sparkgroupId>
            <artifactId>spark-streaming-flume_2.11artifactId>
            <version>${spark.version}version>
        dependency>



    dependencies>
    <build>
        
        <sourceDirectory>src/main/scalasourceDirectory>
        
        <testSourceDirectory>src/test/scalatestSourceDirectory>
        <plugins>
            
            <plugin>
                <groupId>net.alchim31.mavengroupId>
                <artifactId>scala-maven-pluginartifactId>
                <version>3.2.0version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compilegoal>
                            <goal>testCompilegoal>
                        goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfilearg>
                                <arg>${project.build.directory}/.scala_dependenciesarg>
                            args>
                        configuration>
                    execution>
                executions>
            plugin>
            
            <plugin>
                <groupId>org.apache.maven.pluginsgroupId>
                <artifactId>maven-shade-pluginartifactId>
                <version>2.3version>
                <executions>
                    <execution>
                        <phase>packagephase>
                        <goals>
                            <goal>shadegoal>
                        goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SFexclude>
                                        <exclude>META-INF/*.DSAexclude>
                                        <exclude>META-INF/*.RSAexclude>
                                    excludes>
                                filter>
                            filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>mainClass>
                                transformer>
                            transformers>
                        configuration>
                    execution>
                executions>
            plugin>
        plugins>
    build>


project>

1.5、代码实现

package com.steven.spark.streaming

import java.net.InetSocketAddress

import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.dstream.{DStream, InputDStream, ReceiverInputDStream}
import org.apache.spark.streaming.flume.{FlumeUtils, SparkFlumeEvent}
import org.apache.spark.streaming.{Durations, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
/**
  * author:seven lin
  * date:2020/6/1422:56
  * description:
  **/
object ListenFile {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("sparkstreamingfile").setMaster("local[2]")
    val sc = new SparkContext(conf)
    sc.setLogLevel("WARN")
    //创建一个streamingcontext对象，并设置批次间隔时间
    val ssc = new StreamingContext(sc, Durations.seconds(5))
    //设置监听的地址
    val address = Seq(new InetSocketAddress("192.168.25.161", 8888))
    //获取flume中数据
    val stream: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createPollingStream(ssc,address,StorageLevel.MEMORY_AND_DISK)
    //从Dstream中获取flume中的数据  {"header":xxxxx   "body":xxxxxx}
    val lineDstream: DStream[String] = stream.map(x => new String(x.event.getBody.array()))
    //打印内容
    lineDstream.print()
    
    ssc.start()
    ssc.awaitTermination()
  }
}

1.6、启动spark

这里只做测试，没有修改配置，就默认配置。默认的driver和executor内存大小为1G。

spark-submit --class com.steven.spark.streaming.ListenFile spark-demo-1.0-SNAPSHOT.jar

1.7、结果展示

6.txt内容

nihaoma 

helloworld

spark is niubi

将6.txt直接放到

-------------------------------------------
Time: 1595083560000 ms
-------------------------------------------
nihaoma 

helloworld

spark is niubi

-------------------------------------------
Time: 1595083570000 ms
-------------------------------------------

pull方式要先启动flume再启动spark，push方式则相反。

2、push方式

2.1、安装flume

同1.1

2.2、配置flume文件

flume-push-spark.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1

#source
a1.sources.r1.channels = c1
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /root/logs
a1.sources.r1.fileHeader = true

#channel
a1.channels.c1.type =memory
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity=1000

#sinks
a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname=192.168.25.161
a1.sinks.k1.port = 8888
a1.sinks.k1.batchSize= 1000

2.3、导入依赖

同1.4

2.4、代码实现

将获取数据的方法修改如下：

//接收flume的数据
    val stream: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createStream(ssc,"192.168.25.161",8888,StorageLevel.MEMORY_AND_DISK)

2.5、启动spark

同1.6

2.6、启动flume

bin/flume-ng agent -n a1 -c conf -f conf/flume-push-spark.conf -Dflume.root.logger=INFO,console

2.7、结果展示

同1.7

总结

1、使用过程中出现过以下问题：

2020-07-18 20:12:05,701 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2020-07-18 20:12:09,702 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.

原因是配置的channel容量太小，以至于我将一个较大文件直接存放的话，容量占满，而sink端原先我是没有设置批处理量的，导致消费速度跟不上。调整batchsize为1000之后解决。

2、注意poll方式的时候，flume创建的sink类型为sparksink，所以要导入相应的jar包，否则它启动之后创建不了。

3、在使用之前，查看一下端口是否有被占用。查看命令ss -lntpd | grep :8888。