Spark写ES的遇到的坑

        完整的项目：https://github.com/JeemyJohn/SparkToEs.git


        由于项目需要，最近搞Spark Streaming做数据分析，并最终将分析完的数据写入ElasticSearch。我们项目使用的是Spark 2.1.0，而我们公司的ElasticSearch版本使用的是2.1.2。项目过程中遇到了很多坑，浪费了不少时间，故此在这里总结一下，希望大家以后免于趟这些坑以节约时间做一些更有意义的事。由于Maven依赖很长，故此放在最后，请参看： 3. 添加Maven依赖

1. RDD操作中使用自定义类型

/** * Created by zhanghuayan on 2017/4/6. */
class MyValue {

  var hostIp = ""
  var remoteIp = ""

  override def toString: String = {
    "(" + this.hostIp + " " + this.remoteIp + ")"
  }
}

        出现“org.apache.spark.SparkException: Task not serializable”这个错误，一般是因为在map、filter等的参数使用了外部的变量，但是这个变量不能序列化（不是说不可以引用外部变量，只是要做好序列化工作，具体后面详述）。其中最普遍的情形是：当引用了某个类（经常是当前类）的成员函数或变量时，会导致这个类的所有成员（整个类）都需要支持序列化。虽然许多情形下，当前类使用了“extends Serializable”声明支持序列化，但是由于某些字段不支持序列化，仍然会导致整个类序列化时出现问题，最终导致出现Task未序列化问题。所以在Spark中定义类应该加上extends Serializable，如下所示：

/** * Created by zhanghuayan on 2017/4/6. */
class MyValue extends Serializable {

  var hostIp = ""
  var remoteIp = ""

  override def toString: String = {
    "(" + this.hostIp + " " + this.remoteIp + ")"
  }
}

2. 写入ES

        首先，Spark写ElasticSearch需要使用使用 elasticsearch-hadoop 依赖：

      <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
      <dependency>
          <groupId>org.elasticsearch</groupId>
          <artifactId>elasticsearch-hadoop</artifactId>
          <version>5.0.0</version>
      </dependency>

       此处version选择5.0.0的原因是因为Spark版本的问题，elasticsearch-hadoop 5.0.0 依赖的spark版本是2.0.0。这一点可以从IDEA集成开发环境的pom文件中使用 Ctrl 键并点击 elasticsearch-hadoop 可以进入它依赖的所有包，其中能找到Spark的版本。2.1.2 依赖的是Spark 1.5.1，他与Spark 2.x.x不兼容。所以此时如果使用2.1.2的话就会导致最终提交spark任务时出现 method not found 等bug。提交Spark任务时凡是出现 method not found 或者 class not found 或者 class not definition等，基本上都可以考虑是依赖库的版本不兼容出现的问题。

完整的Spark Streaming 程序如下

/** * InterfaceCallingAnalysis.scala */

import java.io.PrintWriter

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
import org.elasticsearch.spark._
import scala.collection.JavaConversions._

object InterfaceCallingAnalysis {

  def main(args: Array[String]): Unit = {


    val conf = new SparkConf().setAppName("InterfaceCallingAnalysis")
    conf.set("es.index.auto.create", "true")
    conf.set("es.nodes", "192.168.1.2")
    conf.set("es.port", "9200")


    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Seconds(60))

    val lines = ssc.socketTextStream("192.168.1.1", 9999)
    val pairs = lines.map(MyFunctions.func1)

    val result = pairs.reduceByKey((a, b) => a + b)

    result.foreachRDD(rdd => {
      rdd.map(line => {
        val time = (System.currentTimeMillis() / 60000) * 60
        Map("hostIp" -> line._1.hostIp,
          "remoteIp" -> line._1.remoteIp,
          "callTimes" -> line._2.toLong,
          "time" -> time)
      }).saveToEs("jsf_node_speed/infoType")
    })

    ssc.start()
    ssc.awaitTermination()
  }
}

       上述的 es.port 设置为 9200，这里稍作解释（大牛请绕道），ElasticSearch 客户端程序除了Java 使用TCP的方式连接ES集群以外，其他的语言基本上都是使用的Http的方式。众所周知，ES 客户端默认的TCP端口为9300，而HTTP默认端口为9200。elasticsearch-hadoop 使用的就是HTTP的方式连接的ES集群。

/** * MyFunctions.scala */
object MyFunctions {

  def func1(str: String): (MyValue, Long) = {

    var ans = new MyValue()

    val jsonParser = new JSONParser()
    val jsonObj: JSONObject = jsonParser.parse(str).asInstanceOf[JSONObject]

    val hostIp = jsonObj.getAsString("host")
    val remoteIp = jsonObj.getAsString("remote")
    val callTimes = jsonObj.getAsString("callTimes").toLong

    ans.hostIp = hostIp
    ans.remoteIp = remoteIp

    // 返回类型(MyValue, Int)
    (ans, callTimes)
  }
}

3. 添加Maven依赖

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.jd.security</groupId>
    <artifactId>SparkTest</artifactId>
    <version>1.0.0</version>
    <inceptionYear>2008</inceptionYear>
    <properties>
        <scala.version>2.11.8</scala.version>
        <java.version>1.8</java.version>
        <spark.version>2.1.0</spark.version>
    </properties>

    <repositories>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <!-- Spark dependencies -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/net.minidev/json-smart -->
        <dependency>
            <groupId>net.minidev</groupId>
            <artifactId>json-smart</artifactId>
            <version>2.3</version>
        </dependency>

        <!-- Scala dependencies -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scalap</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-hadoop</artifactId>
            <version>5.0.0</version>
        </dependency>


    </dependencies>


    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <resources>
            <resource>
                <directory>src/main/resource</directory >
            </resource>
        </resources>

        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.8</arg>
                    </args>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>com.jd.security.InterfaceCallingAnalysis</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>assembly</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
</project>