【Spark十五】Intellj Idea搭建Spark源代码开发环境

 

Profile的选择:

 

1. hadoop-2.4和YARN勾选,复选框中显示黑色的对号

2. hbase-hadoop1,maven-3,scala-2.10默认选择,复选框中显示灰的对号

 

 

 问题:

1. PermGen内存

[ERROR] PermGen space -> [Help 1]
java.lang.OutOfMemoryError: PermGen space
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at scala.tools.nsc.Global.formatExplain(Global.scala:1140)
	at scala.tools.nsc.Global.supplementErrorMessage(Global.scala:1157)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1565)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553)
	at scala.tools.nsc.Global$Run.compile(Global.scala:1662)

Maven显示内存的使用情况Final Memory: 39M/827M,表示PermGen space为39M?

解决办法:https://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError

在/etc/profile中添加如下Maven的内存设置(本机物理内存2G)

 

 

export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m"

 

这种做法适用于使用mvn在shell终端进行构建,如果在Intellij Idea中构建报这个错,则需要打开Idea的Maven配置窗口,在

Maven

      Runner的VM OPTS中输入如下参数:

 

-Xmx1024m -XX:MaxPermSize=512m

 

2. Maven编译两个小时多之后,报编译错误,CodeGenerator.scala报很多错误.不过系统仍然可以编译

 

        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:387: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:403: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:411: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:418: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:440: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:456: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:468: error: value q is not a member of StringContext
        q"""
        ^
/home/hadoop/sparksourcecode/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:486: error: value q is not a member of StringContext
        q"""
        ^

 

3.EventBatch.scala找不到

编译两个多小时,这个文件依然不存在,

 j

 

4。经过4个小时的构建,终于编完了。。。竟然编译成功了,但是EventBatch依然不存在(所以,FlumeBatchFetcher.scala报错EventBatch和SparkFlumeProtocol不存在)和CodeGenerator.scala中依然无数的第二步的错

 

[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [ 18.895 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 55.038 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 41.395 s]
[INFO] Spark Project Core ................................. SUCCESS [11:58 min]
[INFO] Spark Project Bagel ................................ SUCCESS [02:01 min]
[INFO] Spark Project GraphX ............................... SUCCESS [09:00 min]
[INFO] Spark Project Streaming ............................ SUCCESS [18:49 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [24:13 min]
[INFO] Spark Project SQL .................................. SUCCESS [28:07 min]
[INFO] Spark Project ML Library ........................... SUCCESS [39:05 min]
[INFO] Spark Project Tools ................................ SUCCESS [02:20 min]
[INFO] Spark Project Hive ................................. SUCCESS [24:52 min]
[INFO] Spark Project REPL ................................. SUCCESS [08:47 min]
[INFO] Spark Project YARN ................................. SUCCESS [07:45 min]
[INFO] Spark Project Assembly ............................. SUCCESS [10:06 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [01:38 min]
[INFO] Spark Project External Flume Sink .................. SUCCESS [01:48 min]
[INFO] Spark Project External Flume ....................... SUCCESS [03:02 min]
[INFO] Spark Project External MQTT ........................ SUCCESS [01:35 min]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [01:27 min]
[INFO] Spark Project External Kafka ....................... SUCCESS [02:34 min]
[INFO] Spark Project Examples ............................. SUCCESS [24:45 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [01:22 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:47 h
[INFO] Finished at: 2015-01-10T03:29:36+08:00
[INFO] Final Memory: 102M/882M
[INFO] ------------------------------------------------------------------------

 

5.既然编译成功了,那就运行LocalPi.scala,触发Intellj Idea Make整个项目,又是一顿等待(这种情况产生的原因是什么?为什么不能只编译一个文件),make的结果是

 

Compilation completed with 44 errors and 37 warnings in 15 min 23 sec

 

查看出错信息,基本都是Spark SQL的catalyst相关的报错:

 

Error:(317, 109) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function1[_, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                            ^
Error:(317, 109) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function1[_, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                            ^
Error:(319, 112) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function2[_, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                               ^
Error:(319, 112) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function2[_, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                               ^
Error:(321, 115) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function3[_, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                  ^
Error:(321, 115) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function3[_, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                  ^
Error:(323, 118) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function4[_, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                     ^
Error:(323, 118) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function4[_, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                     ^
Error:(325, 121) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function5[_, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                        ^
Error:(325, 121) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function5[_, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                        ^
Error:(327, 124) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function6[_, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                           ^
Error:(327, 124) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function6[_, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                           ^
Error:(329, 127) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function7[_, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                              ^
Error:(329, 127) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function7[_, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                              ^
Error:(331, 130) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function8[_, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                 ^
Error:(331, 130) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function8[_, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                 ^
Error:(333, 133) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function9[_, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                    ^
Error:(333, 133) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function9[_, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                    ^
Error:(335, 137) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function10[_, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                        ^
Error:(335, 137) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function10[_, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                        ^
Error:(337, 140) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function11[_, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                           ^
Error:(337, 140) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function11[_, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                           ^
Error:(339, 143) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function12[_, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                              ^
Error:(339, 143) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function12[_, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                              ^
Error:(341, 146) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function13[_, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                 ^
Error:(341, 146) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function13[_, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                 ^
Error:(343, 149) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function14[_, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                    ^
Error:(343, 149) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function14[_, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                    ^
Error:(345, 152) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                       ^
Error:(345, 152) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                       ^
Error:(347, 155) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                          ^
Error:(347, 155) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                          ^
Error:(349, 158) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                             ^
Error:(349, 158) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                             ^
Error:(351, 161) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                ^
Error:(351, 161) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                ^
Error:(353, 164) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                   ^
Error:(353, 164) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                   ^
Error:(355, 167) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                      ^
Error:(355, 167) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                      ^
Error:(357, 170) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                         ^
Error:(357, 170) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                         ^
Error:(359, 173) polymorphic expression cannot be instantiated to expected type;
 found   : [T(in method apply)]org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T(in method apply)]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T(in method functionToUdfBuilder)]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                            ^
Error:(359, 173) type mismatch;
 found   : org.apache.spark.sql.catalyst.dsl.ScalaUdfBuilder[T]
 required: org.apache.spark.sql.catalyst.dsl.package.ScalaUdfBuilder[T]
  implicit def functionToUdfBuilder[T: TypeTag](func: Function22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T]): ScalaUdfBuilder[T] = ScalaUdfBuilder(func)
                                                                                                                                                                            ^

 

 

EventBatch不能编译出来的解决

 

Log4j冲突问题的解决:

 


【Spark十五】Intellj Idea搭建Spark源代码开发环境_第1张图片
 

 

 

 远程调试代码:

http://www.iteblog.com/archives/1192

 

 

参考:

http://www.cnblogs.com/luogankun/p/3905148.html

你可能感兴趣的:(Spark)