Spark源码环境问题

异常1

点击idea–build–build project,或者打开源码显示导入异常。但是maven包都是下载好了的

Error:(45, 66) not found: type SparkFlumeProtocol
  val transactionTimeout: Int, val backOffInterval: Int) extends SparkFlumeProtocol with Logging {
	                                                                 ^

解决方法：
点击右侧 maven project ,右键选中相应的project , Generate Sources and update folders

异常2

右键—run sparkpi.scala报错异常如下：

Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 1 more

解决方法：
点击Run/debug configurations
选中“Include dependencies with provided scope”

spark 源码编译打包

mvn -Pyarn -Phive -Phive-thriftserver -Phadoop-2.7 -Dhadoop.version=2.7.4 -DskipTests clean package


./dev/make-distribution.sh --name spark-2.2.2 --tgz -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dhadoop-version=hadoop-2.7.4

spark 源码导入idea

直接open就行，然后设置为maven sbt project 依赖会自动下载。不要import 工程 as maven之类的。

idea debug 设置

以SparkPi为例，在运行参数中加入 -Dspark.master=local

远程debug设置

export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
./spark-submit --master "local[2]" --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.11-2.2.2.jar 100

scala style 注释问题

在源码中添加注释都报错

解决办法：
下载 http://www.scalastyle.org/scalastyle_config.xml 替换 工程根目录下的 scalastyle_config.xml即可