spark-compile spark 1.4.1

abstract,spark can be compiled with:

maven,

sbt,

intellj ideal

ref:Spark1.0.0 源码编译和部署包生成

 

  also,if u want to load spark-project into eclipse ,then it is necessary to make a 'eclipse project' first by one of below solutions:

1.mvn eclipse:eclipse [optional]

2. ./sbt/sbt clean compile package

 or sbt/sbt then input 'eclipse'

 

note:

-the spark.ui.port is used by 'spark-shell'(or spark-submit?no) to show info about jobs,stages,executors,environments(these will be shown also by any completed apps) etc.related logs will be printed when it starts up:

 

15/11/11 15:12:45 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/11/11 15:12:45 INFO SparkUI: Started SparkUI at http://192.168.1.138:4040
15/11/11 15:12:45 INFO Executor: Starting executor ID driver on host localhost

 

ref:

http://blog.csdn.net/yunlong34574/article/details/39213503

https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IDESetup

http://blog.csdn.net/chenxingzhen001/article/details/25901237