apache spark 初体验

spark

1、       Setting up Maven’s Memory Usage

l  linux平台

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

 

l  win平台（不需要引号）

set MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m

 

2、       Building with build/mvn

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

3、       静静等待编译完成

[INFO] Spark Project REPL ................................. SUCCESS [01:02 min]

[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 19.437 s]

[INFO] Spark Project YARN ................................. SUCCESS [01:14 min]

[INFO] Spark Project Assembly ............................. SUCCESS [01:29 min]

[INFO] Spark Project External Twitter ..................... SUCCESS [ 54.224 s]

[INFO] Spark Project External Flume Sink .................. SUCCESS [ 23.389 s]

[INFO] Spark Project External Flume ....................... SUCCESS [ 59.639 s]

[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 25.066 s]

[INFO] Spark Project External MQTT ........................ SUCCESS [05:05 min]

[INFO] Spark Project External MQTT Assembly ............... SUCCESS [01:01 min]

[INFO] Spark Project External ZeroMQ ...................... SUCCESS [01:17 min]

[INFO] Spark Project External Kafka ....................... SUCCESS [02:43 min]

[INFO] Spark Project Examples ............................. SUCCESS [37:14 min]

[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 12.713 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:22 h

[INFO] Finished at: 2016-02-17T15:45:54+08:00

[INFO] Final Memory: 100M/1468M

[INFO] ------------------------------------------------------------------------

居然耗时1个半小时，晕，中间下载maven依赖失败N次

……耐心