1.配置基本环境:
jdk1.7
scala-2.10.4
maven 3.3.9
具体请看【2】和系列文章
2.编译运行
(1)下载:
https://github.com/apache/spark
(2)编译:
D:\1win7\java\spark-1.5.2>set MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512M -XX:Reserve dCodeCacheSize=512m D:\1win7\java\spark-1.5.2>mvn -DskipTests clean package
3.编译成功:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [04:40 min] [INFO] Spark Project Launcher ............................. SUCCESS [03:19 min] [INFO] Spark Project Networking ........................... SUCCESS [ 24.102 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.934 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 40.276 s] [INFO] Spark Project Core ................................. SUCCESS [09:48 min] [INFO] Spark Project Bagel ................................ SUCCESS [ 29.549 s] [INFO] Spark Project GraphX ............................... SUCCESS [01:41 min] [INFO] Spark Project Streaming ............................ SUCCESS [02:51 min] [INFO] Spark Project Catalyst ............................. SUCCESS [03:24 min] [INFO] Spark Project SQL .................................. SUCCESS [04:27 min] [INFO] Spark Project ML Library ........................... SUCCESS [09:14 min] [INFO] Spark Project Tools ................................ SUCCESS [ 17.949 s] [INFO] Spark Project Hive ................................. SUCCESS [12:06 min] [INFO] Spark Project REPL ................................. SUCCESS [01:05 min] [INFO] Spark Project Assembly ............................. SUCCESS [01:54 min] [INFO] Spark Project External Twitter ..................... SUCCESS [ 36.062 s] [INFO] Spark Project External Flume Sink .................. SUCCESS [02:01 min] [INFO] Spark Project External Flume ....................... SUCCESS [ 37.934 s] [INFO] Spark Project External Flume Assembly .............. SUCCESS [ 4.781 s] [INFO] Spark Project External MQTT ........................ SUCCESS [03:22 min] [INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 26.335 s] [INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 50.894 s] [INFO] Spark Project External Kafka ....................... SUCCESS [02:19 min] [INFO] Spark Project Examples ............................. SUCCESS [06:28 min] [INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 8.595 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:13 h [INFO] Finished at: 2016-05-12T18:31:54+08:00 [INFO] Final Memory: 89M/1375M [INFO] ------------------------------------------------------------------------
4. 编译记录:
2 overlapping classes: [WARNING] - org.apache.commons.beanutils.WrapDynaBean [WARNING] - org.apache.commons.beanutils.Converter [WARNING] - org.apache.commons.beanutils.converters.IntegerConverter [WARNING] - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean [WARNING] - org.apache.commons.beanutils.locale.converters.DecimalLocaleConver ter [WARNING] - org.apache.commons.beanutils.locale.converters.DoubleLocaleConvert er [WARNING] - org.apache.commons.beanutils.converters.ShortConverter [WARNING] - org.apache.commons.beanutils.converters.StringArrayConverter [WARNING] - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean [WARNING] - org.apache.commons.beanutils.LazyDynaClass [WARNING] - 72 more... [WARNING] spark-streaming-twitter_2.10-1.5.2.jar, spark-streaming-kafka_2.10-1.5 .2.jar, spark-streaming-mqtt_2.10-1.5.2.jar, spark-streaming-flume_2.10-1.5.2.ja r, spark-streaming-zeromq_2.10-1.5.2.jar, unused-1.0.0.jar, spark-streaming-flum e-sink_2.10-1.5.2.jar define 1 overlapping classes: [WARNING] - org.apache.spark.unused.UnusedStubClass [WARNING] hadoop-common-2.2.0-tests.jar, hadoop-mapreduce-client-jobclient-2.2.0 -tests.jar define 4 overlapping classes: [WARNING] - org.apache.hadoop.util.TestReflectionUtils [WARNING] - org.apache.hadoop.ipc.TestSocketFactory [WARNING] - org.apache.hadoop.util.TestRunJar [WARNING] - org.apache.hadoop.util.TestReflectionUtils$1 [WARNING] cassandra-thrift-1.2.6.jar, cassandra-all-1.2.6.jar define 2 overlappi ng classes: [WARNING] - org.apache.cassandra.thrift.ITransportFactory [WARNING] - org.apache.cassandra.thrift.TFramedTransportFactory [WARNING] jsp-2.1-6.1.14.jar, jasper-compiler-5.5.23.jar define 143 overlapping classes: [WARNING] - org.apache.jasper.compiler.TagLibraryInfoImpl [WARNING] - org.apache.jasper.xmlparser.SymbolTable [WARNING] - org.apache.jasper.compiler.Generator$FragmentHelperClass$Fragment [WARNING] - org.apache.jasper.compiler.Generator$1TagHandlerPoolVisitor [WARNING] - org.apache.jasper.compiler.Node$AttributeGenerator [WARNING] - org.apache.jasper.compiler.SmapStratum$LineInfo [WARNING] - org.apache.jasper.compiler.ScriptingVariabler [WARNING] - org.apache.jasper.compiler.Node [WARNING] - org.apache.jasper.compiler.Node$JspAttribute [WARNING] - org.apache.jasper.compiler.tagplugin.TagPlugin [WARNING] - 133 more... [WARNING] javax.servlet-api-3.0.1.jar, javax.servlet-3.1.jar define 32 overlappi ng classes: [WARNING] - javax.servlet.Registration [WARNING] - javax.servlet.DispatcherType [WARNING] - javax.servlet.ServletRegistration$Dynamic [WARNING] - javax.servlet.FilterRegistration [WARNING] - javax.servlet.Registration$Dynamic [WARNING] - javax.servlet.annotation.WebFilter [WARNING] - javax.servlet.annotation.ServletSecurity$TransportGuarantee [WARNING] - javax.servlet.ServletContainerInitializer [WARNING] - javax.servlet.FilterRegistration$Dynamic [WARNING] - javax.servlet.descriptor.JspPropertyGroupDescriptor [WARNING] - 22 more... [WARNING] maven-shade-plugin has detected that some class files are [WARNING] present in two or more JARs. When this happens, only one [WARNING] single version of the class is copied to the uber jar. [WARNING] Usually this is not harmful and you can skip these warnings, [WARNING] otherwise try to manually exclude artifacts based on [WARNING] mvn dependency:tree -Ddetail=true and the above output. [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO] [INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-examp les_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\examples\target\spark-examples_2. 10-1.5.2-sources.jar [INFO] [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark- examples_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\examples\target\spark-examples_2. 10-1.5.2-test-sources.jar [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Spark Project External Kafka Assembly 1.5.2 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-streaming-kafk a-assembly_2.10 --- [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ spark-streamin g-kafka-assembly_2.10 --- [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-stre aming-kafka-assembly_2.10 --- [INFO] Add Source directory: D:\1win7\java\spark-1.5.2\external\kafka-assembly\s rc\main\scala [INFO] Add Test Source directory: D:\1win7\java\spark-1.5.2\external\kafka-assem bly\src\test\scala [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-streaming -kafka-assembly_2.10 --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-stre aming-kafka-assembly_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\1win7\java\spark-1.5.2\external\ka fka-assembly\src\main\resources [INFO] Copying 3 resources [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-stream ing-kafka-assembly_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ spark-streaming -kafka-assembly_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-streaming-kafka- assembly_2.10 --- [INFO] Executing tasks main: [mkdir] Created dir: D:\1win7\java\spark-1.5.2\external\kafka-assembly\targe t\tmp [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sp ark-streaming-kafka-assembly_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\1win7\java\spark-1.5.2\external\ka fka-assembly\src\test\resources [INFO] Copying 3 resources [INFO] [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spa rk-streaming-kafka-assembly_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ spark-s treaming-kafka-assembly_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-dependency-plugin:2.10:build-classpath (default) @ spark-stream ing-kafka-assembly_2.10 --- [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ spark-streaming-ka fka-assembly_2.10 --- [INFO] Tests are skipped. [INFO] [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-streaming-kafka-assemb ly_2.10 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) @ spark-streaming-ka fka-assembly_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp ark-streaming-kafka-assembly_2.10-1.5.2-tests.jar [INFO] [INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ spark-streaming-kafka-assemb ly_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp ark-streaming-kafka-assembly_2.10-1.5.2.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-s treaming-kafka-assembly_2.10 --- [INFO] [INFO] --- maven-shade-plugin:2.4.1:shade (default) @ spark-streaming-kafka-asse mbly_2.10 --- [INFO] Including org.apache.spark:spark-streaming-kafka_2.10:jar:1.5.2 in the sh aded jar. [INFO] Including org.apache.kafka:kafka_2.10:jar:0.8.2.1 in the shaded jar. [INFO] Including com.yammer.metrics:metrics-core:jar:2.2.0 in the shaded jar. [INFO] Including org.apache.kafka:kafka-clients:jar:0.8.2.1 in the shaded jar. [INFO] Including com.101tec:zkclient:jar:0.3 in the shaded jar. [INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.6 in the shaded jar. [INFO] Including commons-io:commons-io:jar:2.1 in the shaded jar. [INFO] Including org.apache.avro:avro:jar:1.7.7 in the shaded jar. [INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar . [INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.2.0 in the shaded ja r. [INFO] Including com.google.inject:guice:jar:3.0 in the shaded jar. [INFO] Including javax.inject:javax.inject:jar:1 in the shaded jar. [INFO] Including aopalliance:aopalliance:jar:1.0 in the shaded jar. [INFO] Including com.sun.jersey.jersey-test-framework:jersey-test-framework-griz zly2:jar:1.9 in the shaded jar. [INFO] Including com.sun.jersey.jersey-test-framework:jersey-test-framework-core :jar:1.9 in the shaded jar. [INFO] Including javax.servlet:javax.servlet-api:jar:3.0.1 in the shaded jar. [INFO] Including com.sun.jersey:jersey-client:jar:1.9 in the shaded jar. [INFO] Including com.sun.jersey:jersey-grizzly2:jar:1.9 in the shaded jar. [INFO] Including org.glassfish.grizzly:grizzly-http:jar:2.1.2 in the shaded jar. [INFO] Including org.glassfish.grizzly:grizzly-framework:jar:2.1.2 in the shaded jar. [INFO] Including org.glassfish.gmbal:gmbal-api-only:jar:3.0.0-b023 in the shaded jar. [INFO] Including org.glassfish.external:management-api:jar:3.0.0-b012 in the sha ded jar. [INFO] Including org.glassfish.grizzly:grizzly-http-server:jar:2.1.2 in the shad ed jar. [INFO] Including org.glassfish.grizzly:grizzly-rcm:jar:2.1.2 in the shaded jar. [INFO] Including org.glassfish.grizzly:grizzly-http-servlet:jar:2.1.2 in the sha ded jar. [INFO] Including org.glassfish:javax.servlet:jar:3.1 in the shaded jar. [INFO] Including com.sun.jersey:jersey-json:jar:1.9 in the shaded jar. [INFO] Including org.codehaus.jettison:jettison:jar:1.1 in the shaded jar. [INFO] Including com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 in the shaded jar. [INFO] Including javax.xml.bind:jaxb-api:jar:2.2.2 in the shaded jar. [INFO] Including javax.activation:activation:jar:1.1 in the shaded jar. [INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar . [INFO] Including org.codehaus.jackson:jackson-xc:jar:1.9.13 in the shaded jar. [INFO] Including com.sun.jersey.contribs:jersey-guice:jar:1.9 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.2.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.2.0 in the shaded ja r. [INFO] Including org.apache.avro:avro-ipc:jar:1.7.7 in the shaded jar. [INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar. [INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shade d jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. [WARNING] hadoop-yarn-common-2.2.0.jar, hadoop-yarn-api-2.2.0.jar define 3 overl apping classes: [WARNING] - org.apache.hadoop.yarn.factories.package-info [WARNING] - org.apache.hadoop.yarn.util.package-info [WARNING] - org.apache.hadoop.yarn.factory.providers.package-info [WARNING] unused-1.0.0.jar, spark-streaming-kafka_2.10-1.5.2.jar define 1 overla pping classes: [WARNING] - org.apache.spark.unused.UnusedStubClass [WARNING] javax.servlet-api-3.0.1.jar, javax.servlet-3.1.jar define 74 overlappi ng classes: [WARNING] - javax.servlet.ServletRequestWrapper [WARNING] - javax.servlet.Registration [WARNING] - javax.servlet.ServletContextAttributeListener [WARNING] - javax.servlet.http.HttpSessionAttributeListener [WARNING] - javax.servlet.ServletRegistration$Dynamic [WARNING] - javax.servlet.http.HttpServlet [WARNING] - javax.servlet.FilterRegistration [WARNING] - javax.servlet.http.HttpServletRequest [WARNING] - javax.servlet.annotation.WebFilter [WARNING] - javax.servlet.ServletResponse [WARNING] - 64 more... [WARNING] maven-shade-plugin has detected that some class files are [WARNING] present in two or more JARs. When this happens, only one [WARNING] single version of the class is copied to the uber jar. [WARNING] Usually this is not harmful and you can skip these warnings, [WARNING] otherwise try to manually exclude artifacts based on [WARNING] mvn dependency:tree -Ddetail=true and the above output. [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\spark- streaming-kafka-assembly_2.10-1.5.2.jar with D:\1win7\java\spark-1.5.2\external\ kafka-assembly\target\spark-streaming-kafka-assembly_2.10-1.5.2-shaded.jar [INFO] Dependency-reduced POM written at: D:\1win7\java\spark-1.5.2\external\kaf ka-assembly\dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: D:\1win7\java\spark-1.5.2\external\kaf ka-assembly\dependency-reduced-pom.xml [INFO] [INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-strea ming-kafka-assembly_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp ark-streaming-kafka-assembly_2.10-1.5.2-sources.jar [INFO] [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark- streaming-kafka-assembly_2.10 --- [INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp ark-streaming-kafka-assembly_2.10-1.5.2-test-sources.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [04:40 min] [INFO] Spark Project Launcher ............................. SUCCESS [03:19 min] [INFO] Spark Project Networking ........................... SUCCESS [ 24.102 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.934 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 40.276 s] [INFO] Spark Project Core ................................. SUCCESS [09:48 min] [INFO] Spark Project Bagel ................................ SUCCESS [ 29.549 s] [INFO] Spark Project GraphX ............................... SUCCESS [01:41 min] [INFO] Spark Project Streaming ............................ SUCCESS [02:51 min] [INFO] Spark Project Catalyst ............................. SUCCESS [03:24 min] [INFO] Spark Project SQL .................................. SUCCESS [04:27 min] [INFO] Spark Project ML Library ........................... SUCCESS [09:14 min] [INFO] Spark Project Tools ................................ SUCCESS [ 17.949 s] [INFO] Spark Project Hive ................................. SUCCESS [12:06 min] [INFO] Spark Project REPL ................................. SUCCESS [01:05 min] [INFO] Spark Project Assembly ............................. SUCCESS [01:54 min] [INFO] Spark Project External Twitter ..................... SUCCESS [ 36.062 s] [INFO] Spark Project External Flume Sink .................. SUCCESS [02:01 min] [INFO] Spark Project External Flume ....................... SUCCESS [ 37.934 s] [INFO] Spark Project External Flume Assembly .............. SUCCESS [ 4.781 s] [INFO] Spark Project External MQTT ........................ SUCCESS [03:22 min] [INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 26.335 s] [INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 50.894 s] [INFO] Spark Project External Kafka ....................... SUCCESS [02:19 min] [INFO] Spark Project Examples ............................. SUCCESS [06:28 min] [INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 8.595 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:13 h [INFO] Finished at: 2016-05-12T18:31:54+08:00 [INFO] Final Memory: 89M/1375M [INFO] ------------------------------------------------------------------------ D:\1win7\java\spark-1.5.2>
参考:
【1】 http://www.jianshu.com/p/6ad9dc90b7a5
【2】http://blog.csdn.net/xubo245/article/details/51385683