maven环境下使用java、scala混合开发spark应用

熟悉java的开发者在开发spark应用时，常常会遇到spark对java的接口文档不完善或者不提供对应的java接口的问题。这个时候，如果在java项目中能直接使用scala来开发spark应用，同时使用java来处理项目中的其它需求，将在一定程度上降低开发spark项目的难度。下面就来探索一下java、scala、spark、maven这一套开发环境要怎样来搭建。

 

 

1、下载scala sdk

 

http://www.scala-lang.org/download/ 直接到这里下载sdk，目前最新的稳定版为2.11.7，下载后解压就行

（后面在intellijidea中创建.scala后缀源代码时，ide会智能感知并提示你设置scala sdk，按提示指定sdk目录为解压目录即可）

 也可以手动配置scala SDK：ideal =>File =>project struct.. =>library..=> +...

2、下载scala forintellij idea的插件

 

如上图，直接在plugins里搜索Scala，然后安装即可，如果不具备上网环境，或网速不给力。也可以直接到http://plugins.jetbrains.com/plugin/?idea&id=1347手动下载插件的zip包，手动下载时，要特别注意版本号，一定要跟本机的intellij idea的版本号匹配，否则下载后无法安装。下载完成后，在上图中，点击“Install plugin from disk...”，选择插件包的zip即可。

3、如何跟maven整合

使用maven对项目进行打包的话，需要在pom文件中配置scala-maven-plugin这个插件。同时，由于是spark开发，jar包需要打包为可执行java包，还需要在pom文件中配置maven-assembly-plugin和maven-shade-plugin插件并设置mainClass。经过实验摸索，下面贴出一个可用的pom文件，使用时只需要在包依赖上进行增减即可使用。

2. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

3. 4.0.0

4. my-project-groupid

5. sparkTest

6. jar

7. 1.0-SNAPSHOT

8. sparkTest

9. http://maven.apache.org

11. UTF-8

12. 0.98.3

14. 1.6.0

15. 1.7

16. 2.10.5

21. repo1.maven.org

22. http://repo1.maven.org/maven2

24. true

27. false

30.  

32. repository.jboss.org

33. http://repository.jboss.org/nexus/content/groups/public/

36. false

39.  

41. cloudhopper

42. Repository for Cloudhopper

43. http://maven.cloudhopper.com/repos/third-party/

45. true

48. false

51.  

53. mvnr

54. Repository maven

55. http://mvnrepository.com/

57. true

60. false

63.  

65. scala

66. Scala Tools

67. https://mvnrepository.com/

69. true

72. false

76.  
77.  

80. scala

81. Scala Tools

82. https://mvnrepository.com/

84. true

87. false

91.  

94. org.scala-lang

95. scala-library

96. ${scala.version}

97. compile

100. org.scala-lang

101. scala-compiler

102. ${scala.version}

103. compile

107. javax.mail

108. javax.mail-api

109. 1.4.7

112. junit

113. junit

114. 3.8.1

115. test

119. org.apache.spark

120. spark-core_2.10

121. ${spark.version}

123.  

126. org.apache.spark

127. spark-sql_2.10

128. ${spark.version}

132. org.apache.spark

133. spark-streaming_2.10

134. ${spark.version}

138. org.apache.spark

139. spark-mllib_2.10

140. ${spark.version}

144. org.apache.spark

145. spark-hive_2.10

146. ${spark.version}

150. org.apache.spark

151. spark-graphx_2.10

152. ${spark.version}

155. mysql

156. mysql-connector-java

157. 5.1.30

170. com.google.guava

171. guava

172. 14.0.1

175. org.apache.hadoop

176. hadoop-common

177. 2.6.0

180. org.apache.hadoop

181. hadoop-client

182. 2.6.0

185. org.apache.spark

186. spark-hive_2.10

187. ${spark.version}

190. com.alibaba

191. fastjson

192. 1.2.3

195. p6spy

196. p6spy

197. 1.3

200. org.apache.commons

201. commons-math3

202. 3.3

204.  

206. org.jdom

207. jdom

208. 2.0.2

210.  

212. com.google.guava

213. guava

214. 14.0.1

217. org.apache.hadoop

218. hadoop-common

219. 2.6.0

222. org.apache.hadoop

223. hadoop-hdfs

224. 2.6.0

227. redis.clients

228. jedis

229. 2.6.0

233. org.apache.hbase

234. hbase-client

235. 0.98.6-hadoop2

239. org.apache.hbase

240. hbase

241. 0.98.6-hadoop2

242. pom

246. org.apache.hbase

247. hbase-common

248. 0.98.6-hadoop2

252. org.apache.hbase

253. hbase-server

254. 0.98.6-hadoop2

257. org.testng

258. testng

259. 6.8.8

260. test

263. mysql

264. mysql-connector-java

265. 5.1.30

268. com.fasterxml.jackson.jaxrs

269. jackson-jaxrs-json-provider

270. 2.4.4

273. com.fasterxml.jackson.core

274. jackson-databind

275. 2.4.4

278. net.sf.json-lib

279. json-lib

280. 2.4

281. jdk15

285. javax.mail

286. javax.mail-api

287. 1.4.7

289.  

291. junit

292. junit

293. 3.8.1

294. test

297.  

302. maven-assembly-plugin

304. false

306. jar-with-dependencies

310. rrkd.dt.sparkTest.HelloWorld

316. make-assembly

317. package

319. assembly

325. org.apache.maven.plugins

326. maven-compiler-plugin

327. 3.1

329. ${jdk.version}

330. ${jdk.version}

331. ${project.build.sourceEncoding}

335. org.apache.maven.plugins

336. maven-shade-plugin

337. 2.1

339. false

343. package

345. shade

348. true

349. allinone

352. *:*

355.  

358. *:*

360. META-INF/*.SF

361. META-INF/*.DSA

362. META-INF/*.RSA

366.  

369. implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

370. reference.conf

373. rrkd.dt.sparkTest.HelloWorld

380.  

383. net.alchim31.maven

384. scala-maven-plugin

385. 3.2.0

388. compile-scala

389. compile

391. add-source

392. compile

396. test-compile-scala

397. test-compile

399. add-source

400. testCompile

405. ${scala.version}

408.  
410.  
412.  

主要是build部分的配置，其它的毋须过多关注。

 

项目的目录结构，大体跟maven的默认约定一样，只是src下多了一个scala目录，主要还是为了便于组织java源码和scala源码，如下图：

 

 

在java目录下建立HelloWorld类HelloWorld.class：

1. package test;

2.  

3. import test.Hello;

4. /**

5. * Created by L on 2017/1/5.

6. */

7. public class HelloWorld {

8.  

9. public static void main(String[] args){

10. System.out.print("test");

11. Hello.sayHello("scala");

12. Hello.runSpark();

13. }

14. }

 

在scala目录下建立hello类hello.scala：

1. package test

2.  

3. import org.apache.spark.graphx.{Graph, Edge, VertexId, GraphLoader}

4. import org.apache.spark.rdd.RDD

5. import org.apache.spark.{SparkContext, SparkConf}

6. import breeze.linalg.{Vector, DenseVector, squaredDistance}

7. /**

8. * Created by L on 2017/1/5.

9. */

10. object Hello {

11. def sayHello(x: String): Unit = {

12. println("hello," + x);

13. }

14.  

15. // def main(args: Array[String]) {

16. def runSpark() {

17. val sparkConf = new SparkConf().setAppName("SparkKMeans").setMaster("local[*]")

18. val sc = new SparkContext(sparkConf)

19. // Create an RDD for the vertices

20. val users: RDD[(VertexId, (String, String))] =

21. sc.parallelize(Array((3L, ("rxin", "student")), (7L, ("jgonzal", "postdoc")),

22. (5L, ("franklin", "prof")), (2L, ("istoica", "prof")),

23. (4L, ("peter", "student"))))

24. // Create an RDD for edges

25. val relationships: RDD[Edge[String]] =

26. sc.parallelize(Array(Edge(3L, 7L, "collab"), Edge(5L, 3L, "advisor"),

27. Edge(2L, 5L, "colleague"), Edge(5L, 7L, "pi"),

28. Edge(4L, 0L, "student"), Edge(5L, 0L, "colleague")))

29. // Define a default user in case there are relationship with missing user

30. val defaultUser = ("John Doe", "Missing")

31. // Build the initial Graph

32. val graph = Graph(users, relationships, defaultUser)

33. // Notice that there is a user 0 (for which we have no information) connected to users

34. // 4 (peter) and 5 (franklin).

35. graph.triplets.map(

36. triplet => triplet.srcAttr._1 + " is the " + triplet.attr + " of " + triplet.dstAttr._1

37. ).collect.foreach(println(_))

38. // Remove missing vertices as well as the edges to connected to them

39. val validGraph = graph.subgraph(vpred = (id, attr) => attr._2 != "Missing")

40. // The valid subgraph will disconnect users 4 and 5 by removing user 0

41. validGraph.vertices.collect.foreach(println(_))

42. validGraph.triplets.map(

43. triplet => triplet.srcAttr._1 + " is the " + triplet.attr + " of " + triplet.dstAttr._1

44. ).collect.foreach(println(_))

45. sc.stop()

46. }

47. }

这样子，在scala项目中调用spark的接口来运行一些spark应用，在java项目中再调用scala。

4、scala项目maven的编译打包

java/scala混合的项目，怎么先编译scala再编译java，可以使用以下maven 命令来进行编译打包：

mvn clean scala:compile assembly:assembly

5、spark项目的jar包的运行问题

在开发时，我们可能会以local模式在IDEA中运行，然后使用了上面的命令进行打包。打包后的spark项目必须要放到spark集群下以spark-submit的方式提交运行。

--------------------- 本文来自 大愚若智_ 的CSDN 博客 ，全文地址请点击：https://blog.csdn.net/zbc1090549839/article/details/54290233?utm_source=copy