spark的maven项目构建（基于idea 和maven）

环境预安装

• idea安装
• java安装
• scala安装
• idea安装scala插件

1.新建一个maven项目

开始创建项目体系结构
File --> Project

image

选择org.scala-tools.archetypes:scala-archetype-simple

image

maven的设置

image

项目名称

image

pom.xml

    4.0.0
    test.gzstrong
    Maven_BigData
    1.0-SNAPSHOT
    2008

    
      2.11.0
      UTF-8
    

    
      
        scala-tools.org
        Scala-Tools Maven2 Repository
        http://scala-tools.org/repo-releases
      
    

    
      
        scala-tools.org
        Scala-Tools Maven2 Repository
        http://scala-tools.org/repo-releases
      
    

    
      
        org.scala-lang
        scala-library
        ${scala.version}
      
      
        org.specs
        specs
        1.2.5
        test
      

      
        org.apache.spark
        spark-core_2.11
        2.3.0
      
    


    
      src/main/scala
      src/test/scala
      
        
          org.scala-tools
          maven-scala-plugin
          
            
              
                compile
                testCompile
              
            
          
          
            ${scala.version}
            
              -target:jvm-1.5
            
          
        
        
          org.apache.maven.plugins
          maven-eclipse-plugin
          
            true
            
              ch.epfl.lamp.sdt.core.scalabuilder
            
            
              ch.epfl.lamp.sdt.core.scalanature
            
            
              org.eclipse.jdt.launching.JRE_CONTAINER
              ch.epfl.lamp.sdt.launching.SCALA_CONTAINER
            
          
        
      
    
    
      
        
          org.scala-tools
          maven-scala-plugin
          
            ${scala.version}
          
        
      
    
  

2.目录结构如下:

image

3.在src的main目录下新建java和scala文件夹

4.同时修改java和scala文件夹为源码文件夹

image

5.在src的test目录下新建java和scala文件夹

image

6.同时修改java和scala文件夹为测试文件夹

image

然后设置scal为项目的sdk

7.写个spark

SparkApp.scala

package test.gzstrong

import org.apache.spark.{SparkConf, SparkContext}

object SparkApp{
    def main(args: Array[String]): Unit = {
        val conf = new SparkConf().setAppName("test").setMaster("local[2]")
        val sc = new SparkContext(conf)
        val data = Array(1, 2,2,1,2, 3, 4, 5)
        val distData = sc.parallelize(data)
        val pairs = distData.map((_, 1))
        val counts = pairs.reduceByKey((a, b) => a + b)
        counts.collect().foreach(println)
        sc.stop()
    }
}

8.运行

运行