IntelliJ IDEA使用笔记—— 创建Maven Scala项目

IDEA Maven Scala项目pow.xml文件配置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mycompany.spark</groupId>
    <artifactId>SparkWC</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.2.0</spark.version>
        <hadoop.version>2.8.4</hadoop.version>
        <encoding>UTF-8</encoding>
    </properties>

    <dependencies>
        <!-- scala的依赖导入 -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- spark的依赖导入 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- hadoop-client API的导入 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- mysql的连接驱动依赖 -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.38</version>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <!-- scala的编译插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- Java的编译插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


            <!-- 打jar包插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF
                                        META-INF/*.DSA
                                        META-INF/*.RSA
                                    
                                
                            
                        
                    
                
            
        
    

删除java包，新建scala包

新建Scala class 编写代码

新建Scala class，可选择Class、Object和Trait3个类型。编写SparkWordCount代码。

编写SparkWordCount代码

import org.apache.spark.{SparkConf, SparkContext}

/**
  *
  * @Project Name: SparkWC
  * @File Name: SparkWordCount 
  * @author Johnson E-mail: [email protected]
  * @date 2019-01-09 15:06 
  * @version 1.0
  * @By IDE: IntelliJ IDEA
  *
  */

object SparkWordCount {
  def main(args: Array[String]): Unit = {
    // 1、设置参数：setAppName设置程序名、setMaster设置本地运行模式的线程数(*代表多个)
    val conf:SparkConf = new SparkConf().setAppName("SparkWordCount").setMaster("local[2]")

    // 2、创建Spark执行程序的入口
    val sc:SparkContext = new SparkContext(conf)

    // 3、加载数据，并且处理(枚举元素，扁平化，聚合，降序排序——ascending = false) ；使用空格做切割符
    sc.textFile(args(0)).flatMap(_.split(" ")).map((_,1))
      .reduceByKey(_+_)
      .sortBy(_._2, false)
      //保存文件
      .saveAsTextFile(args(1))

    // 4、关闭资源
    sc.stop()
  }

使用Alt+Enter快捷键自动导包

IDEA 运行/调试配置

添加／修改配置

添加应用程序配置


Name: SparkWordCount(自定义)

配置主类

已经有配置，可以编辑修改


Program arguments

注意

先不要建立，out目录；否则会运行不成功，提示目录已经存在！

SparkWordCount
D:\test\in D:\test\out

D:\test>tree /f
文件夹 PATH 列表
卷序列号为 B0F2-70C8
D:.
└─in
└─words.txt

IDEA 运行/调试输出结果

PS D:\test\out\in> type .\part-00000
(hello,3)
(love,2)
(Beijing,2)
(China,2)
(I,2)
PS D:\test\out\in> type .\part-00001
(is,1)
(capital,1)
(dilireba,1)
(of,1)
(the,1)
PS D:\test\out>

结果有进行分区。

打成JAR包

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 52.270 s
[INFO] Finished at: 2019-01-09T20:51:32+08:00
[INFO] Final Memory: 28M/183M

文件大小较大那个文件，包含依赖库。
SparkWC-1.0-SNAPSHOT.jar 91.4MB。

Ctrl+i快捷键重写继承类的方法