spark打包jar连接hive数据源完整过程

1：启动hive.metastore

nohup hive --service metastore &

2：启动spark中thriftserver
说明：经过测试如果直接启动hive中的hiveserver2然后启动thriftserver会报冲突，猜测spark中已经包含了hiveserver2
所以无需在hive中启动hiveserver2，主要目的是方便在datagrid中远程访问如图：

./start-thriftserver.sh

3：启动spark master,和slaver主要用于打包好的jar包执行

./start-master.sh -h 本机IP地址
./start-slave.sh spark://本机IP地址:7077

查看是否启动成功 jps -m 如下图所示

4：编写java测试代码

public class SparkSqlToHive {
    public static void main(String[] args) {
        SparkSession session = SparkSession.builder().appName("SparkSessionApp")
                .config("hive.metastore.uris", "thrift://127.0.0.1:9083")
                //直接连接hive
                .enableHiveSupport()
                .getOrCreate();

        session.sql("show databases").show();
    }
}

注意如果事先在开发工具中开发打jar包麻烦，可以直接远程连接调试。代码如下

public class SparkSqlToHive {
    public static void main(String[] args) {
        SparkSession session = SparkSession.builder().appName("SparkSessionApp")
                .master("spark://172.10.70.196:7077")
                .config("hive.metastore.uris", "thrift://172.10.70.196:9083")
                //直接连接hive
                .enableHiveSupport()
                .getOrCreate();

        session.sql("show databases").show();
    }
}

5：打包 pom文件实例

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>SparkTest</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>SparkTest</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.4.8</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.11</artifactId>
      <version>2.4.8</version>
    </dependency>
    <dependency>
      <groupId>org.spark-project.hive</groupId>
      <artifactId>hive-jdbc</artifactId>
      <version>1.2.1.spark2</version>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>net.alchim31.maven</groupId>
          <artifactId>scala-maven-plugin</artifactId>
          <version>3.2.2</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.5.1</version>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <executions>
          <execution>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

6：将打包好的jar包上传服务器运行

spark-submit --class org.example.SparkSqlToHive --master spark://第三步启动master指定的IP地址:7077 /usr/local/src/test/SparkTest-1.0-SNAPSHOT.jar

7：查看执行结果