试用Hive on Spark

试用Hive on Spark

Hive-1.2.0发布已经有几天了，下载后发现hive.execution.engine新增了spark选项，于是配置spark环境，试试Hive on Spark。

配置Hive

配置hive-site.xml

<configuration>

<property>
  <name>javax.jdo.option.ConnectionURLname>
  <value>jdbc:mysql://n1:3306/hive12mb?createDatabaseIfNotExist=truevalue>
  <description>JDBC connect string for a JDBC metastoredescription>
property>

<property>
  <name>javax.jdo.option.ConnectionDriverNamename>
  <value>com.mysql.jdbc.Drivervalue>
  <description>Driver class name for a JDBC metastoredescription>
property>

<property>
  <name>javax.jdo.option.ConnectionUserNamename>
  <value>rootvalue>
  <description>username to use against metastore databasedescription>
property>

<property>
  <name>javax.jdo.option.ConnectionPasswordname>
  <value>123456value>
  <description>password to use against metastore databasedescription>
property>

<property>
  <name>hive.execution.enginename>
  <value>sparkvalue>
property>

<property>
  <name>hive.stats.dbconnectionstringname>
  <value>jdbc:mysql://n1:3306/hive_stats?createDatabaseIfNotExist=true&user=root&password=123456value>
  <description>The default connection string for the database that stores temporary hive statistics.description>
property>


configuration>

编译Spark

下载最新Spark代码，修改pom.xml文件并编译

  
    <module>core</module>
    <module>bagel</module>
    <module>graphx</module>
    <module>mllib</module>
    <module>tools</module>
    <module>network/common</module>
    <module>network/shuffle</module>
    <module>streaming</module>
    <module>sql/catalyst</module>
    <module>sql/core</module>
    
    <module>unsafe</module>
    <module>assembly</module>
    
    <module>repl</module>
    <module>launcher</module>
  </modules>

mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package

使用新编译的assembly/target/scala-2.10/spark-assembly-1.4.0-SNAPSHOT-hadoop2.6.0.jar 文件安装Spark

一切就绪后启动各个组件

start-dfs.sh
start-yarn.sh 
{SPARK_HOME}/sbin/start-all.sh

进入hive后操作如下：

第一次启动时在Spark上启动application，所以第一条SQL比较慢，退出hive，spark application结束

---