hive测试&&生产环境搭建

测试环境搭建

安装

https://hive.apache.org/downloads.html
因为需要使用Phoenix Storage Handler,所以选择支持hadoop2.x.y的 1.2+版本,hive1.2.2

配置环境变量

~/.bash_profile 配置hadoop&&hive变量

export HADOOP_HOME="/Users/titengjiang/Documents/DeveloperTool/hadoop/hadoop-2.7.5"
export PATH="$HADOOP_HOME/bin:$PATH"
export HIVE_HOME="/Users/titengjiang/Documents/DeveloperTool/hive/apache-hive-1.2.2-bin"
export PATH="$HIVE_HOME/bin:$PATH"

配置hive

mv hive-default.xml.template hive-site.xml

hive-site.xml增加如下配置

  
    system:java.io.tmpdir
    /Users/titengjiang/Documents/DeveloperTool/hive/local/tmp
  
  
    system:user.name
    ${user.name}
  

解决如下异常

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

创建warehouse

  $ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
  $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

初始化mysql metastore

create schema metastore_db default character set utf8;
CREATE USER hive@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'%' identified by 'hive' ;
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'localhost identified by 'hive';
 flush privileges;­

配置metastore,使用推荐的mysql替换derby

  
  
    javax.jdo.option.ConnectionURL
    jdbc:mysql://localhost:3306/metastore_db?useSSL=false
    
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/dbName?ssl=true for postgres database.
    
  

  
    javax.jdo.option.ConnectionDriverName
    com.mysql.jdbc.Driver
    Driver class name for a JDBC metastore
  

  
    javax.jdo.option.ConnectionUserName
    hive
    Username to use against metastore database
  
  
  
    javax.jdo.option.ConnectionPassword
    hive
    password to use against metastore database
  

拷贝mysql-connection-version.jar 到hive lib下并初始化metastore_db

注意mysql-connection的版本选择5.1.34最好，较低版本可能会遇到错误

$HIVE_HOME/bin/schematool -dbType mysql -initSchema

如果出现上面的错误,检查下mysql 用户权限，连接之类的配置

  
    hive.metastore.warehouse.dir
    /user/hive/warehouse
    location of default database for the warehouse
    

配置日志

hive-log4j.properties

hive.log.dir=/Users/username/Documents/DeveloperTool/hive/local/logs/${user.name} //日志路径改成指定路径

启动

$HIVE_HOME/bin/hiveserver2
hive --service metastore //启动metatore 提供hcatalog streaming写入端口 9083默认

常见错误

For direct MetaStore DB connections, we don’t support retries at the client level

create table years (year string, event string) row format delimited fields terminated by '\t';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)

修改mysql-connection版本