Hive问题集锦 - Hadoop生态圈

Hive，是一个面向分析的存储系统。也就是所谓的数据仓库。其本质是将SQL转换为MapReduce的任务进行运算，底层由HDFS来提供数据的存储，可以将Hive理解为一个将SQL转换为MapReduce的任务的工具。

所以，为什么要用Hive？
1、采用类SQL语法去操作数据，提供快速开发的能力。
2、避免了去写MapReduce，减少开发人员的学习成本。
3、功能扩展很方便。

接下来一起来了解下怎么使用hive。

如何连接HIVE

使用HIVE命令行

使用/bin/hive进入hive命令行。

[root@crazyaya3 hive-2.1.1]# hive 
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/mnt/bigdata/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> 
hive> create database if not exists mytest;
OK
Time taken: 6.041 seconds
hive> show databases;
OK
default
mytest
Time taken: 0.393 seconds, Fetched: 2 row(s)

使用SHELL命令行

不进入hive的客户端直接执行hive的hql语句。

[root@crazyaya3 hive-2.1.1]# bin/hive -e "create database if not exists mytest;"
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/mnt/bigdata/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
OK
Time taken: 2.827 seconds

通过hive -f 来执行我们的sql脚本

[root@crazyaya3 script]# cat > /mnt/bigdata/script/hive.sql < create database if not exists mytest;
> use mytest;
> create table stu(id int,name string);
> EOF

执行后报错。这里要注意，创建hive的metastore的时候，需要指定数据库编码为latin1。在国内一般默认设置为utf-8，所以这个是比较常见的错误。参考：https://blog.csdn.net/zlx510tsde/article/details/49923651

[root@crazyaya3 hive-2.1.1]# bin/hive -f /mnt/bigdata/script/hive.sql 
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/mnt/bigdata/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
OK
Time taken: 2.845 seconds
OK
Time taken: 0.252 seconds
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:An exception was thrown while adding/validating class(es) : Specified key was too long; max key length is 767 bytes
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
    at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
    at com.mysql.jdbc.Util.getInstance(Util.java:387)

修改编码后运行成功。

[root@crazyaya3 hive-2.1.1]# bin/hive -f /mnt/bigdata/script/hive.sql 
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/mnt/bigdata/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
OK
Time taken: 1.99 seconds
OK
Time taken: 0.151 seconds
OK
Time taken: 1.813 seconds

打开50070检查一下。

Hadoop浏览界面

ok，没问题。

使用JDBC连接

参考以下文章：SQuirrel连接hive配置、Hive 之 Java API 操作、连接hiveserver2-Hive用户配置

HIVE基本操作问题

删库跑路

hive> drop database mytest1 ;
Moved: 'hdfs://crazyaya1:8020/hive/mytest1' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
OK
Time taken: 0.506 seconds

删库时，可能会报错。处理方式：需要初始化元数据库。

[root@crazyaya3 hive-2.1.1]# hive -e "drop database mytest2"
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/mnt/bigdata/hive-2.1.1/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Moved: 'hdfs://crazyaya1:8020/hive' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to clean up com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'hive.txn_components' doesn't exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
...

[root@crazyaya3 ~]# schematool -initSchema -dbType mysql
which: no hbase in (/usr/lib/jvm/java-1.8.0/bin:/mnt/bigdata/hadoop-2.7.5/bin:/mnt/bigdata/hive-2.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/bigdata/hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/bigdata/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:    jdbc:mysql://111.231.232.84:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:   root
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.mysql.sql
Initialization script completed
schemaTool completed

如果库里有数据表，则会报错不能删除。此时，需要级联删除(慎用)。

hive> drop database mytest2;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. InvalidOperationException(message:Database mytest2 is not empty. One or more tables exist.)
hive> drop database mytest2 cascade;
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest2.db/stu' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
Moved: 'hdfs://crazyaya1:8020/user/hive/warehouse/mytest2.db' to trash at: hdfs://crazyaya1:8020/user/root/.Trash/Current
OK
Time taken: 7.418 seconds

持续更新...