win10下使用hive时遇到的错误集锦

背景

老子不知道hive在win10下怎么就这么多错.....不想用虚拟机，想换成mac，但是mac上没有实况19...真的是，我去年买了个表。

 

hive建表报错Column length too big for column 'PARAM_VALUE' (max = 21845); use BLOB or TEXT instead

   解决方法：用navicat创建hive数据库，设置编码为latin1

不能和spark远程共享一张表

解决方法：1、把两者的hive-site.xml中hive.metastore.local的值改为false，hive.metastore.schema.verification值改为false

                  2、在spark的xml中加上一个配置

　

　   hive.metastore.uris

　　　thrift://localhost:9083

　　　Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.

　

ConnectionException:Call from ... to localhost:9000 failed on connection exception:Connection refused no further information

解决方法：先启动hadoop

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):user is not allowed to impersonate hive

解决方法：hadoophome\etc\hadoop\core-site.xml文件里，添加如下内容：

      hadoop.proxyuser.123.groups
      *
      Allow the superuser oozie to impersonate any members of the group group1 and group2
 


      hadoop.proxyuser.123.hosts
      *
      The superuser can connect only from host1 and host2 to impersonate a user

把123换成自己的用户名 ，然后重启hadoop和hiveserver

插入数据时org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10293]: Unable to create temp file for insert values Expression of type TOK_TABLE_OR_COL not supported in insert/values

 解决方法：

客户端插入时不要用

insert into table_name(id, name) values(1， "szc")

要用

insert into table_name select 1 id, "szc" name

修改删除数据时报错org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.

解决方法：在hiveHome\conf\hive-site.xml中，增加如下配置：

    hive.support.concurrency
    true
  
    
    hive.enforce.bucketing
    true
  
    
    hive.exec.dynamic.partition.mode
    nonstrict
  
  
    hive.txn.manager
    org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
  
    
    hive.compactor.initiator.on
    true
  
  
    hive.compactor.worker.threads
    1
  

修改删除数据时报错oorg.apache.hadoop.hive.ql.parse.SemanticException: Attempt to do update or delete on table default.student that does not use an AcidOutputFormat or is not bucketed

 采用这种方法建表：

create table teammates(id int ,name string， position string )clustered by (id) into 2 buckets stored as orc TBLPROPERTIES('transactional'='true');

Database access problem.Killing off this connection and all remaining connections in the connection pool SQL state = HY000

unable to transaction database Incorrect String value '\xE8\x80\x81\xE5\xAE\x8B...' from HL_HOST at row 1

解决方法： 把电脑名改成全英文，而后重启

Table 'sparktest.aux_table' doesn`t exist

解决方法：把sql脚本文件hiveHome\scripts\metastore\upgrade\mysql\hive-txn-schema-2.3.0.mysql.sql拖入Navicat中对应数据库里执行

schematool.cmd -dbType mysql -initSchema命令语法不正确。

解决方法同上，把sql文件拖入Navicat中执行

NestedThrowablesStackTrace:

java.lang.ClassCastException: org.datanucleus.identity.DatastoreIdImpl cannot be cast to org.datanucleus.identity.OID

at org.datanucleus.identity.OIDFactory.getInstance

解决方法：把sparkHome\jars\下的datanucleus-**.jar换成和hiveHome\lib\下对应的jar包

(datanucleus-api-jdo-3.2.6.jar、datanucleus-core-3.2.10.jar和datanucleus-rdbms-3.2.9.jar三个)

Exception in thread "org.apache.hadoop.util.JvmPauseMonitor$Monitor@1bd0e69"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.apache.hadoop.util.JvmPauseMonitor$Monitor@1bd0e69"

在文件hadoopHome\etc\hadoop\hadoop-env.cmd中，增大HADOOP_CLIENT_OPTS的值

set HADOOP_CLIENT_OPTS=4096

Failed:Error in acquiring locks:error communicating with the metastore

解决方法：查看hive日志，日志路径是hiveHome\conf\hive-log4j2.properties文件的property.hive.log.dir值，可用System.properties()获取java.io.tmpdir和user.name的值，组成路径切过去就是hive日志所在的目录

win10是C:\Users\用户名\AppData\Local\Temp\123,文件名为hive.log

打开后发现最近的异常如下

2019-06-11 11:58:36,756 ERROR [main]: ql.Driver (SessionState.java:printError(960)) - FAILED: Error in acquiring locks: Error communicating with the metastore
org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the metastore
    ....
Caused by: MetaException(message:Unable to update transaction database java.sql.SQLException: Field 'TC_OPERATION_TYPE' doesn't have a default value
    .....

说明communicating with metastore这个异常由tc_operation_type没有默认值引起，我们在mysql里找到简拼为tc的表txn_components(数据库是hiveHome\conf\hive-site.xml里javax.jdo.option.ConnectionURL指定的数据库)，用navicat打开数据库，右击txn_components表，点击设计表，在tc_operation_type一行的允许空值列打上勾，保存，重启hive即可

beeline连接hiveserver2报错：

org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?

解决方法：1、打开hiveHome\conf\hive-site.xml，加入如下配置

    hive.server2.transport.mode
    http



    hive.server2.thrift.http.port
    10001



    hive.server2.thrift.http.path
    cliservice

2、启动metastore

hive --service metastore &

 3、启动hiveserver2

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10001

 4、beeline连接hiveserver2

!connect jdbc:hive2://localhost:10001/hive;transportMode=http;httpPath=cliservice

以后我们用任何方式连接hive，都要用这个新url(jdbc:hive2://localhost:10001/hive;transportMode=http;httpPath=cliservice)了

metasore报错：com.mysql.cj.core.exceptions.InvalidConnectionAttributeException: The server time zone value '�й���׼ʱ��' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support.

解决方法：在mysql命令行下，改变时区设置

set global time_zone='+8:00';

然后重启metastore

结语

如果里面没有自己遇到的错误，可以看看我其他的关于win10下hive、hadoop报错的文章。

光环境就搭了半个月，我觉得大数据可比安卓水深多了...