hive报错Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE)

业务发现跑hive任务总是一次成功,第二次无法找到库,需要重新进行hive,查看HiveMetaStore日志报错:

2016-05-11 11:30:32,912 ERROR [pool-3-thread-178]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(155)) - Retrying HMSHandler after 1000 ms (attempt 1 of 1) with error: javax.jdo.JDOException: Exception thrown when executing query

    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)

NestedThrowablesStackTrace:
java.sql.SQLException: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='

这个错误什么原因,去查文档说是:结果集中有两种字符集。

mysql> SHOW VARIABLES LIKE 'character_set_%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | latin1                     |
| character_set_connection | latin1                     |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | latin1                     |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)


mysql> SHOW global VARIABLES LIKE 'character_set_%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'collation_%';
+----------------------+-------------------+
| Variable_name        | Value             |
+----------------------+-------------------+
| collation_connection | latin1_swedish_ci |
| collation_database   | utf8_general_ci   |
| collation_server     | utf8_general_ci   |
+----------------------+-------------------+
3 rows in set (0.00 sec)


mysql> SHOW global VARIABLES LIKE 'collation_%';
+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database   | utf8_general_ci |
| collation_server     | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)

发现global和当前变量不同,应该是my.cnf文件需要设置下。

mysql和字符集有关的变量
character_set_client:客户端请求数据的字符集
character_set_connection:从客户端接收到数据,然后传输的字符集
character_set_database:默认数据库的字符集,无论默认数据库如何改变,都是这个字符集;如果没有默认数据库,那就使用 character_set_server指定的字符集,这个变量建议由系统自己管理,不要人为定义。

character_set_filesystem:把os上文件名转化成此字符集,即把 character_set_client转换character_set_filesystem, 默认binary是不做任何转换的
character_set_results:结果集的字符集
character_set_server:数据库服务器的默认字符集
character_set_system:这个值总是utf8,不需要设置,是为存储系统元数据的字符集



与业务人员沟通后了解到之前有人新建了中文字段的hive表文件,就是因为这个中文导致的问题,中文的问题就是编码的问题。

临时修改:
set character_set_client =utf8;
set character_set_connection =utf8;
set character_set_results =utf8;


SET collation_server = utf8_general_ci
SET collation_database = utf8_general_ci


永久修改:
[root@ambari03 etc]# cat /etc/my.cnf
[client]
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
init_connect='SET NAMES utf8'
user=mysql
symbolic-links=0
#skip-grant-tables
max_connections=2000
log-bin=mysql-bin
server-id=11
binlog_format = MIXED
[mysql]
no-auto-rehash
default-character-set=utf8

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

你可能感兴趣的:(Hive)