Hive遇到的坑点
1.登录hive客户端，查询表信息出现以下错误
hive (hive_db)> select * from star;
FAILED: SemanticException Unable to determine if hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/star is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/star, expected: hdfs://bigdata11:9000
查询报错信息后，发现是原先修改了集群namenode的端口号，导致hive在hdfs路径不一致的问题
2.解决方案 修改hive的元数据库MySQL
hive元数据库MySQL存储了hive的元数据信息，以下修改DBS、SDS两个表
DBS：存储了hive的database信息
SDS：存储输入输出format等信息，包括表的format和分区的format。关联字段CD_ID,SERDE_ID
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| metastore |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
mysql> use metastore;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_metastore |
+---------------------------+
| BUCKETING_COLS |
| CDS |
| COLUMNS_V2 |
| DATABASE_PARAMS |
| DBS |
| DB_PRIVS |
| FUNCS |
| FUNC_RU |
| GLOBAL_PRIVS |
| IDXS |
| INDEX_PARAMS |
| PARTITIONS |
| PARTITION_KEYS |
| PARTITION_KEY_VALS |
| PARTITION_PARAMS |
| PART_COL_PRIVS |
| PART_COL_STATS |
| PART_PRIVS |
| ROLES |
| SDS |
| SD_PARAMS |
| SEQUENCE_TABLE |
| SERDES |
| SERDE_PARAMS |
| SKEWED_COL_NAMES |
| SKEWED_COL_VALUE_LOC_MAP |
| SKEWED_STRING_LIST |
| SKEWED_STRING_LIST_VALUES |
| SKEWED_VALUES |
| SORT_COLS |
| TABLE_PARAMS |
| TAB_COL_STATS |
| TBLS |
| TBL_COL_PRIVS |
| TBL_PRIVS |
| VERSION |
+---------------------------+
36 rows in set (0.00 sec)
2.1 修改DBS
mysql> mysql> select * from DBS;
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
| DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE |
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
| 1 | Default Hive database | hdfs://bigdata11:50090/user/hive/warehouse | default | public | ROLE |
| 6 | NULL | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db | hive_db | root | USER |
| 16 | NULL | hdfs://bigdata11:50090/user/hive/warehouse/hive_bucket.db | hive_bucket | root | USER |
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
3 rows in set (0.00 sec)
mysql> update DBS set DB_LOCATION_URI="hdfs://bigdata11:9000/user/hive/warehouse" where DB_ID=1;
Query OK, 1 row affected (0.13 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> update DBS set DB_LOCATION_URI=" hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db" where DB_ID=6;
Query OK, 1 row affected (0.04 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> update DBS set DB_LOCATION_URI=" hdfs://bigdata11:9000/user/hive/warehouse/hive_bucket.db" where here DB_ID=16;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> mysql> select * from DBS;
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
| DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE |
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
| 1 | Default Hive database | hdfs://bigdata11:9000/user/hive/warehouse | default | public | ROLE |
| 6 | NULL | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db | hive_db | root | USER |
| 16 | NULL | hdfs://bigdata11:9000/user/hive/warehouse/hive_bucket.db | hive_bucket | root | USER |
+-------+-----------------------+-----------------------------------------------------------+-------------+------------+------------+
3 rows in set (0.00 sec)
2.2修改SDS
一键更新表元数据hdfs的路径：
mysql> update SDS set LOCATION=replace(LOCATION,'bigdata11:50090','bigdata11:9000');
mysql> mysql> select * from SDS;
+-------+-------+------------------------------------------+---------------+---------------------------+----------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
| SD_ID | CD_ID | INPUT_FORMAT | IS_COMPRESSED | IS_STOREDASSUBDIRECTORIES | LOCATION | NUM_BUCKETS | OUTPUT_FORMAT | SERDE_ID |
+-------+-------+------------------------------------------+---------------+---------------------------+----------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
| 6 | 6 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/stuinfo | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 6 |
| 8 | 8 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/star | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 8 |
| 12 | 12 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/information | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 12 |
| 17 | 17 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/travelist | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 17 |
| 23 | 23 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/emptable | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 23 |
| 26 | 26 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/dept | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 26 |
| 33 | 33 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_db.db/bucket | 5 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 33 |
| 37 | 37 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_bucket.db/data | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 37 |
| 41 | 41 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:50090/user/hive/warehouse/hive_bucket.db/travelist_bucket | 1000 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 41 |
+-------+-------+------------------------------------------+---------------+---------------------------+----------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
9 rows in set (0.00 sec)
mysql> update SDS set LOCATION=replace(LOCATION,'bigdata11:50090','bigdata11:9000');
Query OK, 9 rows affected (0.12 sec)
Rows matched: 9 Changed: 9 Warnings: 0
mysql> select * from SDS;
+-------+-------+------------------------------------------+---------------+---------------------------+---------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
| SD_ID | CD_ID | INPUT_FORMAT | IS_COMPRESSED | IS_STOREDASSUBDIRECTORIES | LOCATION | NUM_BUCKETS | OUTPUT_FORMAT | SERDE_ID |
+-------+-------+------------------------------------------+---------------+---------------------------+---------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
| 6 | 6 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/stuinfo | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 6 |
| 8 | 8 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/star | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 8 |
| 12 | 12 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/information | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 12 |
| 17 | 17 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/travelist | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 17 |
| 23 | 23 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/emptable | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 23 |
| 26 | 26 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/dept | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 26 |
| 33 | 33 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_db.db/bucket | 5 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 33 |
| 37 | 37 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_bucket.db/data | -1 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 37 |
| 41 | 41 | org.apache.hadoop.mapred.TextInputFormat | | | hdfs://bigdata11:9000/user/hive/warehouse/hive_bucket.db/travelist_bucket | 1000 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | 41 |
+-------+-------+------------------------------------------+---------------+---------------------------+---------------------------------------------------------------------------+-------------+------------------------------------------------------------+----------+
9 rows in set (0.00 sec)
3.修改后，查询hive表信息
hive (hive_db)> select * from travelist limit 5;
OK
travelist.province travelist.city travelist.name travelist.level travelist.num travelist.price travelist.hot travelist.descri
辽宁 大连 大连圣亚海洋世界 4 96.0 342.0 0.82 乐游星海湾，大连浪漫地标
辽宁 沈阳 沈阳清河半岛温泉度假酒店 0 159.0 254.0 0.0 休闲度假，亲子娱乐好去处
辽宁 鞍山 鞍山千山 5 27.0 174.0 0.9 无穷无尽的天然画卷
辽宁 沈阳 沈阳怪坡国际滑雪场 4 105.0 42.0 0.0 群山环绕，雪海苍茫，滑雪设施专业齐全
辽宁 大连 老虎滩海洋公园 5 50.0 32.0 0.9 浓缩极地世界，展现海洋奇观
Time taken: 0.144 seconds, Fetched: 5 row(s)