Hive MetaStore简单剖析解析,持续整理中...

MetaStore

1.DBS :存数据库信息

mysql> select * from DBS \G;
*************************** 1. row ***************************
DB_ID: 1
DESC: Default Hive database
DB_LOCATION_URI: hdfs://hadoop001:8020/user/hive/warehouse
NAME: default
OWNER_NAME: public
OWNER_TYPE: ROLE
1 row in set (0.07 sec)

再创建一个数据库
hive (default)> create database ruoze_g5;
OK
Time taken: 2.907 seconds
hive (default)> show databases;
OK
default
ruoze_g5

再次查看
mysql> select * from DBS \G;
*************************** 1. row ***************************
DB_ID: 1
DESC: Default Hive database
DB_LOCATION_URI: hdfs://hadoop001:8020/user/hive/warehouse
NAME: default
OWNER_NAME: public
OWNER_TYPE: ROLE
*************************** 2. row ***************************
DB_ID: 6
DESC: NULL
DB_LOCATION_URI: hdfs://hadoop001:8020/user/hive/warehouse/ruoze_g5.db
NAME: ruoze_g5
OWNER_NAME: hadoop
OWNER_TYPE: USER
2 rows in set (0.00 sec)

mysql> select * from version ;

±-------±---------------±--------------------------------------+
| VER_ID | SCHEMA_VERSION | VERSION_COMMENT |
±-------±---------------±--------------------------------------+
| 1 | 1.1.0 | Set by MetaStore [email protected] |
±-------±---------------±--------------------------------------+
1 row in set (0.00 sec)

2.version 有且只有一条 hive才能起来

mysql> select * from version ;
±-------±---------------±--------------------------------------+
| VER_ID | SCHEMA_VERSION | VERSION_COMMENT |
±-------±---------------±--------------------------------------+
| 1 | 1.1.0 | Set by MetaStore [email protected] |
±-------±---------------±--------------------------------------+
1 row in set (0.00 sec)

3.TBLS 存放表

mysql> select * from TBLS ;
±-------±------------±------±-----------------±-------±----------±------±----------------------------------±--------------±-------------------±-------------------+
| TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER | RETENTION | SD_ID | TBL_NAME | TBL_TYPE | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT |
±-------±------------±------±-----------------±-------±----------±------±----------------------------------±--------------±-------------------±-------------------+
| 1 | 1540815024 | 1 | 0 | hadoop | 0 | 1 | user_click | MANAGED_TABLE | NULL | NULL |
| 2 | 1540820859 | 1 | 0 | hadoop | 0 | 2 | tmp_product_basic_info2 | MANAGED_TABLE | NULL | NULL |
| 11 | 1540820386 | 1 | 0 | hadoop | 0 | 11 | product_info | MANAGED_TABLE | NULL | NULL |
| 16 | 1540820609 | 1 | 0 | hadoop | 0 | 16 | city_info | MANAGED_TABLE | NULL | NULL |
| 26 | 1540822404 | 1 | 0 | hadoop | 0 | 26 | tmp_product_basic_info | MANAGED_TABLE | NULL | NULL |
| 27 | 1540822474 | 1 | 0 | hadoop | 0 | 27 | tmp_product_basic_count_info | MANAGED_TABLE | NULL | NULL |
| 28 | 1540822517 | 1 | 0 | hadoop | 0 | 28 | tmp_product_basic_count_info_full | MANAGED_TABLE | NULL | NULL |
| 30 | 1540823286 | 1 | 0 | hadoop | 0 | 30 | area_product_top3 | MANAGED_TABLE | NULL | NULL |
| 31 | 1543634189 | 1 | 0 | hadoop | 0 | 36 | page_views | MANAGED_TABLE | NULL | NULL |
| 33 | 1543635099 | 1 | 0 | hadoop | 0 | 38 | page_views_bzip2 | MANAGED_TABLE | NULL | NULL |

DB_ID
TB_ID

4. table_params :存放表里的列等信息
mysql> select * from table_params \G

*************************** 1. row ***************************
TBL_ID: 1
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540815024
*************************** 2. row ***************************
TBL_ID: 2
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 3. row ***************************
TBL_ID: 2
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 4. row ***************************
TBL_ID: 2
PARAM_KEY: numRows
PARAM_VALUE: 0
*************************** 5. row ***************************
TBL_ID: 2
PARAM_KEY: rawDataSize
PARAM_VALUE: 0
*************************** 6. row ***************************
TBL_ID: 2
PARAM_KEY: totalSize
PARAM_VALUE: 0
*************************** 7. row ***************************
TBL_ID: 2
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540820859
*************************** 8. row ***************************
TBL_ID: 11
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 9. row ***************************
TBL_ID: 11
PARAM_KEY: comment
PARAM_VALUE: Imported by sqoop on 2018/10/29 06:39:36
*************************** 10. row ***************************
TBL_ID: 11
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 11. row ***************************
TBL_ID: 11
PARAM_KEY: numRows
PARAM_VALUE: 0
*************************** 12. row ***************************
TBL_ID: 11
PARAM_KEY: rawDataSize
PARAM_VALUE: 0
*************************** 13. row ***************************
TBL_ID: 11
PARAM_KEY: totalSize
PARAM_VALUE: 3384
*************************** 14. row ***************************
TBL_ID: 11
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540820386
*************************** 15. row ***************************
TBL_ID: 16
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 16. row ***************************
TBL_ID: 16
PARAM_KEY: comment
PARAM_VALUE: Imported by sqoop on 2018/10/29 06:43:20
*************************** 17. row ***************************
TBL_ID: 16
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 18. row ***************************
TBL_ID: 16
PARAM_KEY: numRows
PARAM_VALUE: 0
*************************** 19. row ***************************
TBL_ID: 16
PARAM_KEY: rawDataSize
PARAM_VALUE: 0
*************************** 20. row ***************************
TBL_ID: 16
PARAM_KEY: totalSize
PARAM_VALUE: 128
*************************** 21. row ***************************
TBL_ID: 16
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540820610
*************************** 22. row ***************************
TBL_ID: 26
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 23. row ***************************
TBL_ID: 26
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 24. row ***************************
TBL_ID: 26
PARAM_KEY: numRows
PARAM_VALUE: 10369
*************************** 25. row ***************************
TBL_ID: 26
PARAM_KEY: rawDataSize
PARAM_VALUE: 71512
*************************** 26. row ***************************
TBL_ID: 26
PARAM_KEY: totalSize
PARAM_VALUE: 81881
*************************** 27. row ***************************
TBL_ID: 26
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540822404
*************************** 28. row ***************************
TBL_ID: 27
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 29. row ***************************
TBL_ID: 27
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 30. row ***************************
TBL_ID: 27
PARAM_KEY: numRows
PARAM_VALUE: 600
*************************** 31. row ***************************
TBL_ID: 27
PARAM_KEY: rawDataSize
PARAM_VALUE: 4600
*************************** 32. row ***************************
TBL_ID: 27
PARAM_KEY: totalSize
PARAM_VALUE: 5200
*************************** 33. row ***************************
TBL_ID: 27
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540822474
*************************** 34. row ***************************
TBL_ID: 28
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 35. row ***************************
TBL_ID: 28
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 36. row ***************************
TBL_ID: 28
PARAM_KEY: numRows
PARAM_VALUE: 594
*************************** 37. row ***************************
TBL_ID: 28
PARAM_KEY: rawDataSize
PARAM_VALUE: 8718
*************************** 38. row ***************************
TBL_ID: 28
PARAM_KEY: totalSize
PARAM_VALUE: 9312
*************************** 39. row ***************************
TBL_ID: 28
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540822518
*************************** 40. row ***************************
TBL_ID: 30
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1540823286
*************************** 41. row ***************************
TBL_ID: 31
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 42. row ***************************
TBL_ID: 31
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 43. row ***************************
TBL_ID: 31
PARAM_KEY: numRows
PARAM_VALUE: 0
*************************** 44. row ***************************
TBL_ID: 31
PARAM_KEY: rawDataSize
PARAM_VALUE: 0
*************************** 45. row ***************************
TBL_ID: 31
PARAM_KEY: totalSize
PARAM_VALUE: 19014993
*************************** 46. row ***************************
TBL_ID: 31
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1543634355
*************************** 47. row ***************************
TBL_ID: 33
PARAM_KEY: COLUMN_STATS_ACCURATE
PARAM_VALUE: true
*************************** 48. row ***************************
TBL_ID: 33
PARAM_KEY: numFiles
PARAM_VALUE: 1
*************************** 49. row ***************************
TBL_ID: 33
PARAM_KEY: numRows
PARAM_VALUE: 100000
*************************** 50. row ***************************
TBL_ID: 33
PARAM_KEY: rawDataSize
PARAM_VALUE: 18490893
*************************** 51. row ***************************
TBL_ID: 33
PARAM_KEY: totalSize
PARAM_VALUE: 3722768
*************************** 52. row ***************************
TBL_ID: 33
PARAM_KEY: transient_lastDdlTime
PARAM_VALUE: 1543635099
52 rows in set (0.00 sec)

12 rows in set (0.00 sec)

5.SDS :表存储文件信息

mysql> select * from SDS \G;
*************************** 1. row ***************************
SD_ID: 1
CD_ID: 1
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/user_click
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 1
*************************** 2. row ***************************
SD_ID: 2
CD_ID: 2
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/tmp_product_basic_info2
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 2
*************************** 3. row ***************************
SD_ID: 11
CD_ID: 11
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/product_info
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 11
*************************** 4. row ***************************
SD_ID: 16
CD_ID: 16
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/city_info
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 16
*************************** 5. row ***************************
SD_ID: 22
CD_ID: 1
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/user_click/date=2016-05-05
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 22
*************************** 6. row ***************************
SD_ID: 26
CD_ID: 26
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/tmp_product_basic_info
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 26
*************************** 7. row ***************************
SD_ID: 27
CD_ID: 27
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/tmp_product_basic_count_info
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 27
*************************** 8. row ***************************
SD_ID: 28
CD_ID: 28
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/tmp_product_basic_count_info_full
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 28
*************************** 9. row ***************************
SD_ID: 30
CD_ID: 30
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/area_product_top3
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 30
*************************** 10. row ***************************
SD_ID: 31
CD_ID: 30
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/area_product_top3/date=2016-05-05
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 31
*************************** 11. row ***************************
SD_ID: 36
CD_ID: 31
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/page_views
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 36
*************************** 12. row ***************************
SD_ID: 38
CD_ID: 33
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/page_views_bzip2
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 38
12 rows in set (0.00 sec)

6.CDS

mysql> select * from CDS \G;
*************************** 1. row ***************************
CD_ID: 1
*************************** 2. row ***************************
CD_ID: 2
*************************** 3. row ***************************
CD_ID: 11
*************************** 4. row ***************************
CD_ID: 16
*************************** 5. row ***************************
CD_ID: 26
*************************** 6. row ***************************
CD_ID: 27
*************************** 7. row ***************************
CD_ID: 28
*************************** 8. row ***************************
CD_ID: 30
*************************** 9. row ***************************
CD_ID: 31
*************************** 10. row ***************************
CD_ID: 33
10 rows in set (0.00 sec)

7.serdes :序列化信息

mysql> select * from serdes \G;
*************************** 1. row ***************************
SERDE_ID: 1
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 2. row ***************************
SERDE_ID: 2
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 3. row ***************************
SERDE_ID: 11
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 4. row ***************************
SERDE_ID: 16
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 5. row ***************************
SERDE_ID: 22
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 6. row ***************************
SERDE_ID: 26
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 7. row ***************************
SERDE_ID: 27
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 8. row ***************************
SERDE_ID: 28
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 9. row ***************************
SERDE_ID: 30
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 10. row ***************************
SERDE_ID: 31
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 11. row ***************************
SERDE_ID: 36
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
*************************** 12. row ***************************
SERDE_ID: 38
NAME: NULL
SLIB: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
12 rows in set (0.00 sec)

ERROR:
No query specified

8.columns_v2

mysql> select * from columns_v2;
±------±--------±------------------±----------±------------+
| CD_ID | COMMENT | COLUMN_NAME | TYPE_NAME | INTEGER_IDX |
±------±--------±------------------±----------±------------+
| 1 | NULL | action_time | string | 2 |
| 1 | NULL | city_id | int | 3 |
| 1 | NULL | product_id | int | 4 |
| 1 | NULL | session_id | string | 1 |
| 1 | NULL | user_id | int | 0 |
| 2 | NULL | area | string | 2 |
| 2 | NULL | city_id | int | 0 |
| 2 | NULL | product_id | int | 1 |
| 11 | NULL | extend_info | string | 2 |
| 11 | NULL | product_id | int | 0 |
| 11 | NULL | product_name | string | 1 |
| 16 | NULL | area | string | 2 |
| 16 | NULL | city_id | int | 0 |
| 16 | NULL | city_name | string | 1 |
| 26 | NULL | area | string | 2 |
| 26 | NULL | city_id | int | 0 |
| 26 | NULL | product_id | int | 1 |
| 27 | NULL | area | string | 1 |
| 27 | NULL | click_count | bigint | 2 |
| 27 | NULL | product_id | int | 0 |
| 28 | NULL | area | string | 1 |
| 28 | NULL | click_count | bigint | 2 |
| 28 | NULL | product_name | string | 0 |
| 30 | NULL | area | string | 1 |
| 30 | NULL | click_count | int | 2 |
| 30 | NULL | click_count_grade | string | 4 |
| 30 | NULL | day | string | 5 |
| 30 | NULL | product_name | string | 0 |
| 30 | NULL | rank | int | 3 |
| 31 | NULL | city_id | string | 5 |
| 31 | NULL | end_user_id | string | 4 |
| 31 | NULL | referer | string | 3 |
| 31 | NULL | session_id | string | 2 |
| 31 | NULL | track_time | string | 0 |
| 31 | NULL | url | string | 1 |
| 33 | NULL | city_id | string | 5 |
| 33 | NULL | end_user_id | string | 4 |
| 33 | NULL | referer | string | 3 |
| 33 | NULL | session_id | string | 2 |
| 33 | NULL | track_time | string | 0 |
| 33 | NULL | url | string | 1 |
±------±--------±------------------±----------±------------+
41 rows in set (0.00 sec)

INTEGER_IDX: 字段位置

9: partitions 分区

mysql> select * from partitions;
±--------±------------±-----------------±----------------±------±-------+
| PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
±--------±------------±-----------------±----------------±------±-------+
| 1 | 1540821280 | 0 | date=2016-05-05 | 22 | 1 |
| 6 | 1540823524 | 0 | date=2016-05-05 | 31 | 30 |
±--------±------------±-----------------±----------------±------±-------+

10.partition_keys

ERROR 1146 (42S02): Table ‘ruoze_d5.partitions_keys’ doesn’t exist
mysql> select * from partition_keys;
±-------±-------------±----------±----------±------------+
| TBL_ID | PKEY_COMMENT | PKEY_NAME | PKEY_TYPE | INTEGER_IDX |
±-------±-------------±----------±----------±------------+
| 1 | NULL | date | string | 0 |
| 30 | NULL | date | string | 0 |
±-------±-------------±----------±----------±------------+

11.partition_keys

mysql> select * from partition_keys;
±--------±-------------±------------+
| PART_ID | PART_KEY_VAL | INTEGER_IDX |
±--------±-------------±------------+
| 1 | 2016-05-05 | 0 |
| 6 | 2016-05-05 | 0 |
±--------±-------------±------------+

表结果ER图