hive 大数据量 参数设置

hive> set mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
hive> set mapred.map.output.compression.codec;
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec

hive> set hive.exec.compress.intermediate=true;

hive> set mapred.output.compression.type=BLOCK;

hive> set hive.exec.compress.output=true;

hive> set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;

hive> create table china_compress row format delimited fields terminated by '\t' stored as sequencefile as select * from china;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = hadoop_20170502105238_2b0ec41d-4dac-4431-8b4c-76a63966dbfa
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1493689799868_0001, Tracking URL = http://localhost:8088/proxy/application_1493689799868_0001/
Kill Command = /home/hadoop/soft/hadoop-2.6.5/bin/hadoop job  -kill job_1493689799868_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2017-05-02 10:53:11,460 Stage-1 map = 0%,  reduce = 0%
2017-05-02 10:53:21,977 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.9 sec
MapReduce Total cumulative CPU time: 1 seconds 900 msec
Ended Job = job_1493689799868_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/.hive-staging_hive_2017-05-02_10-52-38_066_3443755760076983254-1/-ext-10002
Moving data to directory hdfs://localhost:9000/user/hive/warehouse/china_compress
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 1.9 sec   HDFS Read: 154261 HDFS Write: 28734 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 900 msec
OK
Time taken: 46.494 seconds

hive> select * from china_compress limit 10;
OK
1	北京市	1	北京市	100000	1	东城区
1	北京市	1	北京市	100000	2	西城区
1	北京市	1	北京市	100000	3	崇文区
1	北京市	1	北京市	100000	4	宣武区
1	北京市	1	北京市	100000	5	朝阳区
1	北京市	1	北京市	100000	6	丰台区
1	北京市	1	北京市	100000	7	石景山区
1	北京市	1	北京市	100000	8	海淀区
1	北京市	1	北京市	100000	9	门头沟区
1	北京市	1	北京市	100000	10	房山区
Time taken: 0.133 seconds, Fetched: 10 row(s)