hive中对lzo压缩文件建立索引实现并行处理

1,确保创建索引

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.10.jar  com.hadoop.compression.lzo.DistributedLzoIndexer /user/hive/warehouse/flog

2 如果在hive中新建外部表的语句为

CREATE EXTERNAL TABLE foo (
         columnA string,
         columnB string )
    PARTITIONED BY (date string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t"
    STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
          OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
    LOCATION '/path/to/hive/tables/foo';

3  对于已经存在的表修改语句为

ALTER TABLE foo
    SET FILEFORMAT
        INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
        OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

4 alter table后对已经load进表中的数据，需要重新load和创建索引，要不还是不能分块

5 用hadoop streaming编程执行mapreduce作业语句为

hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/contrib/streaming/hadoop-0.20.2-dev-streaming.jar -file /home/pyshell/map.py -file /home/pyshell/red.py  -mapper /home/pyshell/map.py -reducer /home/pyshell/red.py -input /aojianlog/20120304/gold/gold_38_3.csv.lzo -output /aojianresult/gold38 -inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat -jobconf mapred.output.compress=true -jobconf mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec

注意 如果没有-inputformat com.hadoop.mapred.DeprecatedLzoTextInputFormat 选项的话map作业也不会分片

没有-jobconf mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec选项只设置-jobconf mapred.output.compress=true 选项的话 reduce作业输出文件的格式为.lzo_deflate