HBase配置LZO压缩

阅读更多

系统：  gentoo

HDFS:  hadoop:hadoop-0.20.2-cdh3u0
HBase: HBase:0.90.3

 

 

安装过程：

 

1、安装lzo

emerge dev-libs/lzo

2.1 下载

download from:

 

wget http://www.apache.org/dist//hbase/hbase-0.90.3/hbase-0.90.3.tar.gz
wget http://hadoop-gpl-compression.apache-extras.org.codespot.com/files/hadoop-gpl-compression-0.1.0-rc0.tar.gz

2.2 编译

tar -zxvf hadoop-gpl-compression-0.1.0-rc0.tar.gz
cd hadoop-gpl-compression-0.1.0
cp $HADOOP_HOME//hadoop-core-0.20.2-cdh3u0.jar lib/

nano -w src/java/com/hadoop/compression/lzo/LzoCompressor.java

......
import org.apache.hadoop.conf.Configuration;
.....

    public void reinit(Configuration conf) { 
      // do nothing 
    }  

ant compile-native tar

 

2.3 拷贝需要的文件
cp build/hadoop-gpl-compression-0.1.0-dev.jar $HADOOP_HOME/lib/
cp build/hadoop-gpl-compression-0.1.0-dev.jar $HBASE_HOME/lib/

rsync -auPv lib/native $HADOOP_HOME/lib/
rsync -auPv lib/native $HBASE_HOME/lib/

 

3.修改配置重启HBase和Hadoop
nano core-site.xml

 
    io.compression.codecs 
 com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec 
     

  
    io.compression.codec.lzo.class 
    com.hadoop.compression.lzo.LzoCodec 
     

 

4 测试是否成功
bin/hbase   org.apache.hadoop.hbase.util.CompressionTest hdfs://test1.namenode.uc.uuwatch.com:9000/test_path lzo


5 参考资料

http://wiki.apache.org/hadoop/UsingLzoCompression
http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport

http://koven2049.iteye.com/blog/983502