Hadoop压缩配置

下载LZO支持包

解压

maven编译

将编译好后的hadoop-lzo-0.4.20.jar 放入hadoop-2.7.2/share/hadoop/common/

同步到其他主机

添加配置core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

</configuration>

同步core-site.xml

启动hadoop脚本

这只是我高可用Hadoop的启动脚本，这一步可选

#! /bin/bash
echo "--------start journalNode----------"
for i in slave01 slave02 slave03
do
        ssh $i "/usr/local/hadoop-2.8.5/bin/hadoop-deamon.sh start journalnode"
done

echo "--------start dfs---------"
/usr/local/hadoop-2.8.5/sbin/start-dfs.sh
/usr/local/hadoop-2.8.5/sbin/start-yarn.sh