Hadoop2.6.0 Snappy 压缩安装与配置

先说点题外话，我们集群主要使用hive,Impala,spark sql等来处理数据，介于hdfs里面的数据没有压缩占用空间太大所以我们要对其进行压缩

1. snappy比较适中
2. snappy 原生被hive等支持

首先看一下你的集群是否支持/安装了snappy

hadoop checknative -a

6/07/08 14:20:48 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/07/08 14:20:48 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/docker/hadoop/apache/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so
zlib: true /lib64/libz.so.1
snappy: true /home/docker/hadoop/apache/hadoop-2.6.0-cdh5.7.0/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so

我们集群已经安装过了所以都是true 如果有false 就说明你对应的native库没有安装好

hadoop native lib的定义

Native Hadoop Library

Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.

你需要安装重新编译hadoop源码with native

编程完成后目录应该类似是这样的
[root@dev03 ~]# ll $HADOOP_HOME/lib/native
总用量 19716
-rw-r--r--. 1 hadoop hadoop 1309890 6月 28 12:11 libhadoop.a
-rw-r--r--. 1 hadoop hadoop 1485708 6月 28 12:11 libhadooppipes.a
-rwxr-xr-x. 1 hadoop hadoop 769368 6月 28 12:11 libhadoop.so
-rwxr-xr-x. 1 hadoop hadoop 769368 6月 28 12:11 libhadoop.so.1.0.0
-rw-r--r--. 1 hadoop hadoop 582056 6月 28 12:11 libhadooputils.a
-rw-r--r--. 1 hadoop hadoop 369310 6月 28 12:11 libhdfs.a
-rwxr-xr-x. 1 hadoop hadoop 235700 6月 28 12:11 libhdfs.so
-rwxr-xr-x. 1 hadoop hadoop 235700 6月 28 12:11 libhdfs.so.0.0.0
-rw-r--r--. 1 hadoop hadoop 7705772 6月 28 12:11 libnativetask.a
-rwxr-xr-x. 1 hadoop hadoop 3082245 6月 28 12:11 libnativetask.so
-rwxr-xr-x. 1 hadoop hadoop 3082245 6月 28 12:11 libnativetask.so.1.0.0
-rw-r--r--. 1 hadoop hadoop 233386 6月 28 12:11 libsnappy.a
-rwxr-xr-x. 1 hadoop hadoop 960 6月 28 12:11 libsnappy.la
lrwxrwxrwx. 1 hadoop hadoop 23 6月 28 11:14 libsnappy.so -> /usr/lib64/libsnappy.so
-rwxr-xr-x. 1 hadoop hadoop 147670 6月 28 12:11 libsnappy.so.1
-rwxr-xr-x. 1 hadoop hadoop 147670 6月 28 12:11 libsnappy.so.1.2.0
[root@dev03 ~]#

同步全部节点 重启服务