Hadoop-3.3.4集群部分lib缺失问题

1.问题描述

(base) [hadoop@hadoop1 native]$ hadoop checknative
2023-12-25 14:20:21,615 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2023-12-25 14:20:21,618 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2023-12-25 14:20:21,623 WARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
2023-12-25 14:20:21,623 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2023-12-25 14:20:21,658 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /data/module/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
PMDK:    false The native code was built without PMDK support.

2.原因分析

• Failed to load libisal.so.2
  缺少libisal.so.2，需要安装
• Cannot load libcrypto.so
  必要软件包

3.问题解决

3.1 解决Failed to load libisal.so.2

Intel ISA-L代表英特尔智能存储加速库。ISA-L是针对存储应用程序而优化的低级功能的开源集合。它包括针对Intel AVX和AVX2指令集优化的快速块Reed-Solomon类型纠删码。HDFS纠删码可以利用ISA-L加速编码和解码计算。ISA-L支持大多数主要操作系统，包括Linux和Windows。默认情况下不启用ISA-L。

1. 安装yasm和nasm

sudo yum install -y yasm
sudo yum install -y nasm

注意：isa-l-2.30.0对nasm和yasm有版本要求，低版本在安装时会报错。

2. 编译安装isa-l-2.30.0

tar -zxvf isa-l-2.30.0.tar.gz
cd isa-l-2.30.0
sudo ./autogen.sh
sudo  ./configure --prefix=/usr --libdir=/usr/lib64
sudo  make
sudo  make install
 
sudo make -f Makefile.unx

各部操作无报错表示编译安装成功。

3. 检查libisal.so*是否成功

ll /lib64/libisal.so*
 
############如果有，则跳过##############

############如果没有有，则复制##############
cp bin/libisal.so bin/libisal.so.2 /lib64

4. 复制编译好的libisal.so.2/libisal.so至其他机器的/lib64目录

sudo xsync libisal.so.2 libisal.so

5. 在Hadoop上运行检查命令

hadoop checknative

3.2 解决Cannot load libcrypto.so

参考hadoop官网说明https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

 sudo  yum -y install  lzo-devel  zlib-devel  gcc gcc-c++ autoconf automake libtool openssl-devel fuse-devel cmake

4.测试是否完成

[hadoop@hadoop2 bin]$ hadoop checknative
2023-12-25 16:30:01,759 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2023-12-25 16:30:01,762 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2023-12-25 16:30:01,793 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /data/module/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2
PMDK:    false The native code was built without PMDK support.