Mac环境Hadoop-2.7.0源码编译

切记:使用root用户编译

1. 源码下载及解压

    Hadoop官网:https://hadoop.apache.org/release/2.7.0.html,Download src        

Mac环境Hadoop-2.7.0源码编译_第1张图片

下载完成后将hadoop-2.7.0-src.tar.gz 解压到/opt/目录,

进入解压出来的目录

Mac环境Hadoop-2.7.0源码编译_第2张图片

BUILDING.txt文件是编译hadoop源码的说明文件;打开文件,列出了编译hadoop源码时,要求软件及对应版本要求。

本机环境:jdk 和maven已经安装好,跳过

Mac环境Hadoop-2.7.0源码编译_第3张图片

2. 环境准备

   安装findbugs:brew install findbugs,安装完成查看版本:spotbugs -version

    安装autoconf:brew install autoconf

    安装automake:brew install automake

    安装libtool: brew install libtool , brew link libtool

    安装Protocol buffers (Google's data interchange format),必须是2.5.0:brew search protobuf,发现没有这个版本,那么我们编译安装,安装完成查看版本:protoc --version

    下载安装包:https://github.com/protocolbuffers/protobuf/releases?after=v3.0.0-alpha-1,找到对应的版本    

Mac环境Hadoop-2.7.0源码编译_第4张图片

    下载完成后解压:sudo tar -xzvf protobuf-2.5.0.tar.gz -C /opt/

    进入目录:cd protobuf-2.5.0

    执行安装配置命令:    

        ./autogen.sh & ./configure \ 

        make \

        make install

     最后验证是否安装成功:protoc --version

安装cmake: brew install cmake

安装zlib: brew install zlib

安装openssl:brew install openssl

安装snappy(若想让hadoop支持snappy压缩,得先安装snappy):

注意安装snappy之前,需要先安装cmake,pkg-config,因为snappy是依赖cmake和pkg-config的,如果已经安装了snappy,可以重新install,brew reinstall snappy

    brew install cmake

    brew install pkg-config

安装snappy成功后,查看:brew info snappy

Mac环境Hadoop-2.7.0源码编译_第5张图片

安装jansson:brew install jansson

3. 编译hadoop

    进入源码目录:cd /opt/hadoop-2.7.0-src/

    执行编译命令:mvn package -Pdist,native -DskipTests -Dtar

    如果中途编译失败,并且不要文档、hadoop支持snappy、支持openssl的话,请使用这个命令:

    mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Dsnappy.lib=/usr/local/Cellar/snappy/1.1.7_1/lib  -Dbundle.snappy -Drequire.openssl

说明:支持snappy(若不需要支持snappy,把这些选项去除即可):Drequire.snappy -Dsnappy.lib=/usr/local/Cellar/snappy/1.1.7_1/lib -Dbundle.snappy

    编译过程中遇到的问题:

        1)No such file or directory:一般就是权限的问题,切换到root再执行看看。或者你可以将更改目录权限sudo chown -R chriszhao:staff /opt/hadoop-2.7.0-src/* (chriszhao:改成你自己的用户名)

        2)hadoop-pipes: An Ant BuildException has occured: exec returned: 1:   

            查看build-main.xml

Mac环境Hadoop-2.7.0源码编译_第6张图片

        执行: cmake /opt/hadoop-2.7.0-src/hadoop-tools/hadoop-pipes/src/ -DJVM_ARCH_DATA_MODEL=64,发现如下错误:

        Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the System variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR), suggesting that not found openssl suggested that the need to add environment variables

        然后修改~/.zshrc, 之后再次执行以上cmake 命令,如果没错就再尝试编译

        export LDFLAGS="-L/usr/local/opt/[email protected]/lib"

        export CPPFLAGS="-I/usr/local/opt/[email protected]/include"

        export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

        export OPENSSL_INCLUDE_DIR="/usr/local/opt/[email protected]/include"

        export PATH="/usr/local/opt/[email protected]/bin:$PATH"

        3)因为网络问题导致tomcat包下不来,多尝试几次编译

4. 编译完成后

    想要的结果包在hadoop-dist/target/下

你可能感兴趣的:(Mac环境Hadoop-2.7.0源码编译)