Debian下hadoop安装

Debian下安装hadoop

目标:
配置安装openjdk环境，下载安装hadoop

安装OpenJDK

#先切换到root用户，更新debian的包索引，debian9在系统安装时已选择163的源
su
apt-get update
#然后安装包管理工具aptitude,通过aptitude搜索安装openjdk
apt-get install aptitude

然后尝试搜索openjdk获得安装包名

#aptitude search openjdk
MSDB:/home/ms# aptitude search openjdk
p   openjdk-8-dbg                   - Java runtime based on OpenJDK (debugging s
i   openjdk-8-dbg:i386              - Java runtime based on OpenJDK (debugging s
p   openjdk-8-demo                  - Java runtime based on OpenJDK (demos and e
p   openjdk-8-demo:i386             - Java runtime based on OpenJDK (demos and e
p   openjdk-8-doc                   - OpenJDK Development Kit (JDK) documentatio
i   openjdk-8-jdk                   - OpenJDK Development Kit (JDK)             
p   openjdk-8-jdk:i386              - OpenJDK Development Kit (JDK)             
i A openjdk-8-jdk-headless          - OpenJDK Development Kit (JDK) (headless)  
p   openjdk-8-jdk-headless:i386     - OpenJDK Development Kit (JDK) (headless)  
i A openjdk-8-jre                   - OpenJDK Java runtime, using Hotspot JIT   
i A openjdk-8-jre:i386              - OpenJDK Java runtime, using Hotspot JIT   
p   openjdk-8-jre-dcevm             - Alternative VM for OpenJDK 8 with enhanced
p   openjdk-8-jre-dcevm:i386        - Alternative VM for OpenJDK 8 with enhanced
i A openjdk-8-jre-headless          - OpenJDK Java runtime, using Hotspot JIT (h
i A openjdk-8-jre-headless:i386     - OpenJDK Java runtime, using Hotspot JIT (h
p   openjdk-8-jre-zero              - Alternative JVM for OpenJDK, using Zero/Sh
p   openjdk-8-jre-zero:i386         - Alternative JVM for OpenJDK, using Zero/Sh
p   openjdk-8-source                - OpenJDK Development Kit (JDK) source files

因为已经安装了，前面的 i A代表软件包的标记状态，如果没有安装，和上述的有差异，别在意，现在需要安装openjdk-8的jdk和jre(安装软件需要root权限，下面默认登录的用户是root),i386是32位的安装包，目前系统是amd64

aptitude install openjdk-8-jdk
aptitude install openjdk-8-jre

配置运行环境

1.查看openjdk安装目录
openjdk安装目录为/usr/lib/jvm/xxxx（版本共存）

#cd /usr/lib/jvm
#ls -l
总用量 12
lrwxrwxrwx 1 root root   24 1月   6  2017 default-java -> java-1.8.0-openjdk-amd64
drwxr-xr-x 4 root root 4096 10月 19 23:48 java-1.5.0-gcj-6-amd64
lrwxrwxrwx 1 root root   20 7月  23 20:12 java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64
lrwxrwxrwx 1 root root   19 11月  1 22:17 java-1.8.0-openjdk-i386 -> java-8-openjdk-i386
drwxr-xr-x 7 root root 4096 11月  9 21:16 java-8-openjdk-amd64
drwxr-xr-x 5 root root 4096 11月  9 02:07 java-8-openjdk-i386

default-java是当前系统默认的java，如果没有这个软连接，就手动创建软连接

#ln -s java-1.8.0-openjdk-amd64 default-java

2.配置default-java到环境变量，作为JAVA_HOME,debian9默认shell为bash，编辑~/.bashrc
添加以下内容

export JAVA_HOME=/usr/lib/jvm/default-java
export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib/rt.jar
export PATH=$JAVA_HOME/bin:$PATH

JAVA_HOME用default-java软连接代替实际目录，方便后续java环境升级更改
PATH中设置JAVA_HOME/bin在前保证java命令搜索路径优先

设置完毕后，输入bash回车使得环境变量生效（新shell载入bashrc,后续的bash都生效，已打开的终端）

3.验证环境变量设置

##查看java路径
#which java
/usr/lib/jvm/default-java/bin/java
#which javac
/usr/lib/jvm/default-java/bin/javac

java环境设置完毕

---

安装Hadoop框架

Hadoop官网:http://hadoop.apache.org
Hadoop官网参考中文文档:http://hadoop.apache.org/docs/r1.0.4/cn/

1.获得Hadoop

通过官网找到稳定发行版,打开页面，找到Download,下载Hadoop2.8.2的binary文件，然后通过浏览器下载，这里使用wget下载

#cd ~
#wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.2/hadoop-2.8.2.tar.gz
正在解析主机 mirror.bit.edu.cn (mirror.bit.edu.cn)... 114.247.56.117
正在连接 mirror.bit.edu.cn (mirror.bit.edu.cn)|114.247.56.117|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：243900138 (233M) [application/octet-stream]
正在保存至: “hadoop-2.8.2.tar.gz”
....

下载文件位于/root/hadoop-2.8.2.tar.gz(本次实验为了方便，直接使用root目录)
解压文件到当前目录

tar -xzvf hadoop-2.8.2.tar.gz
....

解压后目录位于/root/hadoop-2.8.2(以下以$HADOOP_HOME描述此目录)

#cd hadoop-2.8.2
#ls
bin  etc  include lib  libexec  LICENSE.txt  logs  NOTICE.txt  README.txt  sbin  share

其中bin下存着hadoop文件
sbin则是hadoop的启动脚本

2.配置Hadoop
由于实验过程中发现了，JAVA_HOME无法被识别，直接配置hadoop-env的JAVA_HOME为实际路径default-java,配置文件位于$HADOOP_HOME/etc/hadoop/hadoop-env.sh(Linux下)

#vim etc/hadoop/hadoop-env.sh

找到文件中JAVA_HOME，修改如下

# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/default-java

修改完毕后:wq保存

3.JAVA_HOME配置绝对路径后,hadoop不再提示找不到java_home

# bin/hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar             run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp   copy file or directories recursively
  archive -archiveName NAME -p  *  create a hadoop archive
  classpath            prints the class path needed to get the
                       Hadoop jar and the required libraries
  credential           interact with credential providers
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.

4.Hadoop的java环境变量以及基本安装完毕