一、准备Linux环境
使用Vmvare虚拟机安装Linux环境Ubuntu 16.04 LTS
二、JDK安装和配置
2.1 安装JDK
# apt-get install + 选择的JDK版本
2.2 环境变量配置
配置/etc/profile
# vim /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
使配置生效
# source /etc/profile
三、Hadoop安装和配置
3.1 下载Hadoop
下载Hadoop，并解压到/opt/
# wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
# mv hadoop-1.2.1.tar.gz /opt/
# cd /opt/
# tar -zxvf hadoop-1.2.1.tar.gz
3.2 配置Hadoop
需要配置/conf 目录下的四个文件：hadoop-env.sh，core-site.xml，hdfs-site.xml，mapred-site.xml
# cd /opt/hadoop-1.2.1/conf
3.2.1 配置hadoop-env.sh
查看$JAVA_HOME路径
# echo $JAVA_HOME
/usr/lib/jvm/java-9-openjdk-amd64
配置hadoop-env.sh文件中的$JAVA_HOME
# vim hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
3.2.2 配置core-site.xml
# vim core-site.xml
hadoop.tmp.dir
/hadoop
dfs.name.dir
/hadoop/name
fs.default.name
hdfs://localhost:9000
3.2.3 配置hdfs-site.xml
# vim hdfs-site.xml
dfs.data.dir
/hadoop/data
3.2.4 配置mapred-site.xml
# vim mapred-site.xml
mapred.job.tracker
localhost:9001
3.2.5 配置/etc/profile
# vim /etc/profile
export HADOOP_HOME=/opt/hadoop-1.2.1
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
# source /etc/profile
3.3 初始化NameNode
配置好上述文件，运行hadoop命令，结果如下所示
# hadoop
Warning: $HADOOP_HOME is deprecated.
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
namenode -format format the DFS filesystem
secondarynamenode run the DFS secondary namenode
namenode run the DFS namenode
......
Most commands print help when invoked w/o parameters.
问题描述
1、/etc/profile配置文件在以普通用户登陆时修改后，改为以root用户登录，/etc/profile就失效了。
2、运行hadoop命令，出现警告Warning: $HADOOP_HOME is deprecated.
解决方法
# cd ~
# vim .bashrc
export HADOOP_HOME_WARN_SUPPRESS=1
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
export JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/opt/hadoop-1.2.1
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
# source .bashrc
上述问题解决后，对namenode进行格式化操作
# hadoop namenode -format
报错信息
Error: Config file not found: /usr/lib/jvm/java-9-openjdk-amd64/conf/management/ management.properties
解决方法
将java-9-openjdk-amd64目录下的conf文件删除，并新建一个conf文件夹，在新建的conf文件夹下，新建文件夹management，management下新建空的文件management.properties
# cd /usr/lib/jvm/java-9-openjdk-amd64
# rm conf
# mkdir conf
# cd conf
# mkdir -p management
# touch /usr/lib/jvm/java-9-openjdk-amd64/conf/management/management.properties
报错解决后，格式化namenode，运行结果如下
18/02/12 04:32:19 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ubuntu/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 9-internal
************************************************************/
Re-format filesystem in /hadoop/dfs/name ? (Y or N) y
Format aborted in /hadoop/dfs/name
18/02/12 04:32:34 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
************************************************************/
3.4 启动Hadoop
Hadoop安装和配置完成后，在/opt/hadoop-1.2.1/bin目录下可以看到start-all.sh启动命令。运行启动命令
# start-all.sh
报错信息
localhost: ssh: connect to host localhost port 22: Connection refused
解决方法
下载openssh-server
# apt-get install openssh-server
重新运行start-all.sh启动命令，输入密码时，有如下报错信息
root@localhost's password: localhost: Permission denied, please try again.
解决方法：
修改root密码
# sudo passwd root
编辑配置文件，允许 root 用户通过 ssh 登录
# vim /etc/ssh/sshd_config
将 PermitRootLogin prohibit-password注释掉
添加PermitRootLogin yes
# service ssh restart
然后ssh localhost ，输入密码后，结果如下所示，则问题解决
# ssh localhost
root@localhost's password:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.13.0-32-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
157 packages can be updated.
0 updates are security updates.
重新启动hadoop，运行结果如下所示
# start-all.sh
starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-ubuntu.out
root@localhost's password:
localhost: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-ubuntu.out
root@localhost's password:
localhost: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-ubuntu.out
starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-ubuntu.out
root@localhost's password:
localhost: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-ubuntu.out