大数据工具Hadoop快速入门3安装

前提

ubuntu、openssh-server和java安装ok

账号

sudo addgroup hadoop_
sudo adduser --ingroup hadoop_ hduser_
su - hduser_
ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh localhost

图片.png

图片.png

下载安装

http://hadoop.apache.org/releases.html

图片.png

选择最新二进制版本。会打开类似https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz这样的页面，点击上方的镜像，比如http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz
下载。

mv hadoop-2.9.2.tar.gz /opt
cd /opt
sudo tar xzf hadoop-2.9.2.tar.gz
mv hadoop-2.9.2 hadoop

参考资料

• python测试开发项目实战-目录
• python工具书籍下载-持续更新
• python 3.7极速入门教程 - 目录
• 讨论qq群630011153 144081101
• 原文地址
• 本文涉及的python测试开发库 谢谢点赞！
• [本文相关海量书籍下载](https://github.com/china-testing/python-api-tesing/blob/master/books.md
• https://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_16_04_single_node_cluster.php
• https://medium.com/@Alibaba_Cloud/how-to-setup-hadoop-cluster-ubuntu-16-04-8c540ab30fbf

配置

注意hadoop不会从系统中读取变量，一定要在这些文件中配置：

在~/.bashrc添加：

export HADOOP_HOME=/opt/hadoop
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$HADOOP_HOME/bin

HDFS配置

$HADOOP_HOME/etc/hadoop/hadoop-env.sh，修改下面部分。

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

$HADOOP_HOME/etc/hadoop/core-site.xml

在里面添加：

hadoop.tmp.dir
/app/hadoop/tmp
Parent directory for other temporary directories.


fs.defaultFS 
hdfs://localhost:54310
The name of the default file system. 

创建目录

sudo mkdir -p /app/hadoop/tmp
sudo chown -R hduser_:hadoop_  /app/hadoop/tmp
sudo chmod 750  /app/hadoop/tmp

Map Reduce配置

$ vi /etc/profile.d/hadoop.sh
export HADOOP_HOME=/opt/hadoop
$ sudo chmod +x /etc/profile.d/hadoop.sh
# sudo cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml
$ vi $HADOOP_HOME/etc/hadoop/mapred-site.xml
# 在部分添加

mapreduce.jobtracker.address
localhost:54311
MapReduce job tracker runs at this host and port.


$ sudo mkdir -p /usr/local/hadoop_store/hdfs/namenode
$ sudo mkdir -p /usr/local/hadoop_store/hdfs/datanode
$ sudo chown -R hduser_:hadoop_ /usr/local/hadoop_store
$ vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml

 
  dfs.replication
  1
  Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  
 
 
   dfs.namenode.name.dir
   file:/usr/local/hadoop_store/hdfs/namenode
 
 
   dfs.datanode.data.dir
   file:/usr/local/hadoop_store/hdfs/datanode
 


$ $HADOOP_HOME/bin/hdfs namenode -format # 格式化
$ $HADOOP_HOME/sbin/start-dfs.sh　# 启动
$ $HADOOP_HOME/sbin/start-yarn.sh
$ jps # 查看进程
$ $HADOOP_HOME/sbin/stop-dfs.sh　# 停止
$ $HADOOP_HOME/sbin/stop-yarn.sh

图片.png