HDFS单机版安装

一、 准备机器

机器编号	地址	端口
1	10.211.55.8	9000、50070、8088

二、 安装

学习地址http://www.roncoo.com/course/view/5a057438cc2a4231a8c245695faea238

1、 安装java环境

        export JAVA_HOME=/data/program/software/java8

        export JRE_HOME=/data/program/software/java8/jre

        export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib

 export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

 source /etc/profile使其生效

2、 修改hostname

vi /etc/profile

添加10.211.55.8 bigdata2

3、 关闭防火墙

service iptables stop

用久关闭防火墙：chkconfig iptables off

查看防火墙状态：service iptables status

4、 添加hadoop用户和用户组

创建用户组：groupadd hadoop   

新建hadoop用户并且增加到hadoop用户下：useradd –g hadoop hadoop

设置密码：passwd hadoop

5、 下载安装hadoop

    cd /data/program/software

    wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.1/hadoop-2.8.1.tar.gz 

    解压：tar -zxf hadoop-2.8.1.tar.gz

    将hadoop-2.8.1操作权限赋给hadoop用户：chown –R hadoop:hadoop hadoop-2.8.1

6、 创建数据目录

mkdir –p /data/dfs/name

mkdir –p /data/dfs/data

mkdir –p /data/tmp

将/data文件权限赋给hadoop：chown –R hadoop:hadoop /data

7、 配置etc/hadoop/core-site.xml

cd /data/program/software/hadoop-2.8.1

        

fs.defaultFS

hdfs://bigdata2:9000

io.file.buffer.size

131072

hadoop.tmp.dir

file:/data/tmp

Abase for other temporary directories.

hadoop.proxyuser.hadoop.hosts

*

hadoop.proxyuser.hadoop.groups

*

8、 配置etc/hadoop/hdfs-site.xml

dfs.namenode.name.dir

file:/data/dfs/name

Determineswhere on the local filesystem the DFS name node should store the name table. Ifthis is a comma-delimited list of directories then the name table is replicatedin all of the directories, for redundancy.

true

dfs.datanode.data.dir

file:/data/dfs/data

Determineswhere on the local filesystem an DFS data node should store its blocks. If thisis a comma-delimited list of directories, then data will be stored in all nameddirectories, typically on different devices.Directories that do not exist areignored.

true

dfs.replication

1

dfs.permissions

false

9、 配置etc/hadoop/mapred-site.ml

 

mapreduce.framework.name

yarn

10、 配置yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

 

11、 配置slaves

bigdata2

12、 设置hadoop环境变量

vi /etc/profile

HADOOP_HOME=/data/program/software/hadoop-2.8.1

PATH=$HADOOP_HOME/bin:$PATH

export HADOOP_HOME PATH

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

13、 ssh无密码验证配置

切换到hadoop用户下：su hadoop

直接输入cd会切换到/home/hadoop根目录下：cd

创建.ssh目录：mkdir .ssh

生成秘钥(一直回车)：ssh-keygen –t rsa

进入.ssh目录：cd .ssh

复制一份秘钥：cp id_rsa.pub authorized_keys

后退到根目录：cd ..

给.ssh700权限：chmod 700 .ssh

给.ssh里面的文件600权限：chmod 600 .ssh/*

ssh bigdata2

14、 运行hadoop

先格式化一下namenode：bin/hadoop namenode –format

为了让大家看一下hadoop我们将所有的服务全部启动：sbin/start-all.sh

看一下启动的服务：jps

看一下hdfs的管理界面：http://10.211.55.8:50070

看hadoop运行任务：http://10.211.55.8:8088/cluster/nodes

15、 测试

创建一个目录：bin/hadoop fs –mkdir /test

创建一个txt然后放到/test下：bin/hadoop fs –put /home/hadoop/first.txt /text

查看目录下文件：bin/hadoop fs –ls /test

 

启动过程中如果出现如下错误：则需要更改/data/program/software/hadoop-2.8.1/etc/hadoop/ hadoop-env.sh中的 JAVA_HOME为绝对地址。

[hadoop@bigdata2 hadoop-2.8.1]$ sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

17/07/25 13:52:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/07/25 13:52:49 WARN conf.Configuration: bad conf file: element not

17/07/25 13:52:49 WARN conf.Configuration: bad conf file: element not

17/07/25 13:52:49 WARN conf.Configuration: bad conf file: element not

17/07/25 13:52:49 WARN conf.Configuration: bad conf file: element not

Starting namenodes on [bigdata2]

bigdata2: Error: JAVA_HOME is not set and could not be found.

The authenticity of host 'localhost (::1)' can't be established.

RSA key fingerprint is 24:e2:40:a1:fd:ac:68:46:fb:6b:6b:ac:94:ac:05:e3.

Are you sure you want to continue connecting (yes/no)? bigdata2: Error: JAVA_HOME is not set and could not be found.

^Clocalhost: Host key verification failed.

Starting secondary namenodes [0.0.0.0]

0.0.0.0: Error: JAVA_HOME is not set and could not be found.