Hadoop伪分布式安装配置

A、添加hadoop用户

1、添加用户组

[root@hadoop00 ~]# groupadd hadoop

2、添加用户并分配用户组

[root@hadoop00 ~]# useradd -g hadoop hadoop

3、修改hadoop用户密码

[root@hadoop00 ~]# passwd hadoop

B、配置本地YUM源

1、上传系统安装镜像到虚拟机服务器的/root目录

2、新建挂载点目录

[root@hadoop00 ~]# mkdir /mnt/centos

3、挂载镜像

       [root@hadoop00 ~]# mount -o loop /root/CentOS-6.8-x86_64-bin-DVD1.iso /mnt/centos

4、切换目录

       [root@hadoop00 ~]# cd /etc/yum.repos.d

5、新建备份目录

       [root@hadoop00 ~]# mkdir bakup

6、备份文件,把所有以Cent开头的文件全部备份移动到bakup目录中

       [root@hadoop00 ~]# mv Cent* bakup

7、新建YUM源文件,添加如下内容

       [root@hadoop00 ~]# vi local.repo

              [local]

              name=local

              baseurl=file:///mnt/centos

              enabled=1

              gpgcheck=0

8、验证YUM源,出现如下可用包,表示本地YUM源配置成功。

       [root@hadoop00 ~]# yum list

Hadoop伪分布式安装配置_第1张图片

C、SSH无密码配置

1、查看软件openssh与rsync安装状态

[root@hadoop00 ~]# rpm -qa|grep openssh

[root@hadoop00 ~]# rpm -qa|grep rsync

2、安装openssh与rsync

[root@hadoop00 ~]# yum -y install openssh

[root@hadoop00 ~]# yum install openssh-clients.x86_64 -y

[root@hadoop00 ~]# yum -y install rsync

3、切换到hadoop用户

[root@hadoop00 ~]# su - hadoop

4、生成SSH密码对

[root@hadoop00 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

5、将id_dsa.pub追加到授权的key文件中

[root@hadoop00 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

6、设置授权key文件权限

[root@hadoop00 ~]# chmod 600 ~/.ssh/authorized_keys

7、测试ssh连接

[root@hadoop00 ~]# ssh hadoop00

Hadoop伪分布式安装配置_第2张图片

无需输入密码,即可完成登录,表示SSH配置成功。

D、Hadoop安装与配置

1、将Hadoop安装压缩包hadoop-2.7.3.tar.gz,上传至/root目录

2、将压缩包解压至/usr目录

[root@hadoop00 ~]# tar zxvf /root/hadoop-2.7.3.tar.gz -C /usr/

3、修改文件夹名称

[root@hadoop00 ~]# mv /usr/hadoop-2.7.3/ /usr/hadoop

4、将hadoop文件夹授权给hadoop用户

[root@hadoop00 ~]# chown -R hadoop:hadoop /usr/hadoop/

5、设置环境变量

[root@hadoop00 ~]# vi /etc/profile

(文档末尾追加如下内容)
export HADOOP_HOME=/usr/hadoop

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

6、使环境变量生效

[root@hadoop00 ~]# source /etc/profile

7、测试环境变量设置

[root@hadoop00 ~]# hadoop version

Hadoop伪分布式安装配置_第3张图片

E、配置HDFS

1、切换至Hadoop用户

[root@hadoop00 ~]# su - hadoop

2、修改hadoop-env.sh

[hadoop@hadoop00 ~]$ cd /usr/hadoop/etc/hadoop/

[hadoop@hadoop00 ~]$ vi hadoop-env.sh

(文档末尾追加如下内容)

export JAVA_HOME=/usr/java/jdk1.8.0_162

3、修改core-site.xml

[hadoop@hadoop00 ~]$ vi core-site.xml

(添加如下内容,注意标签对的匹配及唯一性)

   

        fs.defaultFS

        hdfs://hadoop00:9000

   

   

        hadoop.tmp.dir

        /usr/hadoop/tmp/

        A base for other temporary directories.

   

4、修改hdfs-site.xml

[hadoop@hadoop00 ~]$ vi hdfs-site.xml

(添加如下内容)

   

        dfs.replication

        1

   

5、格式化hdfs

[hadoop@hadoop00 ~]$ hdfs namenode -format

注:出现Exiting with status 0即为成功

6、启动hdfs

[hadoop@hadoop00 ~]$ start-dfs.sh

(停止命令)# stop-dfs.sh
注:输出如下内容,则启动成功
15/09/21 18:09:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [Hadoop.Master]

Hadoop.Master: starting namenode, logging to /usr/hadoop/logs/hadoop-hadoop-namenode-Hadoop.Master.out

Hadoop.Master: starting datanode, logging to /usr/hadoop/logs/hadoop-hadoop-datanode-Hadoop.Master.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is b5:96:b2:68:e6:63:1a:3c:7d:08:67:4b:ae:80:e2:e3.

Are you sure you want to continue connecting (yes/no)? yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-hadoop-secondarynamenode-Hadoop.Master.out

15/09/21 18:09:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab

7、查看进程

[hadoop@hadoop00 ~]$ jps

注:输出类似如下内容
1763 NameNode

1881 DataNode

2146 Jps

2040 SecondaryNameNode

8、使用web浏览器查看Hadoop运行状态
http://你的服务器ip地址:50070/

Hadoop伪分布式安装配置_第4张图片

9、在HDFS上运行WordCount:

1)创建HDFS用户目录

[hadoop@hadoop00 ~]$ hdfs dfs -mkdir /user

[hadoop@hadoop00 ~]$ hdfs dfs -mkdir /user/hadoop #根据自己的情况调整

2)复制输入文件(要处理的文件)到HDFS上

[hadoop@hadoop00 ~]$ hdfs dfs -put /usr/hadoop/etc/hadoop/ /input

3)查看我们复制到HDFS上的文件

[hadoop@hadoop00 ~]$ hdfs dfs -ls /input

4)运行单词检索(grep)程序

[hadoop@hadoop00 ~]$ hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep /input output 'dfs[a-z.]+'

(WordCount

[hadoop@hadoop00 ~]$ hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input output

#说明:output文件夹如已经存在则需要删除或指定其他文件夹。)

5)查看运行结果

[hadoop@hadoop00 ~]$ hdfs dfs -cat output/*

10、配置YARN

1)修改mapred-site.xml

[hadoop@hadoop00 ~]$ cd /usr/hadoop/etc/hadoop/

[root@hadoop00 ~]# cp mapred-site.xml.template mapred-site.xml

[hadoop@hadoop00 ~]$ vi mapred-site.xml

(添加如下内容)

   

        mapreduce.framework.name

        yarn

   

2)修改yarn-site.xml

[hadoop@hadoop00 ~]$ vi yarn-site.xml

(添加如下内容)

   

        yarn.nodemanager.aux-services

        mapreduce_shuffle

   

3)启动YARN

[hadoop@hadoop00 ~]$ start-yarn.sh

(停止yarn stop-yarn.sh)

4)查看当前java进程

[hadoop@hadoop00 ~]$ jps

5)(输出如下)

4918 ResourceManager

1663 NameNode

1950 SecondaryNameNode

5010 NodeManager

5218 Jps

1759 DataNode

6)运行你的mapReduce程序

配置好如上配置再运行mapReduce程序时即是yarn中运行

7)使用web查看Yarn运行状态
http://你的服务器ip地址:8088/

F、HDFS常用命令参考

1、创建HDFS文件夹

1)在根目录创建input文件夹
[hadoop@hadoop00 ~]$ hdfs dfs -mkdir -p /input
2)在用户目录创建input文件夹
说明:如果不指定“/目录”,则默认在用户目录创建文件夹
[hadoop@hadoop00 ~]$ hdfs dfs -mkdir -p input

(等同于 hdfs dfs -mkdir -p /user/hadoop/input)

2、查看HDFS文件夹

1)查看HDFS根文件夹
[hadoop@hadoop00 ~]$ hdfs  dfs  -ls /
2)查看HDFS用户目录文件夹

[hadoop@hadoop00 ~]$ hdfs  dfs  -ls

3)查看HDFS用户目录文件夹下input文件夹
[hadoop@hadoop00 ~]$ hdfs  dfs  -ls input

(等同与 hdfs  dfs  -ls /user/hadoop/input)

3、复制文件到HDFS
[hadoop@hadoop00 ~]$ hdfs dfs -put /usr/hadoop/etc/hadoop input

4、删除文件夹

[hadoop@hadoop00 ~]$ hdfs  dfs  -rm -r input

你可能感兴趣的:(Hadoop,分布式,hadoop,大数据,apache,centos,服务器)