Hadoop笔记:YARN环境搭建

Hadoop笔记:YARN环境搭建

标签: 大数据

  • Hadoop笔记:YARN环境搭建
    • 伪分布式YARN环境搭建
      • 配置参数
      • 启动进程
      • 验证
      • 定制YARN相关的进程


使用版本:hadoop-2.6.0-cdh5.7.0

配置文件:
yarn-site.xml
mapred-site.xml

伪分布式YARN环境搭建

You can run a MapReduce job on YARN in a pseudo-distributed mode by setting a few parameters and running ResourceManager daemon and NodeManager daemon in addition.

配置参数

Configure parameters as follows:etc/hadoop/mapred-site.xml:

配置mapred-site.xml文件,说明MapReduce是执行在YARN框架上的

<configuration>
    <property>
        <name>mapreduce.framework.namename>
        <value>yarnvalue>
    property>
configuration>

我们可以发现,它只有一个模板:

[japson@localhost hadoop]$ vim mapred-site.xml.template 

我们可以根据这个模板先拷贝一下:

[japson@localhost hadoop]$ cp mapred-site.xml.template mapred-site.xml
[japson@localhost hadoop]$ vim mapred-site.xml

再配置etc/hadoop/yarn-site.xml,告诉框架,yarn的nodemanager用的是哪一个service,用的是mapreduce_shuffle。如果到spark中,就是spark_shuffle:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-servicesname>
        <value>mapreduce_shufflevalue>
    property>
configuration>

启动进程

启动ResourceManager进程和NodeManager进程

$ sbin/start-yarn.sh
[japson@localhost sbin]$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/japson/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-japson-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /home/japson/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-japson-nodemanager-localhost.localdomain.out

验证

jps:

[japson@localhost sbin]$ jps
5161 SecondaryNameNode
4998 DataNode
5399 NodeManager
5310 ResourceManager
4874 NameNode
5626 Jps

http://localhost:8088/

定制YARN相关的进程

$ sbin/stop-yarn.sh

你可能感兴趣的:(Hadoop学习以及踩坑记录)