Hadoop2.0安装之非HA版

主要步骤跟Hadoop1.0(1.0安装地址)一致，主要在配置这块有更改

安装

• 下载地址：http://archive.apache.org/dist/hadoop/core/hadoop-2.6.5/

• hadoop安装目录创建./tmp目录，./dfs/name，./dfs/data

• 修改./etc/hadoop/hadoop-env.sh

# 设置JAVA_HOME
export JAVA_HOME="/usr/local/src/jdk1.8.0_181/"

• 修改./etc/hadoop/yarn-env.sh

# 设置JAVA_HOME
JAVA_HOME="/usr/local/src/jdk1.8.0_181/"

• 修改./etc/hadoop/slaves

slave1
slave2

• 修改./etc/hadoop/core-site.xml

    
        fs.defaultFS
        hdfs://master:9000
    
    
        hadoop.tmp.dir
        file:/usr/local/src/hadoop-2.6.5/tmp
    

• 修改./etc/hadoop/hdfs-site.xml

    
        dfs.namenode.secondary.http-address
        master:9001
    
    
        dfs.namenode.name.dir
        file:/usr/local/src/hadoop-2.6.5/dfs/name
    
    
        dfs.datanode.data.dir
        file:/usr/local/src/hadoop-2.6.5/dfs/data
    
    
        dfs.replication
        2
    

• 修改./etc/hadoop/mapred-site.xml

    
        mapreduce.framework.name
        yarn
    
    
        mapreduce.jobhistory.address
        slave1:10020
    
    
        mapreduce.jobhistory.webapp.address
        slave1:19888
    

• 修改./etc/hadoop/yarn-site.xml

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
        yarn.nodemanager.aux-services.mapreduce.shuffle.class
        org.apache.hadoop.mapred.ShuffleHandler
    
    
        yarn.resourcemanager.address
        master:8032
    
    
        yarn.resourcemanager.scheduler.address
        master:8030
    
    
        yarn.resourcemanager.resource-tracker.address
        master:8035
    
    
        yarn.resourcemanager.admin.address
        master:8033
    
    
        yarn.resourcemanager.webapp.address
        master:8088
    
    
        yarn.log-aggregation-enable
        true
    
    
        yarn.log-aggregation.retain-seconds
        259200
    
    
        yarn.log.server.url
        http://slave1:19888/jobhistory/logs
    
    
        yarn.nodemanager.vmem-pmem-ratio
        4.0
    

• 和Hadoop1.0一样，第一次启动前，需要格式化hdfs：./bin/hadoop namenode -format

• 启动：./sbin/start-all.sh

• 使用：跟Hadoop1.0一样，使用./bin/hadoop命令

• 关闭：./sbin/stop-all.sh

提交MapReduce任务

基本上没什么变化，除了Hadoop streaming地址变了

[wadeyu@master mr_count]$ cat run.sh
HADOOP_CMD=/usr/local/src/hadoop-2.6.5/bin/hadoop
HADOOP_STREAMING_JAR=/usr/local/src/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar

INPUT_FILE=/data/The_Man_of_Property.txt
OUTPUT_DIR=/output/wc

$HADOOP_CMD fs -rmr -skipTrash $OUTPUT_DIR

$HADOOP_CMD jar $HADOOP_STREAMING_JAR \
    -input $INPUT_FILE \
    -output $OUTPUT_DIR \
    -mapper "python map.py" \
    -reducer "python red.py" \
    -file ./map.py \
    -file ./red.py

参考资料

【0】八斗学院内部培训资料

转载于:https://www.cnblogs.com/wadeyu/p/9696044.html