搭建Hadoop伪分布式环境-作业001

搭建Hadoop伪分布式环境-作业001

搭建Linux环境

1 配置VM###

2 配置虚拟机磁盘

3 配置CenOS

4 配置磁盘，选择YES

5 配置主机名

6 配置网络

这里也可以添加DNS####

7 配置普通用户

useradd beifeng
passwd beifeng

2016-07-11_104420.jpg

sudo权限设置###

visudo

2016-07-11_141357.jpg

配置DNS

vim /etc/sysconfig/network-scripts/ifcfg-eth0

2016-07-11_104756.jpg

关闭selinux###

vim /etc/sysconfig/selinux

2016-07-11_104937.jpg

修改主机名###

vim /etc/hosts
192.168.159.111 bigdata-senior01.beifeng.com 192.168.159.112 bigdata-senior02.beifeng.com 192.168.159.113 bigdata-senior03.beifeng.com

2016-07-11_105224.jpg

关闭iptables###

service iptables stop
chkconfig iptables off

2016-07-11_105447.jpg

配置目录，上传对应的包###

2016-07-11_105703.jpg

chown修改权限###

chown -R beifeng:beifeng /opt/*

2016-07-11_110004.jpg

卸载openJDK，安装JDK和MAVEN###

rpm -qa | grep jdk
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

2016-07-11_110716.jpg

tar -zxf jdk-7u67-linux-x64.tar.gz -C /opt/modules/
tar -zxf apache-maven-3.0.5-bin.tar.gz -C /opt/modules/
tar -zxf hadoop-2.5.0.tar.gz -C /opt/modules/

删除掉： /opt/modules/hadoop-2.5.0/lib/native的包并替换####

删除掉：/opt/modules/hadoop-2.5.0/share/doc####

2016-07-11_111251.jpg

配置JDK和MAVEN环境变量###

vim /etc/profile

#JAVA_HOME
export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

#MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

2016-07-11_111950.jpg

source /etc/profile

2016-07-11_112111.jpg

配置hadoop相关xml文件##

配置*-evn.sh文件export JAVA_HOME###

2016-07-11_112341.jpg

Hadoop-env.sh，yarn-env.sh，mapred-env.sh 这3个文件配置export JAVA_HOME

hadoop-env.sh:

2016-07-11_112715.jpg

yarn-env.sh:

2016-07-11_112834.jpg

mapred-env.sh:

2016-07-11_112929.jpg

配置自定义文件*-site.xml###

     配置yarn-site.xml
            
             yarn.nodemanager.aux-services
              mapreduce_shuffle
            
    
             
 yarn.resourcemanager.hostname
        bigdata-senior01.beifeng.com
    
    
    
        yarn.log-aggregation-enable
        true
    
    
    
        yarn.log-aggregation.retain-seconds
        106800
    

    配置salves：
    `bigdata-senior01.beifeng.com`

    配置：mapred-site.xml
    
                                                           mapreduce.framework.name
        yarn
    
    
    配置：hdfs-site.xml
    
        dfs.replication
        1
    
    
    
        dfs.permissions.enabled
        false
    
    
    
        dfs.namenode.secondary.http-address
        bigdata-senior01.beifeng.com:50090
    
    配置core-site.xml
    
        fs.defaultFS
        hdfs://bigdata-senior01.beifeng.com:8020
    
    
    
        hadoop.tmp.dir
        /opt/modules/hadoop-2.5.0/data/tmp
    
    
    
        hadoop.http.staticuser.user
        beifeng