2019-05-15

主流程

• 1.关闭防火墙

• 2.机器间实现免密

• 3.zk部署

• 4.hadoop部署

实现流程

2.免密登录

ssh-keygen

回车一直到最后（有3次）

生成本机秘钥后，同步到自己机器和其他机器

ssh-copy-id host

xshell 可以开启关联，发送键盘到所有会话 3台机器只需要执行三次复制指令即可

ssh host 测试

3.zk部署

-1上解压缩zk压缩包到目录 /opt/module/

tar -zvxf zkXXXX.gz -C /opt/module

cd /opt/module/zk/conf

cp zoo_sample.cfg zoo.cfg

修改zoo为

# example sakes.

dataDir=/opt/module/zookeeper-3.4.10/zkData/

dataLogDir=/opt/module/zookeeper-3.4.10/logs/

# the port at which the clients will connect

clientPort=2181

server.1=98-4.guoding:2888:3888

server.2=98-5.guoding:2888:3888

server.3=98-6.guoding:2888:3888

复制到其他机器

scp -r zk root@host:/opt/module/zk

3台同时启动

bin/zkServer.sh start

4.hadoop 部署

解压缩到/opt/module/hadoop/下

修改etc/hadoop下面的以下文件：

core-site.xml

hdfs-site.xml

yarn-site.xml

mapred-site.xml

hadoop-env.sh

slaves

相应文件内容：

core-site.xml:

    
        fs.defaultFS
            hdfs://nncluster
    
    
        hadoop.tmp.dir
        /opt/module/hadoop-2.8.4/data
    
    
        ha.zookeeper.quorum
        98-4.guoding:2181,98-5.guoding:2181,98-6.guoding:2181
    

hdfs-site.xml

    
        dfs.nameservices
        nncluster
    
    
        dfs.ha.namenodes.nncluster
        nn1,nn2
    

    
        dfs.namenode.rpc-address.nncluster.nn1
        98-4.guoding:9000
    

    
        dfs.namenode.rpc-address.nncluster.nn2
        98-5.guoding:9000
    

    
        dfs.namenode.http-address.nncluster.nn1
        98-4.guoding:50070
    

    
        dfs.namenode.http-address.nncluster.nn2
        98-5.guoding:50070
    

    
        dfs.namenode.shared.edits.dir
    qjournal://98-4.guoding:8485;98-5.guoding:8485;98-6.guoding:8485/nncluster
    

    
        dfs.ha.fencing.methods
        sshfence
    

    
        dfs.ha.fencing.ssh.private-key-files
        /root/.ssh/id_rsa
    

    
        dfs.journalnode.edits.dir
        /opt/module/hadoop-2.8.4/data/jn
    

    
        dfs.permissions.enable
        false
    
    
        dfs.permissions
        false
    

    
        dfs.client.failover.proxy.provider.nncluster
        org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
    
    
        dfs.ha.automatic-failover.enabled
        true
    

yarn-site.xml

 
    yarn.nodemanager.aux-services 
    mapreduce_shuffle 


 
    yarn.resourcemanager.ha.enabled 
    true 


 
    yarn.resourcemanager.cluster-id 
    cluster-yarn1 


 
    yarn.resourcemanager.ha.rm-ids 
    rm1,rm2 


 
    yarn.resourcemanager.hostname.rm1 
    98-4.guoding 


 
    yarn.resourcemanager.hostname.rm2 
    98-5.guoding 


 
    yarn.resourcemanager.zk-address
    98-4.guoding:2181,98-5.guoding:2181,98-6.guoding:2181 


 
    yarn.resourcemanager.recovery.enabled
    true 


 
    yarn.resourcemanager.store.class    
    org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

mapred-site.xml

    
        mapreduce.framework.name
        yarn
    

    
        mapreduce.jobhistory.address
        98-4.guoding:10020
    

    
        mapreduce.jobhistory.webapp.address
        98-5.guoding:19888
    

slaves

98-4.guoding
98-5.guoding
98-6.guoding

hadoop-env.sh 修改jdk执行为真实目录 引用指向不行

hadoop启动

三台执行

./hadoop-daemon.sh start journalnode

nn1格式化并启动namenode

hdfs namenode -format
./hadoop-daemon.sh start namenode

nn2执行同步元数据信息 ,启动namenode

hdfs namenode -bootstrapStandby
./hadoop-daemon.sh start namenode

启动datanode模块 主节点启动会带动其他节点

./hadoop-daemons.sh start datanode

将nn1切换为Active (--forcemanual 问题：脑烈，具体看zk的leader在哪台机器上，强制切换)

   ./hdfs haadmin -transitionToActive  --forcemanual nn1

在nn2上启动yarn集群

./start-yarn.sh

image.png

image.png