hadoop/hbase 环境搭建
hadoop安装
（1）创建hadoop用户组和用户 见（1.1，1.5）
（2）配置hadoop用户信任关系
1）生成 非对称密钥 ：ssh-keygen -t rsa // 一直回车
1）znode01,znode02,znode03 添加信任关系
(1)znode01 无密码访问znode02
(2)在znode02 ：mkdir ~/.ssh/tmp ; touch ~/.ssh/authorized_keys;
(3)在znode01 ：scp ~/.ssh/id_rsa.pub hdpusr@znode02:~/.ssh/tmp //此时输入znode02密码即可
(4)在znode02 ： cat ~/.ssh/tmp/id_rsa.pub >> ~/.ssh/authorized_keys
(5)znode01 ssh 无密码访问自己 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
......重复以上相似操作配置其他机器
（3）安装包hadoop-2.6.0.zip准备
mkdir ~/env;mkdir ~/env/hadoop；将hadoop-2.6.0.zip 解压到hadoop
（4）配置文件修改
（1）修改hadoop-env.sh
配置JAVA_HOME: export JAVA_HOME=/usr/software/jdk1.7.0_79
（2）修改core-site.xml
（3）修改hdfs-stie.xml
（4）修改mapred-site.xml
（5）修改yarn-site.xml
（6）修改slaves
vim slaves
添加
znode02
znode03
（7）创建hadoop目录
mkdir -p ~/env/hadoop/{tmp,dfs/jn,dfs/dn/data01}
（5）环境变量配置
su - hdpusr01
vim ~/.bash_profile
export HADOOP_HOME=${HOME}/env/hadoop
export HADOOP_CONF_DIR=$HOME/env/hadoop/etc/hadoop
#export HADOOP_PID_DIR=$HOME/env/hadoop/pid
export HADOOP_LOG_DIR=$HOME/env/hadoop/logs
export HADOOP_LIBEXEC_DIR=$HOME/env/hadoop/libexec
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_LIB=${HADOOP_HOME}/lib
export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_YARN_USER=pochdp04
export YARN_CONF_DIR=${HADOOP_CONF_DIR}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_LOG_DIR=${HADOOP_LOG_DIR}
export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn
source ~/.bash_profile 使环境变量生效
（6）格式化操作，启动停止操作
(1) 保证zk正常运行
(2)znode01,znode02,znode03 启动sbin目录下执行：./hadoop-daemon.sh start journalnode
28365 JournalNode --正常的情况可以看到该进程
（3）格式化hdfs
znode01 节点bin目录执行：./hdfs namenode -format
显示：
15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417
15/12/18 13:34:35 INFO common.Storage: Storage directory /aifs01/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.
15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0
【将/home/hdpusr/env/hadoop/tmp 拷贝之znode02 /home/hdpusr/env/hadoop/tmp 下】
（4）格式化ZK
znode01 节点bin目录执行：./hdfs zkfc -formatZK
显示：
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down
成功后zk 出现节点 hadoop-ha
[zk: 127.0.0.1:29181(CONNECTED) 1] ls /
[consumers, config, hadoop-ha, zookeeper, brokers, admin, controller_epoch]
（5）启动分布式存储 znode01 sbin上执行： ./start-dfs.sh
启动成功
znode01 进程：
15507 NameNode
14371 JournalNode
23478 DFSZKFailoverController
znode02 进程：
23870 DataNode
23474 JournalNode
23998 DFSZKFailoverController
znode03 进程
16268 JournalNode
16550 DataNode
同时zk注册节点
[zk: 127.0.0.1:29181(CONNECTED) 4] ls /hadoop-ha/hdpcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
此时可在页面输入http://118.190.79.15:59070 访问
（6）启动分布式资源管理
znode01 sbin上执行：./start-yarn.sh
此时znode01,
出现进程：17746 ResourceManager
znode02
出现进程：24975 NodeManager
znode03
出现进程17389 NodeManager
Zk注册节点 ：rmstore, yarn-leader-election
[zk: 127.0.0.1:29181(CONNECTED) 0] ls /
[hadoop-ha, admin, zookeeper, consumers, config, rmstore, yarn-leader-election, brokers, controller_epoch]
(7) 全部启动：./start-all.sh，全部停止./stop-all.sh
[hdpusr@znode01 sbin]$ ./stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
17/03/25 09:56:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [znode01 znode02]
znode02: no namenode to stop
znode01: stopping namenode
znode03: stopping datanode
znode02: stopping datanode
Stopping journal nodes [znode01 znode02 znode03]
znode03: stopping journalnode
znode02: stopping journalnode
znode01: stopping journalnode
17/03/25 09:57:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping ZK Failover Controllers on NN hosts [znode01 znode02]
znode02: stopping zkfc
znode01: no zkfc to stop
stopping yarn daemons
stopping resourcemanager
znode02: stopping nodemanager
znode03: stopping nodemanager
no proxyserver to stop
[hdpusr@znode01 sbin]$ jps
22992 Jps
[hdpusr@znode01 sbin]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
17/03/25 09:57:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [znode01 znode02]
znode02: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode02.out
znode01: starting namenode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-namenode-znode01.out
znode03: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode03.out
znode02: starting datanode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-datanode-znode02.out
Starting journal nodes [znode01 znode02 znode03]
znode02: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode02.out
znode03: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode03.out
znode01: starting journalnode, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-journalnode-znode01.out
17/03/25 09:57:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting ZK Failover Controllers on NN hosts [znode01 znode02]
znode02: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode02.out
znode01: starting zkfc, logging to /home/hdpusr/env/hadoop/logs/hadoop-hdpusr-zkfc-znode01.out
starting yarn daemons
starting resourcemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-resourcemanager-znode01.out
znode03: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode03.out
znode02: starting nodemanager, logging to /home/hdpusr/env/hadoop/logs/yarn-hdpusr-nodemanager-znode02.out
（8）查看集群状态 bin目录 ./hdfs dfsadmin -report
（9）http://118.190.79.15:59070/dfshealth.html#tab-overview
//////////////////////////////////////////////////////////////
***********************************************hadoop完全分布式部署**********************************************
0.集群规划
主机名 ip地址 安装的软件 运行的进程
HBS01 10.46.52.30 hadoop,hbase namenode,zkfc,resourcemanager
HBS02 10.46.52.31 hadoop namenode,zkfc,resourcemanager
HBS03 10.46.52.32 hadoop,hbase datanode
HBS04 10.46.52.33 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS05 10.46.52.34 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
HBS06 10.46.52.35 hadoop,zookeeper,hbase datanode,nodemanager,journalnode
1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 351 -g hadoop -G ibss -d /zn/users/hdpusr01 -m hdpusr01
passwd hdpusr01 --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr01/.bash_profile
useradd -u 352 -g hbase -G ibss -d /zn/users/hbsusr01 -m hbsusr01
passwd hbsusr01 --stdin <<< Linuhbs_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hbsusr01/.bash_profile
vi /etc/hosts
10.46.52.30 HBS01
10.46.52.31 HBS02
10.46.52.32 HBS03
10.46.52.33 HBS04
10.46.52.34 HBS05
10.46.52.35 HBS06
2.配置互信
--每台主机都需要操作
su - hdpusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys
--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS02:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hdpusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hdpusr01/.ssh/authorized_keys
--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date
3.安装jdk
which java
4.安装Hadoop
4.1 安装zookkeper集群(HBS04 - HBS06)
--配置zookeeper集群
[root@HBS04 hdpusr01]# tar xvf /opt/software/zookeeper.ccs01.tgz
[root@HBS04 hdpusr01]# chown -R hdpusr01:hadoop zookeeper/
[root@HBS04 hdpusr01]# rm -rf zookeeper/logs/*
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/version-2/
[root@HBS04 hdpusr01]# rm -rf zookeeper/zk-data/zookeeper_server.pid
[root@HBS04 hdpusr01]# vi zookeeper/conf/zoo.cfg
dataDir=/zn/users/hdpusr01/zookeeper/zk-data
server.1=HBS04:29888:39889
server.2=HBS05:29888:39889
server.3=HBS06:29888:39889
hdpusr01@HBS04:/zn/users/hdpusr01> vi zookeeper/bin/zkEnv.sh
export ZOOCFGDIR=/zn/users/hdpusr01/zookeeper/conf
export ZOO_LOG_DIR=/zn/users/hdpusr01/zookeeper/logs
[root@HBS04 hdpusr01]# su - hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> tar cvf zkper.tar zookeeper/
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS05:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp zkper.tar HBS06:/zn/users/hdpusr01
hdpusr01@HBS05:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS05:/zn/users/hdpusr01> echo 2 >zookeeper/zk-data/myid
hdpusr01@HBS06:/zn/users/hdpusr01> tar xvf zkper.tar
hdpusr01@HBS06:/zn/users/hdpusr01> echo 3 >zookeeper/zk-data/myid
--启动zookeeper
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh start
4.2 安装hadoop集群（AI-OPT-HBS01 - HBS06）
[root@HBS01 hdpusr01]# tar xvf /opt/software/hadoop-2.6.0.tgz
[root@HBS01 hdpusr01]# chown -R hdpusr01:hadoop hadoop-2.6.0/
hdpusr01@HBS01:/zn/users/hdpusr01> mv hadoop-2.6.0 hadoop
--修改hadoop-env.sh
hdpusr01@HBS01:/zn/users/hdpusr01> vi hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79
--修改core-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi core-site.xml
--修改hdfs-stie.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi hdfs-site.xml
--修改mapred-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi mapred-site.xml
--修改yarn-site.xml
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi yarn-site.xml
--修改slaves
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/etc/hadoop> vi slaves
HBS03
HBS04
HBS05
HBS06
--创建hadoop目录
hdpusr01@HBS01:/zn/users/hdpusr01> mkdir -p hadoop/{tmp,dfs/jn,dfs/dn/data01}
--将配置好的hadoop复制到其他节点
hdpusr01@HBS01:/zn/users/hdpusr01> tar cvf hdp.tar hadoop
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS02:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS03:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS04:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS05:/zn/users/hdpusr01
hdpusr01@HBS04:/zn/users/hdpusr01> scp hdp.tar HBS06:/zn/users/hdpusr01
--解压hadoop包（AI-OPT-HBS02 - HBS05）
tar xvf hdp.tar
--查看zookeeper集群状态（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower
hdpusr01@HBS05:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: leader
hdpusr01@HBS06:/zn/users/hdpusr01> zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /zn/users/hdpusr01/zookeeper/conf/zoo.cfg
Mode: follower
--启动journalnode（AI-OPT-HBS04 - HBS06）
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> ./hadoop-daemon.sh start journalnode
hdpusr01@HBS04:/zn/users/hdpusr01/hadoop/sbin> jps
27035 QuorumPeerMain
28365 JournalNode --正常的情况可以看到该进程
--格式化HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs namenode -format
15/12/18 13:34:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1518459121-10.46.52.30-1450416875417
15/12/18 13:34:35 INFO common.Storage: Storage directory /zn/users/hdpusr01/hadoop/tmp/dfs/name has been successfully formatted.
15/12/18 13:34:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/12/18 13:34:35 INFO util.ExitUtil: Exiting with status 0
--拷贝/zn/users/hdpusr01/hadoop/tmp（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01> scp -r /zn/users/hdpusr01/hadoop/tmp/* HBS02:/zn/users/hdpusr01/hadoop/tmp
--格式化ZK（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs zkfc -formatZK
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=HBS04:29181,HBS05:29181,HBS06:29181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@6456c5f4
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Opening socket connection to server HBS05/10.46.52.34:29181. Will not attempt to authenticate using SASL (unknown error)
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Socket connection established to HBS05/10.46.52.34:29181, initiating session
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: Session establishment complete on server HBS05/10.46.52.34:29181, sessionid = 0x251b09471e80001, negotiated timeout = 80000
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Session connected.
15/12/18 13:49:04 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdpcls1 in ZK.
15/12/18 13:49:04 INFO zookeeper.ZooKeeper: Session: 0x251b09471e80001 closed
15/12/18 13:49:04 INFO zookeeper.ClientCnxn: EventThread shut down
/* 在格式化ZK后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 7] ls /
[hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 8] ls /hadoop-ha
[hdpcls1]
[zk: 10.46.52.34:29181(CONNECTED) 9] ls /hadoop-ha/hdpcls1
[]
--启动HDFS（AI-OPT-HBS01）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-dfs.sh
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
29964 DFSZKFailoverController
29657 NameNode
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop> jps
28086 NameNode
28185 DFSZKFailoverController
hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
27826 DataNode --（AI-OPT-HBS03 - HBS06）
/* 在启动HDFS后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 15] ls /hadoop-ha/hdpcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: 10.46.52.34:29181(CONNECTED) 16] ls /hadoop-ha/hdpcls1/ActiveBreadCrumb
[]
[zk: 10.46.52.34:29181(CONNECTED) 17] ls /hadoop-ha/hdpcls1/ActiveStandbyElectorLock
[]
[zk: 10.46.52.34:29181(CONNECTED) 18] get /hadoop-ha/hdpcls1/ActiveBreadCrumb
hdpcls1nn1
HBS01 ?E(?>
cZxid = 0x10000000a
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x10000000a
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x10000000a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 34
numChildren = 0
[zk: 10.46.52.34:29181(CONNECTED) 19] get /hadoop-ha/hdpcls1/ActiveStandbyElectorLock
hdpcls1nn1
HBS01 ?E(?>
cZxid = 0x100000008
ctime = Fri Dec 18 13:55:41 CST 2015
mZxid = 0x100000008
mtime = Fri Dec 18 13:55:41 CST 2015
pZxid = 0x100000008
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x351b09482220000
dataLength = 34
numChildren = 0
--启动YARN(HBS01由于resourcemanager进程占用资源较多，机器充足的情况下可以考虑将其与namenode进程分开）
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
HBS03: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS03.out
HBS04: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS04.out
HBS05: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS05.out
HBS06: starting nodemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-nodemanager-HBS06.out
注：这个时候主节点上的resourcemanager进程还没启动，从节点上的nodemanager进程已经启动
hdpusr01@HBS03:/zn/users/hdpusr01/hadoop/etc/hadoop> jps
28004 NodeManager --（AI-OPT-HBS03 - HBS06）
--启动resourcemanager
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS01.out
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/sbin> jps
30921 ResourceManager --终于看到这个进程了
注：主机AI-OPT-HBS02上的resourcemanager还没启动，需要手工启动
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /zn/users/hdpusr01/hadoop/logs/yarn-hdpusr01-resourcemanager-HBS02.out
hdpusr01@HBS02:/zn/users/hdpusr01/hadoop/sbin> jps
28945 ResourceManager
/* 在启动YARN和resourcemanager后zookeeper中多出的znode */
[zk: 10.46.52.34:29181(CONNECTED) 22] ls /
[rmstore, yarn-leader-election, hadoop-ha, zookeeper]
[zk: 10.46.52.34:29181(CONNECTED) 23] ls /rmstore
[ZKRMStateRoot]
[zk: 10.46.52.34:29181(CONNECTED) 24] ls /rmstore/ZKRMStateRoot
[AMRMTokenSecretManagerRoot, RMAppRoot, RMVersionNode, RMDTSecretManagerRoot]
[zk: 10.46.52.34:29181(CONNECTED) 25] ls /yarn-leader-election
[rmcls1]
[zk: 10.46.52.34:29181(CONNECTED) 26] ls /yarn-leader-election/rmcls1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
--通过命令查看集群状态
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hdfs dfsadmin -report
5.安装HBase
5.1 安装HBase集群（AI-OPT-HBS01 - HBS06）
--选择一台主机操作（AI-OPT-HBS01）
[root@HBS01 hbsusr01]# tar xvf /opt/software/hbase-1.1.2.tgz
[root@HBS01 hbsusr01]# mv hbase-1.1.2 hbase
[root@HBS01 hbsusr01]# chown -R hbsusr01:hbase hbase
--设置集群节点文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi regionservers
HBS03
HBS04
HBS05
HBS06
--设置hbase配置文件
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-site.xml
--设置hbase环境
hbsusr01@HBS01:/zn/users/hbsusr01/hbase/conf> vi hbase-env.sh
export HBASE_CONF_DIR=/zn/users/hbsusr01/hbase/conf
export HBASE_PID_DIR=/zn/users/hbsusr01/hbase/pid
export HBASE_LOG_DIR=/zn/users/hbsusr01/hbase/logs
# export HBASE_MANAGES_ZK=true
export HBASE_MANAGES_ZK=false
--创建相关目录
hbsusr01@HBS01:/zn/users/hbsusr01> mkdir -p hbase/{conf,pid,logs,tmp}
--拷贝文件到其他节点并解压
hbsusr01@HBS01:/zn/users/hbsusr01> tar cvf hbase.tar hbase
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS03:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS04:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS05:/zn/users/hbsusr01
hbsusr01@HBS01:/zn/users/hbsusr01> scp hbase.tar HBS06:/zn/users/hbsusr01
--配置ssh互信
su - hbsusr01
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 .ssh/authorized_keys
--选择一台master操作(HBS01)
ssh HBS02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS04 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS05 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh HBS06 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS02:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS03:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS04:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS05:/zn/users/hbsusr01/.ssh/authorized_keys
scp ~/.ssh/authorized_keys HBS06:/zn/users/hbsusr01/.ssh/authorized_keys
--验证配置是否生效
ssh HBS01 date
ssh HBS02 date
ssh HBS03 date
ssh HBS04 date
ssh HBS05 date
ssh HBS06 date
--解压hadoop包（AI-OPT-HBS03 - HBS06）
tar xvf hbase.tar
--启动hbase
start-hbase.sh
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbsusr01, access=WRITE, inode="/":hdpusr01:supergroup:drwxr-xr-x
解决方法：hadoop fs添加目录和权限
hdpusr01@HBS01:/zn/users/hdpusr01> hadoop fs -mkdir /hbase
-mkdir: java.net.UnknownHostException: host-10-1-241-18
临时解决：
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop fs -mkdir /hbase
hdpusr01@HBS01:/zn/users/hdpusr01/hadoop/bin> ./hadoop fs -chown hbsusr01:hbase /hbase
/*
解决方法：配置环境变量
su - hdpusr01
export HADOOP_CONF_DIR=$HOME/hadoop/etc/hadoop
#export HADOOP_PID_DIR=$HOME/hadoop/pid
export HADOOP_LOG_DIR=$HOME/hadoop/logs
export HADOOP_LIBEXEC_DIR=$HOME/hadoop/libexec
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_LIB=${HADOOP_HOME}/lib
export HADOOP_LIBRARY_PATH=${HADOOP_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_YARN_USER=pochdp04
export YARN_CONF_DIR=${HADOOP_CONF_DIR}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_LOG_DIR=${HADOOP_LOG_DIR}
export YARN_PID_DIR=${HADOOP_CONF_DIR}/../yarn
*/
su - hbsusr01
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/zn/users/hdpusr01/hadoop/lib/native
export HBASE_HOME=/zn/users/hbsusr01/hbase
export HBASE_CONF_DIR=$HBASE_HOME/conf
export HBASE_LOG_DIR=$HBASE_HOME/logs
export HBASE_PID_DIR=$HBASE_HOME/pid
export PATH=$PATH:$HBASE_HOME/bin
http://blog.itpub.net/20777547/viewspace-1745820/
http://www.aboutyun.com/thread-11909-1-1.html
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/
***********************************************hadoop伪分布式部署**********************************************
1.创建用户以及配置主机名
mkdir -p /zn/users
useradd -u 451 -g hadoop -G ibss -d /zn/users/hdpusr02 -m hdpusr02
passwd hdpusr02 --stdin <<< Linuhdp_0805
echo -e "\n. ~puwadm/wprofile\n" >> ~hdpusr02/.bash_profile
2.配置ssh互相
su - hdpusr02
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh AI-UAT-OPT-HBASE03 date
3.解压文件
tar xvf /opt/software/hadoop-2.6.0.tgz
mv hadoop-2.6.0/ hadoop
4.配置hadoop
vi hadoop-env.sh
export JAVA_HOME=/opt/freeware/jdk1.7.0_79
vi core-site.xml
vi mapred-site.xml
vi yarn-site.xml
vi hdfs-site.xml
vi slaves
AI-UAT-OPT-HBASE03
5.创建相应目录
mkdir -p hadoop/dfs/{name,data}
6.启动hadoop
bin/hdfs namenode -format
sbin/start-dfs.sh
sbin/start-yarn.sh