Sqoop工具MySQL导入数据到Hbase（Hbase部署）

基础环境

集群工具版本
sqoop-1.4.7.bin__hadoop-2.6.0
mysql驱动，插件包jar 提取码：LWXB
hadoop-2.6.5
hbase-2.2.6-bin
以上是集群和工具的版本
hadoop集群搭建这里就不说了，搭建yarn-Ha集群模式

---

Hadoop集群搭建

---

Hbase部署

192.168.17.128 xiaobai1
192.168.17.132 xiaobai2
192.168.17.133 xiaobai3

#三台机器都操作
[xiaobai@xiaobai1] vim /etc/hosts
192.168.17.128 xiaobai1
192.168.17.132 xiaobai2
192.168.17.133 xiaobai3


[xiaobai@xiaobai1] mkdir /apps/svr/hbase/
[xiaobai@xiaobai1] tar xzvf hbase-2.2.6-bin.tar.gz -C /apps/svr/hbase/
[xiaobai@xiaobai1] cd /apps/svr/hbase/hbase-2.2.6
[xiaobai@xiaobai1 hbase-2.2.6] cd conf
[xiaobai@xiaobai1 conf] cp hbase-site.xml hbase-site.xml.bak
[xiaobai@xiaobai1 conf] vim hbase-site.xml
<configuration>
  <property>
        <name>hbase.rootdir</name>
        <value>hdfs://mycluster/hbase</value>
  </property>
  <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
  </property>
  <property>
        <name>hbase.zookeeper.quorum</name>
        <value>xiaobai1:2181,xiaobai2:2181,xiaobai3:2181</value>
  </property>
  <property>
        <name>hbase.unsafe.stream.capability.enforce</name>
        <value>false</value>
  </property>
</configuration>


[xiaobai@xiaobai1 conf] cp hbase-env.sh hbase-env.sh.bak
[xiaobai@xiaobai1 conf] vim hbase-env.sh
export HBASE_MANGES_ZK=false
export JAVA_HOME=/apps/svr/java/jdk1.8.0_111


#把hdaoop下的hdfs-site.xml拷到habse的conf中
[xiaobai@xiaobai1 conf] cp /apps/svr/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml .

[xiaobai@xiaobai1 conf] vim regionservers   #配置集群
xiaobai1
xiaobai2
xiaobai3

[xiaobai@xiaobai1 conf] vim backup-masters   #配置备用主机
xiaobai2

[xiaobai@xiaobai1 conf] vim /etc/profile.d/hbase.sh
export HBASE_HOME=/apps/svr/hbase/hbase-2.2.6
export PATH=$PATH:$HBASE_HOME/bin

[xiaobai@xiaobai1 conf] source /etc/profile
[xiaobai@xiaobai1 conf] source hbase-env.sh

hbase集群搭建好了启动zookeeper，hadoop

#在三台机器上操作
[xiaobai@xiaobai1 conf] cd /apps/svr/zookeeper/zookeeper-3.4.8/bin
[xiaobai@xiaobai1 bin] ./zkServer.sh start
[xiaobai@xiaobai1 bin] jps
1213 QuorumPeerMain
1294 Jps


#在一台hadoop的namenode节点上启动
[xiaobai@xiaobai1 bin] cd
[xiaobai@xiaobai1] start-all.sh
[xiaobai@xiaobai1] jps
5153 ResourceManager
4883 JournalNode
5267 NodeManager
4597 NameNode
5049 DFSZKFailoverController
5593 Jps
4698 DataNode
1213 QuorumPeerMain
#只有主节点有一个ResourceManager进程，所以我们要去备用节点单独启动一个ResourceManager


#在另一个yarn的备用节点启动
[xiaobai@xiaobai2] yarn-daemon.sh start resourcemanager   #启动单个节点的ResourceManager
[xiaobai@xiaobai2] jps
5153 ResourceManager
4883 JournalNode
5267 NodeManager
4597 NameNode
5800 Jps
5049 DFSZKFailoverController
4698 DataNode
1213 QuorumPeerMain
#xiaobai2节点上的服务和xiaobai1一样了


#在hbase的主节点上启动
[xiaobai@xiaobai3] start-hbase.sh
[xiaobai@xiaobai3] jps
1202 QuorumPeerMain
2695 NodeManager
2600 JournalNode
4667 HMaster   #hbase主节点
4909 Jps
2511 DataNode
4815 HRegionServer   #和DataNode在同一台机器上运行，实现数据的本地性。


[xiaobai@xiaobai2] jps
1200 QuorumPeerMain
3920 ResourceManager
5552 HMaster
5664 Jps
3073 NameNode
5478 HRegionServer
3431 NodeManager
3144 DataNode
3340 DFSZKFailoverController
3215 JournalNode


[xiaobai@xiaobai1] jps
5153 ResourceManager
4883 JournalNode
5267 NodeManager
8100 Jps
5049 DFSZKFailoverController
7961 HMaster
4698 DataNode
1213 QuorumPeerMain
7407 HRegionServer


[xiaobai@xiaobai1] hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.2.6, r88c9a386176e2c2b5fd9915d0e9d3ce17d0e456e, Tue Sep 15 17:36:14 CST 2020
Took 0.0018 seconds                                                                                                          
hbase(main):001:0> 

#连接时如果有两个java长报错。翻译是初始化失败就去将一个jar包替换掉
[xiaobai@xiaobai1] cd /apps/svr/hadoop/hadoop-2.6.5/share/hadoop/yarn
[xiaobai@xiaobai1 yarn] rm -rf jline-0.94.jar
#把新版本上传上来
[xiaobai@xiaobai1 yarn] ls
...
jline-2.12.jar
...

#再次hbase shell就OK了

访问IP＋16010端口

文章总览分类目录 @小白