一个region server包含了五部分功能:
1.和zookeeper相关的线程
MasterAddressTracker负责捕获master节点的状态
ClusterStatusTracker追踪hbase集群的状态
CatalogTracker跟踪root表meta表和region的状态
SplitlogWorker竞争获取znode上的splitlog,并切分HLog按照region分组,放到相应region
的recovered.edits目录下
2.region相关的线程
regionserver包含了一个region的集合,每个具体的操作会分到一个指定的region去处理
CompactionChecker用于周期性的检查是否需要compact,如需要交给CompactSplitThread处理
CompactSplitThread用于合并和切分处理的线程
MemStoreFlusher如果memstore满了则flush到HDFS中
3.WAL相关
HLog按照hbase的架构,一个regionserver只有一个hlog,多个region是共享的
LogRoller用于日志回滚
4.和客户端通讯
RPC server模块,这里包含了很多线程,listener,select,handler线程
Leases 用于租借时间检查
5.和master及监控相关
HMasterRegionInterface用户管理hbase
HServerLoad检查hbase负载,并和master通讯
HealthCheckChore服务的监控检查
RegionServerMetrics 获取metrics相关的数据
web server,启用一个jettyserver,可以监控region相关的信息
RegionServer的相关配置
参数名称 | 默认值 | 含义 |
hbase.client.retries.number | 10 | 客户端的重试次数 |
hbase.regionserver.msginterval | 3000 | 未知 |
hbase.regionserver.checksum.verify | false | 是否启用hbase的 checksum |
hbase.server.thread.wakefrequency | 10秒 | 检查线程的频率 |
hbase.regionserver.numregionstoreport | 10 | 未知 |
hbase.regionserver.handler.count | 10 | 处理用户表的工作 线程数量 |
hbase.regionserver.metahandler.count | 10 | 处理meta和root表 的工作线程数量 |
hbase.rpc.verbose | false | 未知 |
hbase.regionserver.nbreservationblocks | false | 未知 |
hbase.regionserver.compactionChecker. majorCompactPriority |
max int | 未知 |
hbase.regionserver.executor.openregion.threads | 3 | 开打用户表region 的线程数量 |
hbase.regionserver.executor.openroot.threads | 1 | 打开root表region 的线程数量 |
hbase.regionserver.executor.openmeta.threads | 1 | 打开meta表region 的线程数量 |
hbase.regionserver.executor.closeregion.threads | 3 | 关闭用户表region 的线程数量 |
hbase.regionserver.executor.closeroot.threads | 1 | 关闭root表region 的线程数量 |
hbase.regionserver.executor.closemeta.threads | 1 | 关闭meta表region 的线程数量 |
HRegionServer的启动入口类
org.apache.hadoop.hbase.regionserver.HRegionServer
hbase-site.xml中可以配置参数 hbase.regionserver.impl来自定自己的实现,但必须继承HRegionServer
之后调用HRegionServerCommandLine (这个类继承自ServerCommandLine,所以master也有一个实现)
HRegionServerCommandLine使用hadoop提供的ToolRunner去运行
ToolRunner#run(Configuration,Tool,String[])
ToolRunner会调用GenericOptionsParser,解析一些固定的参数,如-conf,-D,-fs,-files 这样的参数
解析好之后,配置configuration对象,然后将启动参数传给Tool接口的实现
所以ToolRunner 就是一个启动参数解析,配置configuration对象的工具类,然后将这些信息交给Tool实现类
初始化-调用HRgionServer构造函数
HRegionServerCommandLine反射创建HRegionServer(或其自定义子类)
1.这里对客户端连接配置做了一些初始化工作
2.配置host,DNS相关
3.HRegionServer 调用HBaseRPC,创建一个RpcEngine实现,这里是WritableRpcEngine
hbase-site.xml中可以配置参数 hbase.rpc.engine来自定自己的实现,但必须继承RpcEngine接口
HBaseRPC调用 getServer()获得一个具体的RpcServer实现,即通过RpcEngine --> RpcServer
这里获取的是WritableRpcEngine的内部类WritableRpcEngine$Server,它继承了HBaserServer
4.创建metrics线程(for JVM),LRU检查线程
5.连接zookeeper,做一些验证工作(kerbose)
启动,HRegionServer#run (在新线程中启动)
之后就开始启动server了,启动是从HRegionServer#run()开始的(新启动的线程)
1.创建zookeeper监听线程
2.创建和master通讯的线程
3.创建WAL相关的线程
4.创建metrics线程(for hbase)
5.创建日志回滚线程、cache flush线程、compact线程、心跳检查线程、租借检查线程
6.创建jetty线程
7.创建response线程,listener线程,handle线程,高优先级handle(处理meta表)线程,复制handle线程
8.创建日志切分线程
这里还定义了线程池,以后会通过处理请求的时候,可能会开启这些线程:(在第五步的时候定义的)
hbase.regionserver.executor.openregion.threads3(默认)
hbase.regionserver.executor.openroot.threads1
hbase.regionserver.executor.openmeta.threads1
hbase.regionserver.executor.closeregion.threads3
hbase.regionserver.executor.closeroot.threads1
hbase.regionserver.executor.closemeta.threads1
日志切分线程在启动的时候可能会有很多事情要做
之后整个region server就启动完成了
HRegionServer包含一些功能
HRegion集合
Leases(租借时间检查)
HMasterRegionInterface(管理hbase)
HServerLoad(hbase负载)
CompactSplitThread(用于合并处理)
CompactionChecker(周期性的检查是否需要compact,如需要交给CompactSplitThread处理)
MemStoreFlusher(用于刷新memstore)
HLog(WAL相关)
LogRoller(日志回滚)
ZooKeeperWatcher(zk监听)
SplitLogWorker(用于切分日志)
ExecutorService(用户启动open,close HRegion的线程池)
ReplicationSourceService和ReplicationSinkService(replication相关)
HealthCheckChore(健康检查)
RegionServerMetrics(监控)
一些监听类
MasterAddressTracker
CatalogTracker
ClusterStatusTracker
postOpenDeployTasks 用于更新root表或meta表
各种CURD,scanner,increment操作
multi操作(对于delete和put)
对HRegion的flush,close,open(提交到线程池去做)
split,compact操作,这些最终由一个具体的HRegion去完成
RegionServer的线程
用于小合并的
Daemon Thread [regionserver60020-smallCompactions-1392958977368] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
PriorityBlockingQueue
ThreadPoolExecutor.getTask() line: 957
ThreadPoolExecutor$Worker.run() line: 917
Thread.run() line: 662
打开用户表region的线程
Thread [RS_OPEN_REGION-myhost,60020,1392868973177-0] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
ExecutorService$TrackingThreadPoolExecutor(ThreadPoolExecutor).getTask() line: 957
ThreadPoolExecutor$Worker.run() line: 917
Thread.run() line: 662
这个是跟zookeeper通讯的线程
Daemon Thread [PostOpenDeployTasks:1028785192-EventThread] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
ClientCnxn$EventThread.run() line: 491
跟zookeeper通讯的线程
Daemon Thread [PostOpenDeployTasks:1028785192-SendThread(myhost:2181)] (Suspended)
EPollArrayWrapper.epollWait(long, int, long, int) line: not available [native method]
EPollArrayWrapper.poll(long) line: 210
EPollSelectorImpl.doSelect(long) line: 65
EPollSelectorImpl(SelectorImpl).lockAndDoSelect(long) line: 69
EPollSelectorImpl(SelectorImpl).select(long) line: 80
ClientCnxnSocketNIO.doTransport(int, List
ClientCnxn$SendThread.run() line: 1068
专门处理META表的线程
Thread [RS_OPEN_META-myhost,60020,1392868973177-0] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
ExecutorService$TrackingThreadPoolExecutor(ThreadPoolExecutor).getTask() line: 957
ThreadPoolExecutor$Worker.run() line: 917
Thread.run() line: 662
专门处理ROOT表的线程
Thread [RS_OPEN_ROOT-myhost,60020,1392868973177-0] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
ExecutorService$TrackingThreadPoolExecutor(ThreadPoolExecutor).getTask() line: 957
ThreadPoolExecutor$Worker.run() line: 917
Thread.run() line: 662
日志拆分线程
Thread [SplitLogWorker-myhost,60020,1392868973177] (Suspended)
Object.wait(long) line: not available [native method]
Object.wait() line: 485
SplitLogWorker.taskLoop() line: 219
SplitLogWorker.run() line: 179
Thread.run() line: 662
jetty的扫描线程
Daemon Thread [Timer-0] (Suspended)
Object.wait(long) line: not available [native method]
TimerThread.mainLoop() line: 509
TimerThread.run() line: 462
jetty的接收线程
Thread [1142818380@qtp-1463348369-1 - Acceptor0 [email protected]:60030] (Suspended)
EPollArrayWrapper.epollWait(long, int, long, int) line: not available [native method]
EPollArrayWrapper.poll(long) line: 210
EPollSelectorImpl.doSelect(long) line: 65
EPollSelectorImpl(SelectorImpl).lockAndDoSelect(long) line: 69
EPollSelectorImpl(SelectorImpl).select(long) line: 80
SelectorManager$SelectSet.doSelect() line: 498
SelectChannelConnector$1(SelectorManager).doSelect(int) line: 192
SelectChannelConnector.accept(int) line: 124
AbstractConnector$Acceptor.run() line: 708
QueuedThreadPool$PoolThread.run() line: 582
jetty的工作线程
Thread [869247333@qtp-1463348369-0] (Suspended)
Object.wait(long) line: not available [native method]
QueuedThreadPool$PoolThread.run() line: 626
租借检查相关线程
Daemon Thread [regionserver60020.leaseChecker] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.parkNanos(Object, long) line: 196
AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2025
DelayQueue
Leases.run() line: 83
Thread.run() line: 662
检查压缩的线程
Daemon Thread [regionserver60020.compactionChecker] (Suspended)
Object.wait(long) line: not available [native method]
Sleeper.sleep(long) line: 91
HRegionServer$CompactionChecker(Chore).run() line: 75
Thread.run() line: 662
检查memstore flush的线程
Daemon Thread [regionserver60020.cacheFlusher] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.parkNanos(Object, long) line: 196
AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2025
DelayQueue
DelayQueue
MemStoreFlusher.run() line: 220
Thread.run() line: 662
日志回滚的线程
Daemon Thread [regionserver60020.logRoller] (Suspended)
Object.wait(long) line: not available [native method]
LogRoller.run() line: 77
Thread.run() line: 662
metrics线程
Daemon Thread [Timer thread for monitoring jvm] (Suspended)
Object.wait(long) line: not available [native method]
TimerThread.mainLoop() line: 509
TimerThread.run() line: 462
日志同步线程
Daemon Thread [regionserver60020.logSyncer] (Suspended)
Object.wait(long) line: not available [native method]
HLog$LogSyncer.run() line: 1265
Thread.run() line: 662
HDFS客户端租借检查线程
Daemon Thread [LeaseChecker] (Suspended)
Thread.sleep(long) line: not available [native method]
DFSClient$LeaseChecker.run() line: 1379
Daemon(Thread).run() line: 662
未知
Thread [regionserver60020] (Suspended)
Object.wait(long) line: not available [native method]
Sleeper.sleep(long) line: 91
Sleeper.sleep() line: 55
HRegionServer.run() line: 787
Thread.run() line: 662
LRU相关的线程
Daemon Thread [LRU Statistics #0] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.parkNanos(Object, long) line: 196
AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2025
DelayQueue
ScheduledThreadPoolExecutor$DelayedWorkQueue.take() line: 609
ScheduledThreadPoolExecutor$DelayedWorkQueue.take() line: 602
ScheduledThreadPoolExecutor(ThreadPoolExecutor).getTask() line: 957
ThreadPoolExecutor$Worker.run() line: 917
Thread.run() line: 662
LRU缓存检查线程
Daemon Thread [main.LruBlockCache.EvictionThread] (Suspended)
Object.wait(long) line: not available [native method]
LruBlockCache$EvictionThread(Object).wait() line: 485
LruBlockCache$EvictionThread.run() line: 612
Thread.run() line: 662
RPC监控线程
Daemon Thread [Timer thread for monitoring rpc] (Suspended)
Object.wait(long) line: not available [native method]
TimerThread.mainLoop() line: 509
TimerThread.run() line: 462
reader线程
Daemon Thread [IPC Reader 0 on port 60020] (Suspended)
EPollArrayWrapper.epollWait(long, int, long, int) line: not available [native method]
EPollArrayWrapper.poll(long) line: 210
EPollSelectorImpl.doSelect(long) line: 65
EPollSelectorImpl(SelectorImpl).lockAndDoSelect(long) line: 69
EPollSelectorImpl(SelectorImpl).select(long) line: 80
EPollSelectorImpl(SelectorImpl).select() line: 84
HBaseServer$Listener$Reader.doRunLoop() line: 528
HBaseServer$Listener$Reader.run() line: 514
ThreadPoolExecutor$Worker.runTask(Runnable) line: 895
ThreadPoolExecutor$Worker.run() line: 918
Thread.run() line: 662
工作线程(可以配置多个) REPL是用于复制的,PRI是用于处理META表的,IPC是普通的工作线程
Daemon Thread [REPL IPC Server handler 0 on 60020] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
HBaseServer$Handler.run() line: 1398
Daemon Thread [PRI IPC Server handler 0 on 60020] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
HBaseServer$Handler.run() line: 1398
Daemon Thread [IPC Server handler 1 on 60020] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 156
AbstractQueuedSynchronizer$ConditionObject.await() line: 1987
LinkedBlockingQueue
HBaseServer$Handler.run() line: 1398
select线程
Daemon Thread [IPC Server listener on 60020] (Suspended)
EPollArrayWrapper.epollWait(long, int, long, int) line: not available [native method]
EPollArrayWrapper.poll(long) line: 210
EPollSelectorImpl.doSelect(long) line: 65
EPollSelectorImpl(SelectorImpl).lockAndDoSelect(long) line: 69
EPollSelectorImpl(SelectorImpl).select(long) line: 80
EPollSelectorImpl(SelectorImpl).select() line: 84
HBaseServer$Listener.run() line: 636
响应线程
Daemon Thread [IPC Server Responder] (Suspended)
EPollArrayWrapper.epollWait(long, int, long, int) line: not available [native method]
EPollArrayWrapper.poll(long) line: 210
EPollSelectorImpl.doSelect(long) line: 65
EPollSelectorImpl(SelectorImpl).lockAndDoSelect(long) line: 69
EPollSelectorImpl(SelectorImpl).select(long) line: 80
HBaseServer$Responder.doRunLoop() line: 825
HBaseServer$Responder.run() line: 808
未知
Daemon Thread [hbase-tablepool-1-thread-1] (Suspended)
Object.wait(long) line: not available [native method]
HBaseClient$Call(Object).wait() line: 485
HBaseClient.call(Writable, InetSocketAddress, Class
WritableRpcEngine$Invoker.invoke(Object, Method, Object[]) line: 86
$Proxy14.multi(MultiAction) line: not available
HConnectionManager$HConnectionImplementation$3$1.call() line: 1427
HConnectionManager$HConnectionImplementation$3$1.call() line: 1425
HConnectionManager$HConnectionImplementation$3$1(ServerCallable
HConnectionManager$HConnectionImplementation$3.call() line: 1434
HConnectionManager$HConnectionImplementation$3.call() line: 1422
FutureTask$Sync.innerRun() line: 303
FutureTask
ThreadPoolExecutor$Worker.runTask(Runnable) line: 895
ThreadPoolExecutor$Worker.run() line: 918
Thread.run() line: 662
总体来说线程有这么一些
1.日志回滚,日志syn同步,日志切分
2.大合并,小合并线程
3.LRU缓存检查,memstore检查
4.HDFS客户端,HDFS客户端超时检查,zookeeper通讯
5.专门处理root表,专门处理meta表
6.jetty线程
7.listener线程,reader线程,handle线程,响应线程,用于处理META的handle线程,用于复制的handle线程
postOpenDeployTask线程(用于更新META表)
具体逻辑如下:
leaseChecker线程(执行超时后销毁这些操作)
这个类的作用是当某些执行超时,比如get,scan等,需要释放相应的scan或者行锁等
这里是在异步的线程中执行的
具体逻辑如下:
参考
HBase深入分析之RegionServer
Hbase系统架构及数据结构
HRegionServer启动过程