Hbase node decommission

You can stop an individual region server by running the following script in the HBase directory on the particular node:

$ ./bin/hbase-daemon.sh stop regionserver

The region server will first close all regions and then shut itself down. On shutdown, the region server's ephemeral node in ZooKeeper will expire. The master will notice the RegionServer gone and will treat it as a crashed server: it will reassign the regions the server was carrying.

Disabling the Load Balancer before Decommissioning a Node

If the load balancer runs while a node is shutting down, then there could be contention between the Load Balancer and the master's recovery of the just decommissioned region server. Avoid any problems by disabling the balancer first: use the shell to disable the balancer like so:

hbase(main):001:0> balance_switch false
true
0 row(s) in 0.3590 seconds

This turns the balancer OFF. To re-enable it, enter the following:

hbase(main):002:0> balance_switch true
false
0 row(s) in 0.3590 seconds

A downside to the above stop of a RegionServer is that regions could be offline for a good period of time - up to the configured ZooKeeper timeout period. Regions are closed in order. If there are many regions on the server, the first region to close may not be back online until all regions close and after the master notices the region server's ZooKeeper znode being removed.

HBase 0.90.2 introduced the facility for having a node gradually shed its load and then shutdown itself down. This is accomplished with the graceful_stop.sh script. When you invoke it without any parameters you are presented with its usage explanation:

$ ./bin/graceful_stop.sh 
Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] \
                        [--thrift] [--rest] &hostname>
 thrift      If we should stop/start thrift before/after the hbase stop/start
 rest        If we should stop/start rest before/after the hbase stop/start
 restart     If we should restart after graceful stop
 reload      Move offloaded regions back on to the stopped server
 debug       Move offloaded regions back on to the stopped server
 hostname    Hostname of server we are to stop

When you want to decommission a loaded region server, run the following:

$ ./bin/graceful_stop.sh HOSTNAME

where HOSTNAME is the host carrying the region server you want to decommission.

Note
The HOSTNAME passed to graceful_stop.sh must match the hostname that HBase is using to identify region Servers. Check the list of region servers in the master UI for how HBase is referring to each servers. Its usually hostname, but can also be a FQDN, for example hostname.foobar.com. Whatever HBase is using, this is what you should pass the graceful_stop.sh decommission script.

If you pass IP addresses, the script is not yet smart enough to make a hostname (or FQDN) out of it and will fail when it checks if the server is currently running: the graceful unloading of regions will not run.

The graceful_stop.sh script will move the regions off the decommissioned region server one at a time to minimize region churn. It will verify the region deployed in the new location before it will move the next region and so on until the decommissioned server is carrying no more regions.

At this point, the graceful_stop.sh tells the region server stop. The master will at this point notice the region server gone but all regions will have already been redeployed and because the region server went down cleanly, there will be no WAL logs to split.