Snapshots in HBase 0.96

Snapshots in HBase 0.96 –v0.1 (5/20/12)

 

The initial design document andimplementation for snapshots was proposed in HBASE-50. The original, overalldesign is still valid on the current HBase trunk, but the underlyingimplementation is no longer reasonable to apply to the current codebase. In thelast two years, too much has changed in the HBase implementation in terms oflower level things, e.g. locking mechanisms, and higher-level design paradigms,e.g. managers and monitors.

 

The goal of HBase-6055 is to port up theconcepts from HBase-50, but applying them to the current codebase to enable table-level, point-in-time snapshots.

 

It is currently possible to achievetimestamp-based backup of a table by increasing the retention time on a tableto outlast the time it takes to run the copy (essentially creating a‘snapshot’). However, this is not truly ‘point-in-time’ as different serversare not perfectly clock synchronized, leading to potentially drasticallydifferent table views even over the same clock-tick, given the high-throughputand low-latency nature of HBase (at over a million writes /second, things canchange a lot over the common clock skew seen in NTP). Further, this leads‘snapshot’ method has lots of latency because of its use of M/R and leads to alarge amount of data replication.

 

Point-in-time snapshots can be thought outthe creation of file references, in the copy-on-write paradigm for each regionin a table run a cross-server transaction limited by a timeout. Whew, that’s amouthful!

 

What follows is a more in depth discussionof each of how taking a snapshot actually works. We’ll start with the generaltheory and then move to how the actual implementation works.

 

Point-in-timeSnapshots – the hand-waving theory

 

If you have read the documentation inHBASE-50, you can probably skim, or outright skip, this section and insteadjump right into the nitty-gritty implementation details.

 

For those still around, lets jump rightinto it!

 

It traditional SQL databases, snapshots areenabled via a mechanism known as ‘copy-on-write’. Copy on write semantics canbe simplified to the following: when multiple callers ask for a resource (e.g.a cell in table), everyone gets pointers to the same resource. However, if anyof those callers makes a change to that resource, they get their own local copyto change, while the other viewers still see the original.

 

This ends up working really well forsnapshots because the snapshot always keeps a reference to the old version,while the database just keeps moving on with the newer copies. Taking asnapshot on a single-machine database then amount to writing a single referenceto keep your place. It starts to get a bit more complicated when the databasespans multiple machines and is one of the major concerns with the HBaseimplementation.

 

Don’t worry, we’ll get to that, but for themoment lets look at why copy-on-write is actually a huge boon to HBase as well.

 

The astute reader will have realized bythis point that HBase is actually built around the copy-on-write idea combinedwith a merge-sort to get the most recent version. So it turns out, most of thework for snapshots is built into HBase already!

 

HBase (more specificially HDFS) is builtaround the idea of immutable files, so a point-in-time view is alwaysmaintained as long as can we find the right HFiles (possible if we don’t throwthem away – the point of HBASE-5547) and keep track of the state in thememstore.

 

Then for a snapshot, all we need to do isstore a reference to all those the HFiles and WALs for a table, so we can goback to the right set of files to regenerate the table state. To minimizelatency and maximize throughput, a natural parallelization of the work fallsout of implementation of HBase - the servers hosting the files we need toreference are delegated the work of creating those references.

 

What about compactions you say? Won’t thatblow away the files we just referenced you say? In current implementation ofHBase, yes, this would be a huge problem for snapshots. However, in HBASE-5547the implementation was added to enable ‘backup mode’, wherein HFiles are notdeleted, but instead moved to an hfile archive directory. The beauty here isthat the reference file doesn’t even need to be updated since if you can’t findthe hfile in the original reference, you can do the archive directory and readthe hfile there.

 

So, all we need to do for adding HFiles toa snapshot is just enumerate the files currently being read for a region andwrite references (inherently small files, which are fast to write) to thosefiles. Simple, right?

 

“Woah, hold on,” you might be saying now,“what about the in-memory state in the memstore? That isn’t going to be in theHFiles!” You would be correct -  if wejust referenced the HFiles, we would be missing data.

 

The simple idea would be to first flush thememstore to an HFile and just add that HFile to the set we are archiving.However, that is going to be really painful and dramatically increase the latencyto take a snapshot. Instead, we can capture in-memory state by just keeingtrack of the WAL files as well. As long as we mark where the snapshot startsinto the WAL file for each regionserver and then add references to those WALfiles to the snapshot, we can rebuild the in-memory state by replaying theWALs.

 

The tricky part comes in the point-in-time semantics. Take the caseof two servers A and B. Suppose A is getting hammered with writes but B islightly loaded. At some point, we decide to take a snapshot. When they receivethe snapshot request, A is likely to have a bunch of outstanding writes while Bis going to be able to process the snapshot almost immediately. To get a pointin time, we need to wait for A to process all its outstanding writes and takeits snapshot (write references to all the associated files and put a marker inits WAL) before we let either A or Breceive more writes. We need to get A and B to agree to a single point intime where they can say, “Here is all the data involved in snapshot X.” Thiscreates a coordination bottleneck, where we need to block writes to the tablefor the amount of time it takes to agree on a stable state across the cluster.

 

Essentially, the process is the following:

1.     Tell all the servers hosting atable to get ready to take a snapshot

2.     All the servers prepare for thesnapshot

a.      Stop taking any new writes

b.     Create references to all theirHFiles

c.      Put a write in the WAL

d.     Create references to all WALfiles for the server

e.      Notify the master that they areready to snapshot

3.     Wait for all servers to agreethat they are read

4.     Tell all servers to take asnapshot (they all agree on the current state)

a.      Release the write lock,allowing the table to start taking writes again.

 

In the general case, we are not likely tosee long latency in snapshots because of the rate at which we can process requestson a server and because (as discussed above) taking a snapshot is really alightweight operation. Note that while this is going on, we can continuereading from the table normally, we just have to buffer the writes until thetable becomes available again.

 

Note that a “single point in time” cannot actually beachieved just using clocks synchronized by NTP. We would require perfectly synchronized clocksto ensure that at a single point, we can place a snapshot marker across allservers. NTP is going to be off by a limited amount (generally less that 1second), but it not close enough for perfect synchronization. As such, we needto use the above locking mechanism (limiting availability) to get a consistentsnapshot across multiple servers. See Future Directions for higher availability alternatives for taking snapshots.

 

Failuresituations

 

In the situation where all the servers areup and evenly loaded, it’s likely that snapshots will proceed fairly quickly,with minimal locking. However, server failures or GC pauses can potentiallycause noticeable cluster performance. Let’s then talk about how a failure cancause performance issues and how there we can safeguard against breaking SLAs.

 

Suppose that, while taking a snapshot, oneof the servers crashes while preparing a snapshot. All the other serversinvolved in the snapshot are going to wait for that server to join thesnapshot. If wait is unbounded, the table will be unavailable for writes untilthe regions are reassigned to other regionservers and those servers can handlesnapshotting those regions. For anyone with any sizeable write-rate, this iscompletely unacceptable wait time.

 

To avoid crippling a table in an errorsituation, we enforce a time limit on each server that it is willing to waitfor a snapshot to complete. If it cannot complete the snapshot in the allottedtime, the snapshost manager on the regionserver will:

 

1.     Fail the snapshot locally

2.     Allow writes to proceed on all itslocal region hosting the snapshotted table

3.     Propagate that failure to allthe other servers

 

If a server receives a snapshot failure, itwill propagate that failure down to its own snapshot manager and quickly fail thesnapshot and start accepting writes again (if they haven’t failed already).

 

This gives us a hard limit on the amount oftime a table can possibly block writes. Further testing on the per-cluster isrequired to determine the latency effects of taking a snaphot and if it iswithin acceptable SLAs (see future directions a zero-latency snapshot option,if results are not within requirements). Depending on the write volumes, itcould still cause a noticeable impact, though there should be no appreciableimpact to reads.

 

There are two other concerns that onlyoccur for exceptionally large clusters (+1000 node). In smaller clusters, wecan count on more hardware stability and generally more pronounced off-peaktimes.

 

First, there can be significantnetwork/propagation latency for servers to receive updates at to the snapshotprogress. This can lead to timeout failures just due to the fact there are alot of things going on with the network. This concern is mitigated by twothings: we use ZooKeeper which scales updates well beyond the current scale ofthe largest HBase cluster (but we can easily switch to an RPCupdate/synchronization mechanism if necessary) and running snapshots atlow-write periods (i.e. nightly), to minimize the latency when attempting tofind a ‘stable’ point.

 

With increasing cluster sizes running oncommodity hardware, there is increasing probability for hardware failures - onlarger clusters, it is a much smaller issue to just loose one box than on asmaller cluster. In a well-designed and balanced table, there is a highprobability that the table will have a region on a server that fails.

 

Consider attempting to take a snapshot anda regionserver fails. Then you need to wait for the snapshot timeout before itfails. Then you attempt to take a snapshot again after the table rebalances andagain have a probability that another region server will fail. In a largercluster, this probability will be larger, leading to decreasing likelihood thata snapshot will succeed.

 

In this case, we rely on the snapshottimeout to minimize effect of the failure on the rest of the cluster as wellrunning snapshots at low traffic times to minimize the potentialwindow/probability of failure. However, cascading failures are not very likelyand are likely indicative of larger problems in the cluster that need to bemitigated before worrying about the nightly snapshot.

 

ImplementationDetails

 

Now that you believe that snapshotsactually can be done with minimal latency and effect on the cluster, lets getinto the details of how they are implemented.

 

The client initiates snapshots via theHBaseAdmin on a given table. Each snapshot is given a name that is then used tolabel the output directory – this name must be unique among all other namessnapshots taken or in progress across the cluster.

 

Currently we only support taking one snapshot at a time on the cluster. Thisis a simplification on the master-side implementation, but is actually notlimited on the regionserver. Only slightly more work needs to be done to enablemultiple snapshots at the same time. The other consideration here is if multiple snapshots should be running at thesame time – generally the answer is going to be “no”.

 

To capture all the HFiles, we enable‘backup mode’ for the table we are snapshotting and must remain enabled for atleast the time to compact all the files in the taken snapshot into the backupdirectory. Tn practice, this is going to be quite a while – all the HFiles inthe snapshot need to be compacted and there is really no good (efficient) wayto do the automated checking. A better solution is to distcp the snapshot,where you actually copy the referenced files and then stop the hfile backup forthe snapshotted table since you ensure that you got all the files in thesnapshot. In practice, if you going to be taking regular snapshots it willprobably easiest to just keep backup mode on and then periodically cleanup theHFiles in the archive/backup directory.

 

The HBaseAdmin then requests a snapshot forthe table on the master. At this point, all the work for a snapshot preceedsremotely and the client only waits for confirmation that the snapshotcompleted. Once receiving the “snapshot completed” notification, the snapshothas been completely persisted to the filesystem and can be accessed via anexternal source (its considered immutable and durable, within the bounds ofHDFS).

 

On the HMaster, when it receives a snapshotrequest, it passes off the request to the SnapshotManager. At this point theHMaster is done until it needs to return success or failure to the client.

 

Snapshot running has two distinct phases:(1) prepare, (2) commit (basic two-phase commit). The prepare phase is managedvia a barrier node in zookeeper. The appearance of the barrier means that allthe involved servers are going to prepare their snapshot and when ready, jointhe barrier. Once all the servers involved in the snaphot (hosting the table),the SnapshotSentinel tells the servers to commit via creating a complete node(essentially releasing the barrier). 

 

The SnapshotManager first figures out whichservers should be involved in the snapshot, so later it can check to see ifeveryone interested is ready to start (this is stored in the SnapshotSentinelas the expected set of servers). The SnapshotManager then creates the ‘snapshotznode’ for the given snapshot to propagate the notification to all theregionservers via the MasterZKSnapshotController. This node contains theserialized version of the SnapshotDescriptor.

At this point, the layout of zookeeper isas follows:

 

/hbase/snapshot/start/

                  [snapshot-name] - snapshot desc/

         /end/

              /abort/

 

Wherethe name of the snapshot is the name of the znode and the data is theserialized SnapshotDescriptor. The class layout for snapshot related classes onthe master is shown in Figure 1.

 

On each regionserver we have a RegionServerSnapshotHandler thathas a RegionSnapshotZKController. This controller is listening for updates to /hbase/snapshot/start/andwhen an update occurs for the node, we consider it the start of the snapshot.

 

Each RS then reads the data from that node,which contains the table to snapshot.

 

An optimization can be made here such that we storethe snapshot descriptor under a znode with the table name, avoiding thestampede effect when a new snapshot is made, but this has postponed until itproves necessary.

 

The RegionSnapshotZKController then updatesthe RegionServerSnapshotHandler (through the SnapshotListener interface) thatit received the snapshot start indicator (SnapshotListener::startSnapshot()::boolean)and waits to see if it should update zookeeper that it is ready to snapshot(join the start barrier). If the server does not have any regions for thetable, then we don’t need to update zookeeper that we are ready to snapshot.

 

If we choose to pass the expected servers as part ofthe snapshot description, we could have all the servers listen all otherservers to finish preparing their snapshot – joining the start barrier – andthen proceed on their own, without waiting for master coordination. This isslightly more complex and has been tabled for the moment, but the zookeepercode is well encapsulated, so this change should not be too difficult.

 

When receiving the notification to startthe snapshot, the RegionServerSnapshotHandler then creates a SnapshotRequestHandlerto handle the specific request. Included in this handler is the globalFailureindicator as well as the regions involved in the snapshot.

The SnapshotRequestHandler handles the meatof the request and has three main tasks:

1.     Kick off a RegionSnapshotOperationfor each region in the snapshot

a.      This callsHRegion::startSnapshot() internally and also monitors failure. We’ll get towhat this does in a moment.

2.     Start a WALSnapshotOperation,which internally will:

a.      Wait for the all associated regionsto become stable (no more pending writes). This comes from theRegionSnapshotOperation.

b.     Write a ‘meta’ entry into theWAL for this snapshot

c.      Add a reference for all WALfiles for the server currently present in the server’s .logs folder(.logs/[server name]). It is possible that not all WAL files are necessary, butthis is cheaper than scanning the files to see which we need

3.     Copy the .tableinfo into the.snapshot/[snapshot-name] directory

a.      This is a full copy of the.tableinfo, rather than just a reference creation, since the table info doesn’tinherently follow the copy-on-write semantics. However, this is a small fileand shouldn’t be onerous to copy.

 

Figure 2 gives the general layout of themost of the regionserver specific classes discussed so far.

 

All of these tasks are run asynchronouslyin a thread pool created just for snapshots, allowing us to parallelize thesnapshot preparation. To monitor the status of the preparation, we get back aSnapshotStatusMonitor which internally has a SnapshotStatus (interface) foreach of the operations that the SnapshotHandler started. Once all the statusescomplete, we return true to the RegionSnapshotZKController, indicating that weare ready to start the snapshot. At this point, the zk structure looks likethis:

 

 

/hbase/snapshot/start/

                  [snapshot-name] - snapshot desc/

                       /[regionserversdone preparing snapshot]    

                 ...  

               /end/

        /abort/

 

 

Remember how the WALSnapshotOperation waited onall the regions to become stable? Internally, the SnapshotRequestHandleractually waits on RegionSnapshotOperationStatus::regionsStable() to returntrue. At this point, the MVCC has been rolled forward on all the regions to asingle point and we are blocking all writes to that region. Now we know the WALhas become stable, so we can put a snapshot edit in the WAL and create areference to the WAL files. Note that we do not roll the WAL file, but just adda reference to the file. If we want to recover the snapshot and the WAL file isstill ‘0 bytes’ in size, then we will need to roll the file on the serverthrough the usual mechanisms. However, this rolling was considered too costlyin terms of optimizing the latency of preparing a snapshot.

 

The optimizationhere is that we can create references for all the current WAL files, then checkonce we reach stability that the WAL didn’t get rolled while we are waiting;this wasn’t implemented for simplicity at the moment, but is entirely feasible.

 

The other key part of HRegion::startSnapshot() is creatinga reference file for each of the HFiles that the region currently serves. Eachregion manages its own HFiles, so this was a natural factor of parallelization.We  log the progress periodically for thenumber of HFiles collected, but we wait until all files have references addedbefore considering the region ‘done’.

 

 In the current implementation we do referencecounting for the included HFiles in .META. This ensures that we keep the HFilesaround when doing cleanup. However, this code is likely going to be removed infavor of doing cleanup via checking the reference files in the snapshotdirectory, saving time to update .META. and allowing a snapshot preparation tocomplete faster. A similar method is currently used for the SnapshotLogCleaner,which will be discussed later.

 

Keep in mind,while all this reference file creation is happening, we still have thewrite-lock on each of the involved regions. In fact, we cannot release thewrite lock until the snapshot is ready on allregions on all servers for the table. If we released the write lock, itwould lead to an inconsistent point-in-time across the servers. Figure 3 givesa more detailed view at how all the different statuses and operations cometogether to run a snapshot.

 

 

 

Once the snapshot preparation has completed on allthe regionservers, the master (more specifically the SnapshotSentinel) willnotice that all the servers have ‘joined’ the start barrier for the snapshot.At this point, we release the snapshot barrier by creating the end barrier nodefor the snapshot. Specifically, the zookeeper structure now looks like:

 

/hbase/snapshot/start/

                  [snapshot-name] - snapshot desc/

                        /[regionservers donepreparing snapshot]    

                 ...  

               /end/

 [snapshot-name]/

        /abort/

 

All the RegionZKSnapshotControllers are watchingfor children of the end barrier and then callRegionServerSnapshotHandler::finishSnapshot(snapshotName). If the server ran asnapshot with this name (note the requirement that all snapshot names areunique!), then it finishes the snapshot.

 

Internally, this just released the write lock andcloses the region operation on each of each of the regions (all of this ishandled by the RegionSnapshotOperation::finishSnapshot()). If we successfullycomplete the snapshot, then we add the server name to the end barrier andforget about the snapshot (similarly to how we notify the master that we areprepared to take the snapshot). ZooKeeper now looks like this:

 

/hbase/snapshot/start/

                  [snapshot-name] - snapshot desc/

                        /[regionservers donepreparing snapshot]

                 ...  

               /end/

 [snapshot-name]/

/[regionservers that have committed]    

              ...  

        /abort/

 

 

Once all the regions have successfully completedthe snapshot, we consider the snapshot ‘done’ and the master can move it fromthe .snapshot/.tmp/[snapshot-name] directory to the .snapshot/[snapshot-name]directory and returns successfully to the client.

 

SecondaryUtilities

 

There is slightly more to making sure thatsnapshots are consistent. As of right now, the only independent utility class(outside of the main ‘take a snapshot’ code) is the SnapshotLogCleaner.

 

HLogs are initially stored in the .logs/[server]directory under the server for which they logs edits. Eventually, as the HLogsare periodically rolled and the Memstore is flushed to HFiles the older HLogsare no longer necessary when recovering a region as these edits already existin HFiles. Periodically, these old HFiles are archived to the .oldlogsdirectory, where they are periodically cleaned up (the default is after acertain amount of time the are deleted).

 

The LogCleaner periodically goes through the logfiles in the .oldlogs directory and asks each LogCleanerDelegate if it shoulddelete the specified log file. The SnapshotLogCleaner keeps a cache of thecurrent log files referenced in the snapshots on disk. If the log file is notfound in the local cache, the SnapshotLogCleaner refreshes its cache by justrereading the file system state for the current snapshots and checking for thefile name again.

 

A possible error situtation may occur if a WALfile that was referenced in the snapshot is moved to the .oldlogs directory andthen cleaned up before the snapshot is completed. At this point theSnapshotLogCleaner would think that it is not referenced by any snapshot andconsider the log valid to delete. First, this is highly unlikely becausesnapshot timeouts should be set fairly low (under 30 s) and log retention timefairly high (at least 6 hrs). However, even if these values are incorrectlyset, this log is actually not storing any state that we care about and cansafely ignore when rebuilding the table state from the snapshot (remember thelog archiver only moves logs that hold edits that have all been written out toHFiles already). In the end, this is not an error, only a corner case forreestablishing table state from a snapshot.

 

A likely offshoot of HBASE-5547 is a mechanismsimilar to the logcleaner – a class that monitors the HFiles in the backupdirectory and periodically cleans them up. As such, it is trivial to implementa complementary cleanup tool for making sure that snapshot referenced HFilesare not deleted.

 

And that is basically it!

 

Well, not really, because in distributed systemswe always worry about failures…

 

 

FailureSituations

 

Since snapshots are scaling across multipleservers and synchronize to a point in time are going to inherently be fragile.Further, as an initial implementation we allow this fragility as a trade-offfor data consistency and implementation simplicity.

 

When each RegionServerSnapshotHandler gets astartSnapshot() notification it start a timer. If the timer exceeds theconfigurable snapshot timeout, the snapshot is considered ‘failed’; thismitigates against excessive impact on the cluster when taking a snapshot. Ifthe snapshot doesn’t complete on the current server (both the prepare and completephases), the snapshot is considered failed and the server locally releases itswrite lock. To ensure the snapshot fails globally, at the same time as failinglocally, this error is propagated back to the other servers via writing thisserver’s name to zookeeper under the abort node.

 

For example, if server ‘region-server-A:1234’timed out waiting for the complete notification (so it completed the preparephase), zookeeper would look like:

/hbase/snapshot/start/

                  [snapshot-name] - snapshot desc/

                        /region-server-A:1234

                 ...  

               /end/

    /abort/

  [snapshot-name]/

     /region-server-A:1234

 

Since all the other regionservers, and theSnapshotSentinel on the master, are watching the abort znode for children, theywill quickly see that abort notification and then propagate that failure backto their own snapshot attempts.

 

Internally,we monitor this via the SnapshotFailureMonitor that keeps track of theglobalFailureState, snapshotFailureState and the elapsed time. If any of theseindicates breaks the monitor will throw an exception whenSnapshotFailureMontior::checkFailure() is called. This will cause the snapshotto finish (releasing the region locks). If the error was cause internally, thenit will propagate up the zookeeper and write the above discussed abort znode.If it came externally (another server failed the snapshot), then the snapshotis just aborted locally, again releasing the write locks. This up and down propagationand ensuring lock release has lead to some complexity around the progress andmonitoring infrastructure – it will be refined later.

 

Note that this abort notification does not need tobe caused by a timeout; it can be caused by any internal failure in creatingthe snapshot – e.g. failure to write reference files or, in the currentimplementation, increasing the reference count for a file. Anything that causesa snapshot to not be complete is considered a failure for the snapshot,avoiding an incorrect view of the table state in the snapshot.

 

In addition to each snapshot having a failureindicator, we also have a global failure indicator that is inherently monitoredfor each snapshot running on each server. Global here is a slight misnomer – it’sglobal for the ‘world’ in which the snapshot operation is running: the regionserver. If the server dies/aborts, then we must inherently fail all thesnapshots. This will cause some latency as we attempt to notify zookeeper toabort each of the snapshots, but if server cannot manage that before it dies(for instance a ‘kill -9’ on the process), then the snapshot will naturallytimeout on the other servers.

 

As discussed in the theory section, this timeoutmechanism also comes into play if we have excessive network latency or hardwarefailure. The cluster write throughput is only going to be impacted for amaximum of configurable snapshot timeout interval (less the JVM timer checkingissues), even if we have multiple failures across the cluster.

 

Conclusion

 

At this point we have walked through thegeneral mechanism for doing point-in-time snapshots in a distributed system. Inshort, a snapshot is a two-phase commit enabled across servers via a barriernode in zookeeper, where you:

 

1.     freeze the world for a tablefor a very short time (within SLA boundaries),

2.     create a bunch of soft-links forthe files you care about

3.     thaw the world

4.     Any time files are deleted, youarchive them instead of deleting them.

 

Only in full success situations do weinform the client that a snapshot was successful. We bound the error scenariosby using a configurable timeout that is monitored on each server. During thesnapshot, any failures by involved servers – HRegionServers hosting regions ofthe snapshotted table or the HMaster – will cause a snapshot failure from theclient’s perspective.

 

Futuredirections

 

The proposed snapshot implementation canalso be used to do ‘soft’ snapshots  -using a similar time-stamp based approach discussed in the introduction, butinstead snapshotting ‘x seconds in the future’. This leads leading to minimaldata copy and latency, as well as minimal blocking on each server, commensuratewith a large batch of puts on regions. This enables offline data processing andbackup with minimal impact on the running cluster.

 

Using the above approach for soft snapshotscombined with a ‘timestamp oracle’ we can achieve zero-latency, point-in-time snapshots. Instead of using the commonpractice of just using timestamps assigned by the server for puts, you can usetimestamps that are strictly increasing assigned from a “timestamp oracle” (seePercolator[i]).At Google, they were able to create a timestamp oracle that generated 2 milliontimestamps a second, enough for most applications. A similar timestamp oraclecan be created using HBase increment functionality in special counter table,with a timestamp row per-table.

 

[i] “Large-scale Incremental Processing Using Distributed Transactionsand Notifications”, Daniel Peng, Frank Dabek, Proceedings of the 9th USENIX Symposium on Operating Systems Design andImplementation, 2010. http://research.google.com/pubs/pub36726.html

https://issues.apache.org/jira/browse/HBASE-6055