1 首先watcher机制的watcher是什么？模型是什么？
watcher就像现实生活中的监听器，那server就相当于会议室等，各个客户端（人），都可以访问的地方。我不需要知道其他人干了什么事情，我只监听，我感兴趣的内容（DataNode）。 DataNode内容变更。dataNode 的新增，或者子DataNode改变等事件。都会引起我的兴趣，然后监听器就会，给我推送内容。
watcher的模型，包括最重要的回调,process回调方法。 还有Event事件，还有event发生时server的状态KeeperState。
image.png
2 watcher机制，有什么用？
1 结合zookeeper数据模型中的，临时节点Znode,可以实现，服务注册与发现，集群配置动态更新等功能。
2 结合zookeeper数据模型中的，临时顺序节点Znode特性，可以实现分布式锁，队列。
3 从设计模式的角度看，watcher的注册，触发，就是分布式消息的发布订阅模式，也就是观察者模式。
3 watcher的运行机制
模拟这样一个场景，zk.getChildren(path, childrenWatcher) . childrenWatcher是重写了process(WatchedEvent event)方法的watcher对象。
3.1 watcher的client端，怎么完成注册
ZooKeeper#getChildren(String, Watcher)
public List getChildren(final String path, Watcher watcher)
{
"省略n行代码================"
WatchRegistration wcb = null;
if (watcher != null) {
wcb = new ChildWatchRegistration(watcher, clientPath);
}
RequestHeader h = new RequestHeader();
h.setType(ZooDefs.OpCode.getChildren);
GetChildrenRequest request = new GetChildrenRequest();
request.setPath(serverPath);
"实际网络上传输，只传path，还有watch这个boolean变量"
request.setWatch(watcher != null);
GetChildrenResponse response = new GetChildrenResponse();
ReplyHeader r = cnxn.submitRequest(h, request, response, wcb);
if (r.getErr() != 0) {
throw KeeperException.create(KeeperException.Code.get(r.getErr()),
clientPath);
}
return response.getChildren();
}
构建完了请求对象之后，放入outgoingQueue中。等待SendThread的run方法发送
Packet queuePacket(RequestHeader h, ReplyHeader r, Record request,
Record response, AsyncCallback cb, String clientPath,
String serverPath, Object ctx, WatchRegistration watchRegistration)
{
Packet packet = null;
synchronized (outgoingQueue) {
packet = new Packet(h, r, request, response, watchRegistration);
packet.cb = cb;
packet.ctx = ctx;
packet.clientPath = clientPath;
packet.serverPath = serverPath;
if (!state.isAlive() || closing) {
conLossPacket(packet);
} else {
// If the client is asking to close the session then
// mark as closing
if (h.getType() == OpCode.closeSession) {
closing = true;
}
outgoingQueue.add(packet);
}
}
sendThread.getClientCnxnSocket().wakeupCnxn();
return packet;
}
ClientCnxn.SendThread#run
while (state.isAlive()) {
try {
"省略n行代码======================"
clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
"省略n行代码======================"
}
.ClientCnxnSocketNIO#doIO
if (sockKey.isWritable()) {
synchronized(outgoingQueue) {
"从outpingQueue中拿到需要发送的packet "
Packet p = findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress());
if (p != null) {
updateLastSend();
// If we already started writing p, p.bb will already exist
if (p.bb == null) {
if ((p.requestHeader != null) &&
(p.requestHeader.getType() != OpCode.ping) &&
(p.requestHeader.getType() != OpCode.auth)) {
p.requestHeader.setXid(cnxn.getXid());
}
"这个方法里只序列化了request字段和requestHeader字段到Packet类中的bb字段ByteBuffer"
p.createBB();
}
sock.write(p.bb);
if (!p.bb.hasRemaining()) {
sentCount++;
outgoingQueue.removeFirstOccurrence(p);
if (p.requestHeader != null
&& p.requestHeader.getType() != OpCode.ping
&& p.requestHeader.getType() != OpCode.auth) {
synchronized (pendingQueue) {
"刚才说了requestHeader序列化了，而且有东西，所以pendingQueue加入packet"
pendingQueue.add(p);
}
}
}
}
if (outgoingQueue.isEmpty()) {
disableWrite();
} else if (!initialized && p != null && !p.bb.hasRemaining()) {
disableWrite();
} else {
enableWrite();
}
}
}
到这里实际上，还没有完成客户端的watcher注册。
实际上是，服务端先完成的注册。但是服务端注册，我们后面看，这里先假定他正常返回了getChildrenRequest请求的数据。
这个时候，还是ClientCnxnSocketNIO#doIO的方法。
if (sockKey.isReadable()) {
int rc = sock.read(incomingBuffer);
if (!incomingBuffer.hasRemaining()) {
incomingBuffer.flip();
if (incomingBuffer == lenBuffer) {
recvCount++;
readLength();
}
} else {
"读取服务端返回的数据"
sendThread.readResponse(incomingBuffer);
lenBuffer.clear();
incomingBuffer = lenBuffer;
updateLastHeard();
}
}
}
org.apache.zookeeper.ClientCnxn.SendThread#readResponse
Packet packet;
synchronized (pendingQueue) {
if (pendingQueue.size() == 0) {
throw new IOException("Nothing in the queue, but got "
+ replyHdr.getXid());
}
"拿到之前放在pendingqueue中的packet"
packet = pendingQueue.remove();
}
try {
packet.replyHeader.setXid(replyHdr.getXid());
packet.replyHeader.setErr(replyHdr.getErr());
packet.replyHeader.setZxid(replyHdr.getZxid());
if (replyHdr.getZxid() > 0) {
lastZxid = replyHdr.getZxid();
}
if (packet.response != null && replyHdr.getErr() == 0) {
packet.response.deserialize(bbia, "response");
}
} finally {
"这个方法最重要，才是完成watcher注册的地方"
finishPacket(packet);
}
这个方法最重要，才是完成watcher注册的地方
ClientCnxn#finishPacket
private void finishPacket(Packet p) {
if (p.watchRegistration != null) {
"这里的p.watchRegistration实际上是ChildWatchRegistration"
p.watchRegistration.register(p.replyHeader.getErr());
}
if (p.cb == null) {
synchronized (p) {
p.finished = true;
p.notifyAll();
}
} else {
p.finished = true;
eventThread.queuePacket(p);
}
}
public void register(int rc) {
if (shouldAddWatch(rc)) {
"因为ChildWatchRegistration重写了getWatches方法，返回的是watchManager.childWatches;"
"watchManager实际上就是ZKWatchManager"
Map> watches = getWatches(rc);
synchronized(watches) {
Set watchers = watches.get(clientPath);
if (watchers == null) {
watchers = new HashSet();
watches.put(clientPath, watchers);
}
watchers.add(watcher);
}
}
}
到此，完成了client端的注册。
附上流程图，还有类图：
zookeeper的watch通知机制.png
wacher注册触发整体流程.png
3.2 server端是怎么注册的呢？
在前面的【zookeeper集群的分布式事务请求处理过程】中我们讲过，责任链的确定是根据其角色来确定的。 当前我们是单机版的。所以我们的getChildren请求，最终会到服务端的责任链 PrepRequestProcessor===>SyncRequestProcessor===>FInalRequestProcessor
当我们发送包含watcher的getChildren的请求对象时，PrepRequestProcessor只是checkSession是否过期expired 或者sessIon是否迁移moved
SyncRequestProcessor和持久化事物日志有关。如果不需要持久化，就会走到FInalRequestProcessor。这个类中
FinalRequestProcessor#processRequest
case OpCode.getChildren: {
lastOp = "GETC";
GetChildrenRequest getChildrenRequest = new GetChildrenRequest();
ByteBufferInputStream.byteBuffer2Record(request.request,
getChildrenRequest);
"从内存的dataTree中获取path所对应的节点"
DataNode n = zks.getZKDatabase().getNode(getChildrenRequest.getPath());
if (n == null) {
throw new KeeperException.NoNodeException();
}
PrepRequestProcessor.checkACL(zks, zks.getZKDatabase().aclForNode(n),
ZooDefs.Perms.READ,
request.authInfo);
"如果getChildrenRequest中的watch布尔变量是true的话，传入cnxn对象"
"因为cnxn对象实现了watcher接口方便后面的时候回调"
List children = zks.getZKDatabase().getChildren(
getChildrenRequest.getPath(), null, getChildrenRequest
.getWatch() ? cnxn : null);
rsp = new GetChildrenResponse(children);
break;
}
public List getChildren(String path, Stat stat, Watcher watcher)
throws KeeperException.NoNodeException {
DataNode n = nodes.get(path);
if (n == null) {
throw new KeeperException.NoNodeException();
}
synchronized (n) {
if (stat != null) {
n.copyStat(stat);
}
List children = new ArrayList(n.getChildren());
if (watcher != null) {
"把cnxn对象注册到WatchManager childWatches中"
childWatches.addWatch(path, watcher);
}
return children;
}
}
3.3 watcher在服务端的回调
当我们在path下面新建子节点的时候，我们就会触发，这个NodeChildrenChanged事件。
首先进入的是PrepRequestProcessor
.PrepRequestProcessor#pRequest
case OpCode.create:
CreateRequest createRequest = new CreateRequest();
"这里得到了创建节点对应的zxid，转化成事物请求，主要添加了ChangeRecord"
pRequest2Txn(request.type, zks.getNextZxid(), request, createRequest, true);
break;
接下来是SyncRequestProcessor。append事物日志txnLog到streamsToFlush链表中。
然后flush到磁盘。
在接下来是FInalRequestProcessor。
FinalRequestProcessor#processRequest
synchronized (zks.outstandingChanges) {
while (!zks.outstandingChanges.isEmpty()
&& zks.outstandingChanges.get(0).zxid <= request.zxid) {
ChangeRecord cr = zks.outstandingChanges.remove(0);
if (cr.zxid < request.zxid) {
LOG.warn("Zxid outstanding "
+ cr.zxid
+ " is less than current " + request.zxid);
}
if (zks.outstandingChangesForPath.get(cr.path) == cr) {
zks.outstandingChangesForPath.remove(cr.path);
}
}
if (request.hdr != null) {
TxnHeader hdr = request.hdr;
Record txn = request.txn;
"处理事物请求"
rc = zks.processTxn(hdr, txn);
}
// do not add non quorum packets to the queue.
if (Request.isQuorum(request.type)) {
zks.getZKDatabase().addCommittedProposal(request);
}
}
public String createNode(String path, byte data[], List acl,
long ephemeralOwner, int parentCVersion, long zxid, long time)
throws KeeperException.NoNodeException,
KeeperException.NodeExistsException {
"省略N行代码============"
dataWatches.triggerWatch(path, Event.EventType.NodeCreated);
"其中这个地方是我们之前注册的watcher,现在触发回调"
childWatches.triggerWatch(parentName.equals("") ? "/" : parentName,
Event.EventType.NodeChildrenChanged);
return path;
}
public Set triggerWatch(String path, EventType type, Set supress) {
WatchedEvent e = new WatchedEvent(type,
KeeperState.SyncConnected, path);
HashSet watchers;
synchronized (this) {
"这个地方解释了 ，watcher的一次性注册，需要继续监听的话，得重新注册watcher"
watchers = watchTable.remove(path);
if (watchers == null || watchers.isEmpty()) {
if (LOG.isTraceEnabled()) {
ZooTrace.logTraceMessage(LOG,
ZooTrace.EVENT_DELIVERY_TRACE_MASK,
"No watchers for " + path);
}
return null;
}
for (Watcher w : watchers) {
HashSet paths = watch2Paths.get(w);
if (paths != null) {
paths.remove(path);
}
}
}
for (Watcher w : watchers) {
if (supress != null && supress.contains(w)) {
continue;
}
"这里触发回调，记得之前我们放入的是ServerCnxn对象，所以我们跳到ServerCnxn的代码"
w.process(e);
}
return watchers;
}
.NIOServerCnxn#process
synchronized public void process(WatchedEvent event) {
"注意这里的zxid是-1,在客户端判断的时候会用到"
ReplyHeader h = new ReplyHeader(-1, -1L, 0);
if (LOG.isTraceEnabled()) {
ZooTrace.logTraceMessage(LOG, ZooTrace.EVENT_DELIVERY_TRACE_MASK,
"Deliver event " + event + " to 0x"
+ Long.toHexString(this.sessionId)
+ " through " + this);
}
// Convert WatchedEvent to a type that can be sent over the wire
WatcherEvent e = event.getWrapper();
"因为我们发送的都是watcher，所以watcherEvent才是可以在网络上传输的"
sendResponse(h, e, "notification");
}
3.4 接下来就是最后Client端的回调
.ClientCnxn.SendThread#readResponse
if (replyHdr.getXid() == -1) {
// -1 means notification
if (LOG.isDebugEnabled()) {
LOG.debug("Got notification sessionid:0x"
+ Long.toHexString(sessionId));
}
WatcherEvent event = new WatcherEvent();
event.deserialize(bbia, "response");
// convert from a server path to a client path
if (chrootPath != null) {
String serverPath = event.getPath();
if(serverPath.compareTo(chrootPath)==0)
event.setPath("/");
else if (serverPath.length() > chrootPath.length())
event.setPath(serverPath.substring(chrootPath.length()));
else {
LOG.warn("Got server path " + event.getPath()
+ " which is too short for chroot path "
+ chrootPath);
}
}
WatchedEvent we = new WatchedEvent(event);
if (LOG.isDebugEnabled()) {
LOG.debug("Got " + we + " for sessionid 0x"
+ Long.toHexString(sessionId));
}
"把接收到的Event放入waitingEvents队列中"
eventThread.queueEvent( we );
return;
}
public void queueEvent(WatchedEvent event) {
if (event.getType() == EventType.None
&& sessionState == event.getState()) {
return;
}
sessionState = event.getState();
// materialize the watchers based on the event
WatcherSetEventPair pair = new WatcherSetEventPair(
"根据event，从ClientWatchManager，也就是ZKWatchManager，"
"中找到watcher 集合，我们前面触发的是NodeChildrenChanged，所以从Map> "
"childWatches获得Set"
watcher.materialize(event.getState(), event.getType(),
event.getPath()),
event);
// queue the pair (watch set & event) for later processing
waitingEvents.add(pair);
}
public Set materialize(Watcher.Event.KeeperState state,
Watcher.Event.EventType type,
String clientPath)
{
Set result = new HashSet();
switch (type) {
"省略n行代码===================="
case NodeChildrenChanged:
synchronized (childWatches) {
"从childWatches中删除，删除的watcher都加入到result返回结果中"
addTo(childWatches.remove(clientPath), result);
}
break;
default:
String msg = "Unhandled watch event type " + type
+ " with state " + state + " on path " + clientPath;
LOG.error(msg);
throw new RuntimeException(msg);
}
return result;
}
.ClientCnxn.EventThread#run
public void run() {
try {
isRunning = true;
while (true) {
Object event = waitingEvents.take();
if (event == eventOfDeath) {
wasKilled = true;
} else {
"客户端的回调"
processEvent(event);
}
if (wasKilled)
synchronized (waitingEvents) {
if (waitingEvents.isEmpty()) {
isRunning = false;
break;
}
}
}
} catch (InterruptedException e) {
LOG.error("Event thread exiting due to interruption", e);
}
LOG.info("EventThread shut down for session: 0x{}",
Long.toHexString(getSessionId()));
}
private void processEvent(Object event) {
try {
if (event instanceof WatcherSetEventPair) {
// each watcher will process the event
WatcherSetEventPair pair = (WatcherSetEventPair) event;
for (Watcher watcher : pair.watchers) {
try {
"客户端的回调"
watcher.process(pair.event);
} catch (Throwable t) {
LOG.error("Error while calling watcher ", t);
}
}
}