服务端的创建会话相关类的UML类图
image.png
会话创建过程的流程图：
image.png
代码跟踪：
首先上一张客户端的类关系图：
image.png
客户端代码：
首先ZookeeperMain客户端类的启动类：
public ZooKeeperMain(String args[]) throws IOException, InterruptedException {
cl.parseOptions(args);
System.out.println("Connecting to " + cl.getOption("server"));
connectToZK(cl.getOption("server"));
//zk = new ZooKeeper(cl.getOption("server"),
// Integer.parseInt(cl.getOption("timeout")), new MyWatcher());
}
客户端启动的时候，就connectToZK，发起连接到zkServer中。
public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,
boolean canBeReadOnly)
throws IOException
{
LOG.info("Initiating client connection, connectString=" + connectString
+ " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);
watchManager.defaultWatcher = watcher;
ConnectStringParser connectStringParser = new ConnectStringParser(
connectString);
HostProvider hostProvider = new StaticHostProvider(
connectStringParser.getServerAddresses());
cnxn = new ClientCnxn(connectStringParser.getChrootPath(),
hostProvider, sessionTimeout, this, watchManager,
getClientCnxnSocket(), canBeReadOnly);
cnxn.start();
}
public void start() {
sendThread.start();
eventThread.start();
}
StaticHostProvider 这个类是zk的服务器地址集合的容器。有算法去选择某一个server去连接。
getClientCnxnSocket()方法，实例化了一个ClientCnxnSocketNIO对象，里面包含final字段的selector ，多路复用器。
然后实例化ClientCnxn 对象，这里面有实例化了两个线程，SendThread和EventThread。最后启动了两个线程。
紧接着，我们进入了SendThread的run方法里的
startConnect(serverAddress);
startConnect方法里的clientCnxnSocket.connect(addr);
void connect(InetSocketAddress addr) throws IOException {
SocketChannel sock = createSock();
try {
registerAndConnect(sock, addr);
} catch (IOException e) {
LOG.error("Unable to open socket to " + addr);
sock.close();
throw e;
}
initialized = false;
/*
* Reset incomingBuffer
*/
lenBuffer.clear();
incomingBuffer = lenBuffer;
}
SocketChannel createSock() throws IOException {
SocketChannel sock;
sock = SocketChannel.open();
sock.configureBlocking(false); //非阻塞
sock.socket().setSoLinger(false, -1); //当收到关闭请求的时候，是否让网卡等待一段时间，再关闭连接。 false的话，要等所有数据发送完，才关闭连接，也就是说false时间是最长的
sock.socket().setTcpNoDelay(true); //开启tcpnodelay，就是不让tcp的延迟机制发挥作用，也就是禁用了nagle算法。nagle算法是一种拥塞避免算法。
return sock;
}
注意这里的connect方法是异步的，直接跳过，需要后面的finishConnect去完成这次连接。
void registerAndConnect(SocketChannel sock, InetSocketAddress addr)
throws IOException {
sockKey = sock.register(selector, SelectionKey.OP_CONNECT);
boolean immediateConnect = sock.connect(addr); //前面设置了非阻塞，这个地方，不一定能马上连接上，而且异步的，直接跳过。
if (immediateConnect) {
sendThread.primeConnection();
}
}
然后方法栈pop出来，我们又进入到startConnect方法之后的代码
clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
void doTransport(int waitTimeOut, List pendingQueue, LinkedList outgoingQueue,
ClientCnxn cnxn)
throws IOException, InterruptedException {
selector.select(waitTimeOut);
Set selected;
synchronized (this) {
selected = selector.selectedKeys();
}
// Everything below and until we get back to the select is
// non blocking, so time is effectively a constant. That is
// Why we just have to do this once, here
updateNow();
for (SelectionKey k : selected) {
SocketChannel sc = ((SocketChannel) k.channel());
/ / 因为OP_CONNECT是8 ，也就是00001000，那么和readyOps按位与之后，就看Connect位是否为0 ，如果为0，与之后就是0，如果不为0，那就是CONNECT事件已经ready了。
if ((k.readyOps() & SelectionKey.OP_CONNECT) != 0) {
//这里就是前面所说的finishConnect方法，完成连接，如果这里没连上，那就会进入后面的 selected.clear();，还有EventThread run 方法里的
//cleanup();
// clientCnxnSocket.close(); 进行资源释放
if (sc.finishConnect()) {
updateLastSendAndHeard();
sendThread.primeConnection();
}
} else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {
doIO(pendingQueue, outgoingQueue, cnxn);
}
}
if (sendThread.getZkState().isConnected()) {
synchronized(outgoingQueue) {
if (findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress()) != null) {
enableWrite();
}
}
}
selected.clear();
}
从多路复用器中拿到selectedKeys(key是代表selector和channel的一种关系)，也就是注册的已经ready的channel 。
如果这里连接上了，也就是finishCOnnect成功了的话，那就进入primeCOnnection方法
void primeConnection() throws IOException {
....................省略若干代码
ConnectRequest conReq = new ConnectRequest(0, lastZxid,
sessionTimeout, sessId, sessionPasswd);
synchronized (outgoingQueue) {
....................省略若干代码（先不关心watch机制，还有验证机制）
把ConnectRequest 对象放入 outgoingQueue 这个发送队列中
outgoingQueue.addFirst(new Packet(null, null, conReq,
null, null, readOnly));
}
clientCnxnSocket.enableReadWriteOnly();
}
//设置对通道的读写ready感兴趣
synchronized void enableReadWriteOnly() {
sockKey.interestOps(SelectionKey.OP_READ | SelectionKey.OP_WRITE);
}
然后再回到上面的doTransport方法。selected.clear(); 以便下次select的时候，能拿到selectedKeys ,这里一定要clear一下，或者逐个remove掉，不然下次就selectedKeys==0.
为证明这一点。我还写了个程序去验证这个东西。
server端代码：
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.nio.charset.Charset;
public class Server {
public void start(){
try{
ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
serverSocketChannel.bind(new InetSocketAddress(8080));
serverSocketChannel.configureBlocking(false);
Selector seletor = Selector.open();
serverSocketChannel.register(seletor, SelectionKey.OP_ACCEPT);
while(true){
int selected =seletor.select();
SocketChannel socketChannel =null;
if(selected>0){
for(SelectionKey key:seletor.selectedKeys()){
if(key.isAcceptable()){
socketChannel = ((ServerSocketChannel) key.channel()).accept();
ByteBuffer readBuffer = ByteBuffer.allocate(40);
int totalSize = 0;
int size = socketChannel.read(readBuffer);
while(size>0){
totalSize= totalSize+size;
readBuffer.flip();
byte[] bytes = new byte[readBuffer.remaining()];
readBuffer.get(bytes);
System.out.println("accept a connection and read message from client ====");
System.out.println(new String(bytes, Charset.forName("UTF-8")));
size =socketChannel.read(readBuffer);
}
System.out.println("server get "+totalSize+"字节");
}
}
}
ByteBuffer writeBuffer = ByteBuffer.wrap(("server has got message from client" ).getBytes());
socketChannel.write(writeBuffer);
}
}catch(Exception e){
e.printStackTrace();
}
}
public static void main(String[] args) {
new Server().start();
}
}
Client端代码
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.nio.charset.Charset;
public class Client {
static int count =0;
private SelectionKey sockKey;
public void connect(String host,int port) throws Exception{
SocketChannel socketChannel =SocketChannel.open();
Selector selector = Selector.open();
boolean connected=socketChannel.connect(new InetSocketAddress(host, port));
socketChannel.configureBlocking(false);
socketChannel.register(selector, SelectionKey.OP_READ);
if(connected){
System.out.println("connected==========");
}
String request = new String("hello serverSocketChannel");
byte[] bytes = request.getBytes("UTF-8");
ByteBuffer buffer = ByteBuffer.wrap(bytes);
socketChannel.write(buffer);
buffer.clear();
while(true){
int selected = selector.select();
if(selected>0){
if(count<10){
System.out.println("next loop");
}
for(SelectionKey key:selector.selectedKeys()){
if(key.isReadable()){
System.out.println("read ready");
key.interestOps(key.interestOps() | SelectionKey.OP_WRITE);
SocketChannel socketChannel2 = (SocketChannel) key.channel();
ByteBuffer readBuffer = ByteBuffer.allocate(40);
int size = socketChannel2.read(readBuffer);
System.out.println("size========"+size);
while(size>0){
readBuffer.flip();
byte[] bytes2 = new byte[readBuffer.remaining()];
readBuffer.get(bytes2);
String mssString = new String(bytes2, Charset.forName("UTF-8"));
System.out.println("receive message from server"+mssString);
size = socketChannel2.read(readBuffer);
readBuffer.clear();
}
}else if((key.readyOps() & SelectionKey.OP_WRITE )!=0 ){
count++;
if(count<10){
System.out.println("write ready ============================");
}
}
}
selector.selectedKeys().clear();
}
}
}
public static void main(String[] args) throws Exception {
new Client().connect("127.0.0.1", 8080);
}
}
当client端代码没有selector.selectedKeys().clear(); 这一句时候，write ready只能进入一次。下一次int selected = selector.select(); 这一句代码执行的结果，就是0.
验证完这个之后，我们在回到doTransport方法，doTransport执行完，又会进入，run方法的下一次while循环。while (state.isAlive())
然后会再次进入doTransport方法。因为这个时候，已经设置了对write ready感兴趣。
因为java 的NIO触发方式是水平触发 。 只要内核缓冲区有东西，就会触发read ready事件； 只要内核缓冲区不是已经满了，就会触发 write ready事件。所以这次，会进入doIO 方法
void doIO(List pendingQueue, LinkedList outgoingQueue, ClientCnxn cnxn)
throws InterruptedException, IOException {
SocketChannel sock = (SocketChannel) sockKey.channel();
if (sock == null) {
throw new IOException("Socket is null!");
}
if (sockKey.isReadable()) {
int rc = sock.read(incomingBuffer);
if (rc < 0) {
throw new EndOfStreamException(
"Unable to read additional data from server sessionid 0x"
+ Long.toHexString(sessionId)
+ ", likely server has closed socket");
}
if (!incomingBuffer.hasRemaining()) {
incomingBuffer.flip();
if (incomingBuffer == lenBuffer) {
recvCount++;
readLength();
} else if (!initialized) {
readConnectResult();
enableRead();
if (findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress()) != null) {
// Since SASL authentication has completed (if client is configured to do so),
// outgoing packets waiting in the outgoingQueue can now be sent.
enableWrite();
}
lenBuffer.clear();
incomingBuffer = lenBuffer;
updateLastHeard();
initialized = true;
} else {
sendThread.readResponse(incomingBuffer);
lenBuffer.clear();
incomingBuffer = lenBuffer;
updateLastHeard();
}
}
}
if (sockKey.isWritable()) {
synchronized(outgoingQueue) {
Packet p = findSendablePacket(outgoingQueue,
cnxn.sendThread.clientTunneledAuthenticationInProgress());
if (p != null) {
updateLastSend();
// If we already started writing p, p.bb will already exist
if (p.bb == null) {
if ((p.requestHeader != null) &&
(p.requestHeader.getType() != OpCode.ping) &&
(p.requestHeader.getType() != OpCode.auth)) {
p.requestHeader.setXid(cnxn.getXid());
}
p.createBB();
}
sock.write(p.bb);
if (!p.bb.hasRemaining()) {
sentCount++;
outgoingQueue.removeFirstOccurrence(p);
if (p.requestHeader != null
&& p.requestHeader.getType() != OpCode.ping
&& p.requestHeader.getType() != OpCode.auth) {
synchronized (pendingQueue) {
pendingQueue.add(p);
}
}
}
}
if (outgoingQueue.isEmpty()) {
// No more packets to send: turn off write interest flag.
// Will be turned on later by a later call to enableWrite(),
// from within ZooKeeperSaslClient (if client is configured
// to attempt SASL authentication), or in either doIO() or
// in doTransport() if not.
disableWrite();
} else if (!initialized && p != null && !p.bb.hasRemaining()) {
// On initial connection, write the complete connect request
// packet, but then disable further writes until after
// receiving a successful connection response. If the
// session is expired, then the server sends the expiration
// response and immediately closes its end of the socket. If
// the client is simultaneously writing on its end, then the
// TCP stack may choose to abort with RST, in which case the
// client would never receive the session expired event.
disableWrite();
} else {
// Just in case
enableWrite();
}
}
}
}
最终 ConnectRequest对象 ，sock.write(p.bb); 由这行代码写出去，发送给server端。
服务端代码：
当收到服务端的响应的时候，会进入if(sockKey.isReadable())
readConnectResult 并且触发KeeperState.SyncConnected; 这个event，放入waitingEvents 队列中。
之后我们进入服务端的代码NIOServerCnxnFactory类的run方法：
public void run() {
while (!ss.socket().isClosed()) {
try {
"每隔1秒，select一下，尝试获取accept连接"
selector.select(1000);
Set selected;
synchronized (this) {
selected = selector.selectedKeys();
}
ArrayList selectedList = new ArrayList(
selected);
Collections.shuffle(selectedList);
for (SelectionKey k : selectedList) {
"只关心accept位，OP_ACCEPT=16,如果有accpet事件来了，就accept一下，获取连接"
if ((k.readyOps() & SelectionKey.OP_ACCEPT) != 0) {
SocketChannel sc = ((ServerSocketChannel) k
.channel()).accept();
InetAddress ia = sc.socket().getInetAddress();
int cnxncount = getClientCnxnCount(ia);
if (maxClientCnxns > 0 && cnxncount >= maxClientCnxns){
LOG.warn("Too many connections from " + ia
+ " - max is " + maxClientCnxns );
sc.close();
} else {
LOG.info("Accepted socket connection from "
+ sc.socket().getRemoteSocketAddress());
sc.configureBlocking(false);
SelectionKey sk = sc.register(selector,
SelectionKey.OP_READ);
NIOServerCnxn cnxn = createConnection(sc, sk);
sk.attach(cnxn);
addCnxn(cnxn);
}
} else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {
NIOServerCnxn c = (NIOServerCnxn) k.attachment();
c.doIO(k);
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Unexpected ops in select "
+ k.readyOps());
}
}
}
selected.clear();
} catch (RuntimeException e) {
LOG.warn("Ignoring unexpected runtime exception", e);
} catch (Exception e) {
LOG.warn("Ignoring exception", e);
}
}
closeAll();
LOG.info("NIOServerCnxn factory exited run method");
}
因为我们接受到了客户端发过来的COnnectRequest对象，所以我们进入了
NIOServerCnxn c = (NIOServerCnxn) k.attachment();
c.doIO(k);
进入了ZOokeeperServer的processConnectRequest方法
public void processConnectRequest(ServerCnxn cnxn, ByteBuffer incomingBuffer) throws IOException {
"===============省略n行代码，反序列化出connectRequest对象"
int sessionTimeout = connReq.getTimeOut();
byte passwd[] = connReq.getPasswd();
int minSessionTimeout = getMinSessionTimeout();
if (sessionTimeout < minSessionTimeout) {
sessionTimeout = minSessionTimeout;
}
int maxSessionTimeout = getMaxSessionTimeout();
if (sessionTimeout > maxSessionTimeout) {
sessionTimeout = maxSessionTimeout;
}
"设定会话的超时时间"
cnxn.setSessionTimeout(sessionTimeout);
// We don't want to receive any packets until we are sure that the
// session is setup
cnxn.disableRecv();
long sessionId = connReq.getSessionId();
if (sessionId != 0) {
long clientSessionId = connReq.getSessionId();
LOG.info("Client attempting to renew session 0x"
+ Long.toHexString(clientSessionId)
+ " at " + cnxn.getRemoteSocketAddress());
serverCnxnFactory.closeSession(sessionId);
cnxn.setSessionId(sessionId);
reopenSession(cnxn, sessionId, passwd, sessionTimeout);
} else {
LOG.info("Client attempting to establish new session at "
+ cnxn.getRemoteSocketAddress());
"进入这里的createSession方法"
createSession(cnxn, passwd, sessionTimeout);
}
}
/进入这里的createSession方法
long createSession(ServerCnxn cnxn, byte passwd[], int timeout) {
long sessionId = sessionTracker.createSession(timeout);
Random r = new Random(sessionId ^ superSecret);
r.nextBytes(passwd);
ByteBuffer to = ByteBuffer.allocate(4);
to.putInt(timeout);
cnxn.setSessionId(sessionId);
submitRequest(cnxn, sessionId, OpCode.createSession, 0, to, null);
return sessionId;
}
sessionTracker是zk的会话管理器。就把当前会话，放到sessionTracker这个容器里面。然后submitRequest到责任链。
我们看到firstProcessor就是PrepRequestProcessor
protected void setupRequestProcessors() {
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
RequestProcessor syncProcessor = new SyncRequestProcessor(this,
finalProcessor);
((SyncRequestProcessor)syncProcessor).start();
firstProcessor = new PrepRequestProcessor(this, syncProcessor);
((PrepRequestProcessor)firstProcessor).start();
}
最后是FinalRequestProcessor.
我们先看PrepRequestProcessor,从submittedRequests队列中拿去请求，pRequest方法
case OpCode.createSession:
case OpCode.closeSession:
"转化成事物请求"
pRequest2Txn(request.type, zks.getNextZxid(), request, null, true);
break;
转化成事物请求之后
nextProcessor.processRequest(request); 交给下一个RequestProcessor.
SyncRequestProcessor 这个只是让session落到磁盘，然后我们再看FinalRequestProcessor
case OpCode.createSession: {
zks.serverStats().updateLatency(request.createTime);
lastOp = "SESS";
cnxn.updateStatsForResponse(request.cxid, request.zxid, lastOp,
request.createTime, Time.currentElapsedTime());
zks.finishSessionInit(request.cnxn, true);
return;
}
构建 ConnectResponse 对象，并且cnxn.sendBuffer(bb); sock.write(bb);最后写出去，发送给客户端。