传输层的各种模式——ZeroMQ 库的使用 .

最近在研究 ZeroMQ 库的使用,所以在这里总结一下各种模式,以便日后拿来使用。

关于 ZeroMQ 库,我就不多介绍了,大家可以参考下面一些文章,以及他的官网、使用指南、API 参考、项目仓库等内容。

开源点评:ZeroMQ简介

ZeroMQ的学习和研究

ZeroMQ 的模式

ZeroMQ 的目标是成为 OSI 模型的传输层(Transport Layer)的标准协议,所以他支持各种操作系统,支持多达30种以上的语言,是跨平台、跨语言的传输层库,而且性能是其绝对优势。所以对于进程间通信、节点间通信均可以使用他,可以用他直接替代 socket 的操作。而且用过之后就知道,他用起来非常的简单,学习成本很低,我只用了 1 天时间就把他的 3 种常用模式用 Python 实现了,代码在这里,大家可以参考一下,接下来准备用 C++ 再实现一遍。

ZeroMQ 总结的通信模式如下:

  • PUB and SUB
  • REQ and REP
  • REQ and ROUTER
  • DEALER and REP
  • DEALER and ROUTER
  • DEALER and DEALER
  • ROUTER and ROUTER
  • PUSH and PULL
  • PAIR and PAIR

ZeroMQ 总结的应用模式如下:

  • Request-reply, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern.
  • Publish-subscribe, which connects a set of publishers to a set of subscribers. This is a data distribution pattern.
  • Pipeline, which connects nodes in a fan-out / fan-in pattern that can have multiple steps, and loops. This is a parallel task distribution and collection pattern.

当然,实际使用中还得了解一些解决具体问题的模式,所以下面把使用指南中的一些模式整理如下,方便自己日后拿来使用。

最常用的三种模式:

1. Request-Reply

Request-Reply

Extended Request-reply

Request-reply Broker

Multithreaded Server

服务器端代码:

package zmq20140508;

import org.zeromq.ZMQ;
/**
 * 
 * @author lijianhua
 *
 */

public class ZMQServer {
	
	public static void main(String[] args) {
		Client client =new Client();
		client.start();
	}
public static class Client extends Thread{
	private String url_worker="inproc://workers";
	private String url_client="tcp://127.0.0.1:6666";
	private ZMQ.Poller poller;
	private ZMQ.Context context;
	private ZMQ.Socket clients;
	private ZMQ.Socket workers;
	@SuppressWarnings("deprecation")
	public void run(){
		context = ZMQ.context(1);
		clients = context.socket(ZMQ.ROUTER);
		clients.bind(url_client);
		
		workers = context.socket(ZMQ.DEALER);
		workers.bind(url_worker);
		
		for(int i=0;i<10;i++){
			new Worker(context,url_worker).start();
		}

		ZMQ.device(ZMQ.QUEUE, clients, workers);//开始因这个犯错,搞了半天
		poller=new ZMQ.Poller(2);//创建一个大小为2的poller
		//分别将上述的pull注册到poller上,注册的事件是读
		ZMQ.PollItem citem = new ZMQ.PollItem(clients, ZMQ.Poller.POLLIN);
		ZMQ.PollItem witem = new ZMQ.PollItem(workers, ZMQ.Poller.POLLIN);
		poller.register(citem);
		poller.register(witem);
		boolean ok =true;
		while(ok){
			poller.poll(); 
				if(poller.getItem(0).isReadable()){
					System.out.println("当前发送:clients");
					byte[] recv = clients.recv();
					workers.send(recv);
				}else{
					System.out.println("当前发送:workers");
					byte[] recv = workers.recv();
					clients.send(recv);
				}
		}
		closeAll();
	}
	public void closeAll(){
		clients.close();
		workers.close();
		context.term();
	}
}
	public static class Worker extends Thread{
		private String url_worker1;
		private ZMQ.Context context1;
		public Worker(ZMQ.Context context,String url_worker) {
			this.url_worker1=url_worker;
			this.context1=context;
		}
		public void run() {
			ZMQ.Socket socket = context1.socket(ZMQ.REP);
			socket.connect(url_worker1);
			ZMQ.Poller poller=new ZMQ.Poller(1);
			ZMQ.PollItem item = new ZMQ.PollItem(socket, ZMQ.Poller.POLLIN);
			poller.register(item);
			while (true){
				if(poller.poll()==ZMQ.Poller.POLLIN){
					byte[] recv = socket.recv();
					System.out.println("fix========"+new String(recv));
					try {
						Thread.sleep(1);
					} catch (InterruptedException e) {
						e.printStackTrace();
					}
					socket.send("world========");
				}
			}
		}
	}
}

客服端代码

package zmq20140508;

import org.zeromq.ZMQ;
import org.zeromq.ZMQ.Context;
import org.zeromq.ZMQ.Socket;

public class Request {

	public static void main(String[] args) {
		System.out.println("客户端开始.........");
		Context context = ZMQ.context(1);
		Socket resSocket = context.socket(ZMQ.REQ);//客服端发送消息
//		resSocket.connect("tcp://115.236.73.253:5570");
		resSocket.connect("tcp://127.0.0.1:6666");
		int i=0;
		while(i<5){
			i++;
			try {
				resSocket.send("hello");
				byte[] msg = resSocket.recv();
				String outputStr = new String(msg);
				System.out.println("#### Client Receive:" + outputStr);
			} catch (Exception e) {
				e.printStackTrace();
			}
		}

	}

}


2. Publish-Subscribe

Publish-Subscribe

Pub-Sub Network with a Proxy

Extended Publish-Subscribe

Pub-Sub Forwarder Proxy

Pub-Sub Synchronization


3. Parallel Pipeline

Parallel Pipeline

Parallel Pipeline with Kill Signaling


其他模式:

其他模式只是上面三种模式的加强而已,所以也是分为三大类。

A. Request-Reply Patterns

1. The Load-balancing Pattern

Load-Balancing Broker

代码示例:

package zmq20140505;
import java.util.LinkedList;

import org.zeromq.ZFrame;
import org.zeromq.ZMQ;
import org.zeromq.ZMsg;

public class Balance {
	
	public static class Client {
		public void start() {
			new Thread(new Runnable(){

				public void run() {
					// TODO Auto-generated method stub
					ZMQ.Context context = ZMQ.context(1);
					ZMQ.Socket socket = context.socket(ZMQ.REQ);
					socket.connect("ipc://127.0.0.1:5555");  //连接router,想起发送请求
					
					while (!Thread.currentThread().isInterrupted()) {
						socket.send("hello".getBytes(), 0);  //发送hello请求
						String bb = new String(socket.recv());  //获取返回的数据
						System.out.println("recv Worker : "+bb);	
					}
					socket.close();
					context.term();
				}
				
			}).start();
		}
	}
	
	public static class Worker {
		public void start() {
			new Thread(new Runnable(){

				public void run() {
					// TODO Auto-generated method stub
					ZMQ.Context context = ZMQ.context(1);
					ZMQ.Socket socket = context.socket(ZMQ.REQ);
					
					socket.connect("ipc://127.0.0.1:6666");  //连接,用于获取要处理的请求,并发送回去处理结果
					
					socket.send("ready".getBytes());  //发送ready,表示当前可用
					 
					while (!Thread.currentThread().isInterrupted()) {
						ZMsg msg = ZMsg.recvMsg(socket);  //获取需要处理的请求,其实这里msg最外面的标志frame是router对分配给client的标志frame
						ZFrame request = msg.removeLast();   //最后一个frame其实保存的就是实际的请求数据,这里将其移除,待会用新的frame代替
						
						String now = new String(request.getData());
						System.out.println("recv Client : " + now);
						
						ZFrame frame = new ZFrame("hello fjs".getBytes());  
						msg.addLast(frame);  //将刚刚创建的frame放到msg的最后,worker将会收到
						msg.send(socket);  //将数据发送回去
					}
					socket.close();
					context.term();
				}
				
			}).start();
		}
	}
	
	public static class Middle {
		private LinkedList workers;
		private LinkedList requests;
		private ZMQ.Context context;
		private ZMQ.Poller poller;
		
		public Middle() {
			this.workers = new LinkedList();
			this.requests = new LinkedList();
			this.context = ZMQ.context(1);
			this.poller = new ZMQ.Poller(2);
		}
		
		public void start() {
			ZMQ.Socket fronted = this.context.socket(ZMQ.ROUTER);  //创建一个router,用于接收client发送过来的请求,以及向client发送处理结果
			ZMQ.Socket backend = this.context.socket(ZMQ.ROUTER);  //创建一个router,用于向后面的worker发送数据,然后接收处理的结果
			
			fronted.bind("ipc://127.0.0.1:5555");  //监听,等待client的连接
			backend.bind("ipc://127.0.0.1:6666");  //监听,等待worker连接
			
			//创建pollItem
			ZMQ.PollItem fitem = new ZMQ.PollItem(fronted, ZMQ.Poller.POLLIN);  
			ZMQ.PollItem bitem = new ZMQ.PollItem(backend, ZMQ.Poller.POLLIN);
			
			this.poller.register(fitem);  //注册pollItem
			this.poller.register(bitem);
			
			
			while (!Thread.currentThread().isInterrupted()) {
				this.poller.poll();
				if (fitem.isReadable()) {  //表示前面有请求发过来了
					ZMsg msg = ZMsg.recvMsg(fitem.getSocket());  //获取client发送过来的请求,这里router会在实际请求上面套一个连接的标志frame
					/**
					 * //以下3行查看中间层客服端发过来的值
					 */
					ZFrame request = msg.getLast();
					String now = new String(request.getData());
					System.out.println("fix===="+now);
					
					this.requests.addLast(msg);   //将其挂到请求队列
				}
				if (bitem.isReadable()) {  //这里表示worker发送数据过来了
					ZMsg msg = ZMsg.recvMsg(bitem.getSocket());  //获取msg,这里也会在实际发送的数据前面包装一个连接的标志frame
					//这里需要注意,这里返回的是最外面的那个frame,另外它还会将后面的接着的空的标志frame都去掉
					ZFrame workerID = msg.unwrap();  //把外面那层包装取下来,也就是router对连接的标志frame
					this.workers.addLast(workerID);  //将当前的worker的标志frame放到worker队列里面,表示这个worker可以用了
					ZFrame readyOrAddress = msg.getFirst(); //这里获取标志frame后面的数据,如果worker刚刚启动,那么应该是发送过来的ready,
					if (new String(readyOrAddress.getData()).equals("ready")) {  //表示是worker刚刚启动,发过来的ready
						msg.destroy();
					} else {
						msg.send(fronted);  //表示是worker处理完的返回结果,那么返回给客户端
					}
				}
				while (this.workers.size() > 0 && this.requests.size() > 0) {
					ZMsg request = this.requests.removeFirst();
					ZFrame worker = this.workers.removeFirst();
					
					request.wrap(worker);  //在request前面包装一层,把可以用的worker的标志frame包装上,这样router就会发给相应的worker的连接
					request.send(backend);  //将这个包装过的消息发送出去
				}
			}
			fronted.close();
			backend.close();
			this.context.term();
		}
	}
	
	
	public static void main(String args[]) {
		Worker worker = new Worker();
		worker.start();
		Client client = new Client();
		client.start();
		Middle middle = new Middle();
		middle.start();
		
	}
}


2. The Asynchronous Client-Server Pattern

Asynchronous Client-Server

Detail of Asynchronous Server

Cluster Architecture

Multiple Clusters

Idea 1 - Cross-connected Workers

Idea 2 - Brokers Talking to Each Other

Cross-connected Brokers in Federation Model


3. Client-side Reliability (Lazy Pirate Pattern)

The Lazy Pirate Pattern


4. Basic Reliable Queuing (Simple Pirate Pattern)

The Simple Pirate Pattern


5. Robust Reliable Queuing (Paranoid Pirate Pattern)

The Paranoid Pirate Pattern


6. Service-Oriented Reliable Queuing (Majordomo Pattern)

The Majordomo Pattern


7. Disconnected Reliability (Titanic Pattern)

The Titanic Pattern


8. High-availability Pair (Binary Star Pattern)

High-availability Pair, Normal Operation

High-availability Pair During Failover

Binary Star Finite State Machine


9. Brokerless Reliability (Freelance Pattern)

The Freelance Pattern


B. Publish-Subscribe Patterns

1. Pub-sub Tracing (Espresso Pattern)

2. Slow Subscriber Detection (Suicidal Snail Pattern)

3. High-speed Subscribers (Black Box Pattern)

The Simple Black Box Pattern

Mad Black Box Pattern


4. Reliable Publish-Subscribe (Clone Pattern)

Publishing State Updates

State Replication

Republishing Updates

Clone Client Finite State Machine

High-availability Clone Server Pair


C. Parallel Pipeline Patterns

你可能感兴趣的:(ZMQ)