Rabbitmq 概述及代码示例
1、概述
amqp（advanced message queuing protocol）协议是一个高级抽象层消息协议，即高级消息队列协议，是应用层协议的一个开放标准，为面向消息的中间价设计。消息中间价主要用于组件之间的解耦，消息的发送者无需知道消息使用者的存在。
amqp的特征是面向消息、队列、路由（包括点对点和发布/订阅）、可靠性和安全性。
2、使用场景
a、两个系统间需要通过定时任务来同步数据
b、异构系统的不同进程间相互调用、通讯
c、生产者消费者模型的使用
3、基本概念
ConnectionFactory、Connection、Channel都是Rabbitmq对外提供的API中最基本的对象。Connection是rabbitmq的socket连接，它封装了socket协议的相关部分逻辑。connectionfactory为Connection的制造工厂。
channel是我们与Rabbitmq打交道的一个借口，我们大部分的业务操作都是在channel这个借口中完成，包括定义queue、发布exchange、绑定queue和exchange、发布消息等。
4、基本消息类型介绍【Queue】
queue是rabbitmq的内部对象，用于存储消息。
rabbitmq中的所有消息都存储在queue中，上图为生产者消费者模型，图一为一个生产者对应一个消费者，图二为一个生产者对应多个（两个）消费者，那么在对应的使用场景下，我们可以应用于这样的场景：A系统和B系统间需要定时通信，如A系统中的一些订单数据需要到B系统中进行实时统计分析，那么将订单数据放入消息队列，B系统从消息队列中读取便实现系统间通讯。场景二：A系统中需要给用户发送打折消息，用户量很大，需要发送时间很长，将需要发送的人员写入消息队列，同时开启多个线程从消息队列中读取人员信息，并发送消息，就是一个生产者对应多个消费者，实现异步处理。
#!/usr/bin/env python#生产者端import pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.queue_declare(queue=\'task_queue\', durable=True)message = \' \'.join(sys.argv[1:]) or "Hello World!"channel.basic_publish(exchange=\'\', routing_key=\'task_queue\', body=message, properties=pika.BasicProperties( delivery_mode = 2, # make message persistent ))print(" [x] Sent %r" % message)connection.close()
#!/usr/bin/env python#消费者端import pikaimport timeconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.queue_declare(queue=\'task_queue\', durable=True)print(\' [*] Waiting for messages. To exit press CTRL+C\')def callback(ch, method, properties, body): print(" [x] Received %r" % body) time.sleep(body.count(b\'.\')) print(" [x] Done") ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_qos(prefetch_count=1)channel.basic_consume(callback, queue=\'task_queue\')channel.start_consuming()
5、Message acknowledge
在应用场景中，我们可能需要掌握消息是否被消费者正确消费，这时消费者消费完后发送一条消息给生产者，队列知道消息被消费了，从队列中移除。
但是产生另外一个问题，如果开发人员处理完逻辑后，忘了发送回执为rabbitmq，会导致queue中国堆积消息越来越多，消费者重启后消息重复发送给消费者，导致重复执行，因此要注意。
6、message durability
如果我们希望rabbitmq重启后，消息不丢失，可以设置queue为持久化的，重启后消息依然存在。
7、prefetch count
有多个消费者时，会出现问题，有的消费者干的活很多，而有的很闲，因此通过设置prefetchCount=1，使得消息公平分发。
8、exchange
消息我们需要通过一个交换器，由路由器将消息分发到不同的消费者上，此时消息可能被分发到一个队列或者多个队列。
9、routing key
生产者将消息发送给exchang的时候，一般会指定一个routing key，来指定这个消息的路由规则，而这个routing key 需要与exhcange type 及binding key 联合使用才能生效。
生产者将消息发送给exchange时，通过指定routing key来确定消息流流向。
emit_producer.py
#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'logs\', type=\'fanout\')message = \' \'.join(sys.argv[1:]) or "info: Hello World!"channel.basic_publish(exchange=\'logs\', routing_key=\'\', body=message)print(" [x] Sent %r" % message)connection.close()
receive_log.py
#!/usr/bin/env pythonimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'logs\', type=\'fanout\')result = channel.queue_declare(exclusive=True)queue_name = result.method.queuechannel.queue_bind(exchange=\'logs\', queue=queue_name)print(\' [*] Waiting for logs. To exit press CTRL+C\')def callback(ch, method, properties, body): print(" [x] %r" % body)channel.basic_consume(callback, queue=queue_name, no_ack=True)channel.start_consuming()
10、binding
rabbitmq通过binding将exchange与queue关联起来，这样rabbitmq就知道如何正确将消息路由到指定的queue了。
11、binding key
绑定exchange与queue的同时，一般会指定binding key；binding key 与routing key匹配时，消息会被路由到相应的queue。
12、exchange type
rabbitmq常用的exchange type有fanout、direct、topic、headers四种。
13、fanout
fanout类型的exchange 路由规则非常简单，他会把所有发送到该exchange的消息路由到所有与他绑定的queue中
14、direct
direct会把消息路由到那些binding key 与routing key 完全匹配的queue中
emit_log_direct.py
#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'direct_logs\', type=\'direct\')severity = sys.argv[1] if len(sys.argv) > 1 else \'info\'message = \' \'.join(sys.argv[2:]) or \'Hello World!\'channel.basic_publish(exchange=\'direct_logs\', routing_key=severity, body=message)print(" [x] Sent %r:%r" % (severity, message))connection.close()
recieve_logs_direct.py
#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'direct_logs\', type=\'direct\')result = channel.queue_declare(exclusive=True)queue_name = result.method.queueseverities = sys.argv[1:]if not severities: sys.stderr.write("Usage: %s [info] [warning] [error]\\n" % sys.argv[0]) sys.exit(1)for severity in severities: channel.queue_bind(exchange=\'direct_logs\', queue=queue_name, routing_key=severity)print(\' [*] Waiting for logs. To exit press CTRL+C\')def callback(ch, method, properties, body): print(" [x] %r:%r" % (method.routing_key, body))channel.basic_consume(callback, queue=queue_name, no_ack=True)channel.start_consuming()
15、topic
前面讲到direct类型的exchange路由规则完全匹配binding key 与routing key，但这种严格的匹配方法在很多情况下不能满足实际需求，可能需要模糊匹配的情况，因此通过topic类型进行模糊匹配
emit_log_topic.py
#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'topic_logs\', type=\'topic\')routing_key = sys.argv[1] if len(sys.argv) > 1 else \'anonymous.info\'message = \' \'.join(sys.argv[2:]) or \'Hello World!\'channel.basic_publish(exchange=\'topic_logs\', routing_key=routing_key, body=message)print(" [x] Sent %r:%r" % (routing_key, message))connection.close()
client.py
#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.exchange_declare(exchange=\'topic_logs\', type=\'topic\')result = channel.queue_declare(exclusive=True)queue_name = result.method.queuebinding_keys = sys.argv[1:]if not binding_keys: sys.stderr.write("Usage: %s [binding_key]...\\n" % sys.argv[0]) sys.exit(1)for binding_key in binding_keys: channel.queue_bind(exchange=\'topic_logs\', queue=queue_name, routing_key=binding_key)print(\' [*] Waiting for logs. To exit press CTRL+C\')def callback(ch, method, properties, body): print(" [x] %r:%r" % (method.routing_key, body))channel.basic_consume(callback, queue=queue_name, no_ack=True)channel.start_consuming()
16、headers
header是根据消息内容中的header属性进行匹配的。
17、RPC
- When the Client starts up, it creates an anonymous exclusive callback queue.
- For an RPC request, the Client sends a message with two properties: reply_to, which is set to the callback queue and correlation_id, which is set to a unique value for every request.
- The request is sent to an rpc_queue queue.
- The RPC worker (aka: server) is waiting for requests on that queue. When a request appears, it does the job and sends a message with the result back to the Client, using the queue from thereply_to field.
- The client waits for data on the callback queue. When a message appears, it checks thecorrelation_id property. If it matches the value from the request it returns the response to the application.
rpc_server.py
#!/usr/bin/env pythonimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\'))channel = connection.channel()channel.queue_declare(queue=\'rpc_queue\')def fib(n): if n == 0: return 0 elif n == 1: return 1 else: return fib(n-1) + fib(n-2)def on_request(ch, method, props, body): n = int(body) print(" [.] fib(%s)" % n) response = fib(n) ch.basic_publish(exchange=\'\', routing_key=props.reply_to, properties=pika.BasicProperties(correlation_id = \\ props.correlation_id), body=str(response)) ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_qos(prefetch_count=1)channel.basic_consume(on_request, queue=\'rpc_queue\')print(" [x] Awaiting RPC requests")channel.start_consuming()
rpc_receiver.py
#!/usr/bin/env pythonimport pikaimport uuidclass FibonacciRpcClient(object): def __init__(self): self.connection = pika.BlockingConnection(pika.ConnectionParameters( host=\'localhost\')) self.channel = self.connection.channel() result = self.channel.queue_declare(exclusive=True) self.callback_queue = result.method.queue self.channel.basic_consume(self.on_response, no_ack=True, queue=self.callback_queue) def on_response(self, ch, method, props, body): if self.corr_id == props.correlation_id: self.response = body def call(self, n): self.response = None self.corr_id = str(uuid.uuid4()) self.channel.basic_publish(exchange=\'\', routing_key=\'rpc_queue\', properties=pika.BasicProperties( reply_to = self.callback_queue, correlation_id = self.corr_id, ), body=str(n)) while self.response is None: self.connection.process_data_events() return int(self.response)fibonacci_rpc = FibonacciRpcClient()print(" [x] Requesting fib(30)")response = fibonacci_rpc.call(30)print(" [.] Got %r" % response)
自己写的一些实验程序
producer.py
#coding=utf-8import sysimport pikaimport timeimport random"""在数据处理结束后发送ack，这样RabbitMQ Server会认为Message Deliver 成功。持久化queue，可以防止RabbitMQ Server 重启或者crash引起的数据丢失。持久化Message，理由同上。答案是否定的。问题就在与RabbitMQ需要时间去把这些信息存到磁盘上，这个time window虽然短，但是它的确还是有。在这个时间窗口内如果数据没有保存，数据还会丢失。还有另一个原因就是RabbitMQ并不是为每个Message都做fsync：它可能仅仅是把它保存到Cache里，还没来得及保存到物理磁盘上。因此这个持久化还是有问题。但是对于大多数应用来说，这已经足够了。当然为了保持一致性，你可以把每次的publish放到一个transaction中。这个transaction的实现需要user defined codes。那么商业系统会做什么呢？一种可能的方案是在系统panic时或者异常重启时或者断电时，应该给各个应用留出时间去flash cache，保证每个应用都能exit gracefully。"""class RbMQProducer(object): def __init__(self): self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=\'localhost\')) def send_message(self): channel = self.connection.channel() channel.queue_declare(queue=\'hello\') for i in xrange(10): time.sleep(1) message = " ".join(sys.argv[1:]) or "hello world! i am {}".format(i) channel.basic_publish(exchange=\'\', routing_key=\'hello\', body=message) print(" [x] Sent \'Hello World!\'") def send_message_durable(self): channel = self.connection.channel() channel.queue_declare(queue=\'hello\',durable=True) for i in xrange(10): time.sleep(1) message = " ".join(sys.argv[1:]) or "hello world! i am {}".format(i) channel.basic_publish(exchange=\'\', routing_key="hello", body=message, properties=pika.BasicProperties( delivery_mode=2, #make the message persistent )) def exchange_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'logs\',type=\'fanout\') for i in xrange(10): message = " ".join(sys.argv[1:]) or "exchange info : Hello world {}!".format(i) channel.basic_publish(exchange=\'logs\', routing_key=\'\', body=message) print " [{}] send {}".format(i,message) time.sleep(1.5) def direct_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'direct_logs\', type=\'direct\') severity = sys.argv[1] if len(sys.argv)>1 else [\'info\',\'warning\',\'error\'] for i in xrange(10): message = " ".join(sys.argv[2:]) or \'hello world!\' routing_key_random = severity[random.randint(0,len(severity)-1)] channel.basic_publish(exchange=\'direct_logs\', routing_key=routing_key_random, body=message) print "[{}]: {}".format(routing_key_random,message) time.sleep(1.5) def topic_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'topic_logs\', type=\'topic\') routing_key = sys.argv[1] if len(sys.argv)>1 else ["anonymous.info","lck.info","lck.error"] for i in xrange(10): routing_key_value = routing_key[random.randint(0,len(routing_key)-1)] message = " ".join(sys.argv[2:]) or "hello world! " channel.basic_publish(exchange=\'topic_logs\', routing_key=routing_key_value, body=message) print "[{}] : {}".format(routing_key_value,message) time.sleep(1.5) def rpc_message(self): channel = self.connection.channel() channel.queue_declare(queue=\'rpc_queue\') channel.basic_qos(prefetch_count=1) channel.basic_consume(self.on_request,queue=\'rpc_queue\') print "[x] Awaiting rpc requests" channel.start_consuming() def on_request(self,ch, method, props, body): n = int(body) response = random.randint(100,1000) print "[.]{}".format(response) ch.basic_publish(exchange=\'\', routing_key=props.reply_to, properties=pika.BasicProperties(correlation_id=props.correlation_id), body=str(response)) ch.basic_ack(delivery_tag=method.delivery_tag) def __del__(self): self.connection.close()if __name__ == "__main__": rm_producer = RbMQProducer() # rm_producer.send_message() # rm_producer.send_message_durable() rm_producer.exchange_message() # rm_producer.direct_message() # rm_producer.topic_message() # rm_producer.rpc_message()
receiver.py
#coding=utf-8import sysimport pikaimport timeimport uuidclass RbMQconsume(object): def __init__(self): self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=\'localhost\')) def callback(self, ch, method, properties, body): print(" [x] Received %r" % body) time.sleep( body.count(\'.\')) print(" [x] Done") def receive_message(self): channel = self.connection.channel() channel.queue_declare(queue=\'hello\') channel.basic_consume(self.callback, queue=\'hello\', no_ack=True) print(\' [*] Waiting for messages. To exit press CTRL+C\') channel.start_consuming() def safe_callback(self,ch,method,properties,body): print " [x] Received %r"%(body) time.sleep(body.count(\'.\')) print( " [x] done !") ch.basic_ack(delivery_tag=method.delivery_tag) def safe_receive_message(self): channel = self.connection.channel() channel.queue_declare(queue=\'hello\',durable=True) channel.basic_qos(prefetch_count=1) channel.basic_consume(self.safe_callback, queue=\'hello\') print(\' [*] Waiting for messages. To exit press CTRL+C\') channel.start_consuming() #exchange方法 def exchange_callback(self,ch,method,properties,body): print " [ x ] {}".format(body) def receieve_exchange_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'logs\', type=\'fanout\') result = channel.queue_declare(exclusive=True) queue_name = result.method.queue channel.queue_bind(exchange=\'logs\',queue=queue_name) channel.basic_consume(self.exchange_callback,queue=queue_name,no_ack=True) channel.start_consuming() print(\' [*] Waiting for messages. To exit press CTRL+C\') #direct 方法 def direct_callback(self, ch, method, properties, body): print " [ {} ] {}".format(method.routing_key,body) def receieve_direct_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'direct_logs\', type =\'direct\') result = channel.queue_declare(exclusive=True) queue_name = result.method.queue severities = [\'warning\', \'error\'] for severity in severities: channel.queue_bind(exchange=\'direct_logs\', queue=queue_name, routing_key=severity) channel.basic_consume(self.direct_callback, queue=queue_name, no_ack=True) channel.start_consuming() #topic def topic_callback(self, ch, method, properties, body): print " [ {} ] {}".format(method.routing_key, body) def receieve_topic_message(self): channel = self.connection.channel() channel.exchange_declare(exchange=\'topic_logs\', type=\'topic\') result = channel.queue_declare(exclusive=True) queue_name = result.method.queue binding_keys = ["anonymous.info","lck.info","lck.error"] binding_key = \'lck.*\' channel.queue_bind(exchange=\'topic_logs\', queue=queue_name, routing_key=binding_key) channel.basic_consume(self.topic_callback, queue= queue_name, no_ack=True) channel.start_consuming() #rpc def rpc_callback(self,ch,method,props,body): if self.corr_id == props.correlation_id: self.response = body def receieve_rpc_message(self,n): channel = self.connection.channel() result = channel.queue_declare(exclusive=True) callback_queue = result.method.queue channel.basic_consume(self.rpc_callback, no_ack=True, queue=callback_queue) self.response = None self.corr_id = str(uuid.uuid4()) channel.basic_publish(exchange=\'\', routing_key=\'rpc_queue\', properties=pika.BasicProperties(reply_to=callback_queue,correlation_id=self.corr_id), body=str(n)) while self.response is None: self.connection.process_data_events() return int(self.response) def __del__(self): self.connection.close()if __name__=="__main__": rqconsume = RbMQconsume() # rqconsume.safe_receive_message() rqconsume.receieve_exchange_message() # rqconsume.receieve_direct_message() # rqconsume.receieve_topic_message() # print rqconsume.receieve_rpc_message(30)