PythonSocket套接字中的异步、多线程等特性对服务器性能有很大影响,总结一些关键的框架以便开发过程中很快找到提高并发性性能的模型。
(1)TCP连接流程如下图:
SocketTCP服务器编程步骤:1、打开socket,2、绑定到一个地址和端口,3、侦听进来的连接,4、接受连接,5、读写数据,6、关闭socket。
SocketTCP客户端编程步骤:1、打开socket,2、连接到服务器,3、读写数据,4、关闭socket。
(2)UDP连接流程如下图:
SocketUDP服务器编程步骤:1、打开socket,2、绑定到一个地址和端口,3、接收客户端数据,4、发送数据,6、关闭。
SocketUDP客户端编程步骤:1、打开socket,2、绑定到地址和端口(可省略),3、发送数据,4、接收数据,5、关闭。
#!/usr/bin/python #encoding=utf-8 from SocketServer import TCPServer, ForkingMixIn, StreamRequestHandler import time class Server(ForkingMixIn, TCPServer): #自定义Server类 pass class MyHandler(StreamRequestHandler): def handle(self): #重载handle函数 addr = self.request.getpeername() print 'Get connection from', addr #打印客户端地址 time.sleep(5) #休眠5秒钟 self.wfile.write('This is a ForkingMixIn tcp socket server') #发送信息 host = '' port = 1234 server = Server((host, port), MyHandler) server.serve_forever() #开始侦听并处理连接多个连接同时到达服务器端的时候,每个连接主进程都生成一个子进程专门用来处理此连接,而主进程则依旧保持在侦听状态。因主进程和子进程是同时进行的,所以不会阻塞新的连接。但由于生成进程消耗的资源比较大,这种处理方式在有很多连接的时候会带来性能问题。Server类须继承ForkingMixIn和TCPServer两个类。
#!/usr/bin/python #encoding=utf-8 import socket s = socket.socket() #生成一个socket对象 server = socket.gethostname() port = 1234 s.connect((server, port)) #连接服务器 print s.recv(1024) #读取数据 s.close() #关闭连接服务器端运行结果:
#!/usr/bin/python #encoding=utf-8 from SocketServer import TCPServer, ThreadingMixIn, StreamRequestHandler import time class Server(ThreadingMixIn, TCPServer): #自定义Server类 pass class MyHandler(StreamRequestHandler): def handle(self): #重载handle函数 addr = self.request.getpeername() print 'Get connection from', addr #打印客户端地址 time.sleep(5) #休眠5秒钟 self.wfile.write('This is a ForkingMixIn tcp socket server') #发送信息 host = '' port = 1234 server = Server((host, port), MyHandler) server.serve_forever() #开始侦听并处理连接
线程是一种轻量级的进程,比Fork消耗的资源更少,而且主线程和子线程之间具有相同的地址空间,处理效率高。但大量的使用线程会带来线程之间的数据同步问题,处理不好可能使服务程序失去响应。可以参考Stackless Python是Python的增强版本,能有效的利用线程。上述与Fork方式中代码基本相同,仅仅是采用的ThreadingMixIn类不同。
import SocketServer class MyTCPHandler(SocketServer.BaseRequestHandler): def handle(self): while True: self.data = self.request.recv(1024).strip() cur_thread = threading.current_thread() print cur_thread if not self.data: print "client:%s leave!" % self.client_address[0] break print "%s wrote:%s" % (self.client_address[0], self.data) self.request.sendall(self.data.upper()) if __name__ == "__main__": HOST, PORT = "localhost", 9999 server = SocketServer.ThreadingTCPServer((HOST, PORT), MyTCPHandler) server.serve_forever()
#!/usr/bin/python #encoding=utf-8 import socket, select s = socket.socket() #生成socket对象 host = socket.gethostname() port = 1234 s.bind((host, port)) #绑定套接字接口地址 s.listen(5) #开始服务器端监听 inputs = [s] while True: rs, ws, es = select.select(inputs, [], []) #使用select方法 for r in rs: if r is s: c, addr = s.accept() #处理连接 print 'Get connection from', addr inputs.append(c) else: try: data = r.recv(1024) #接收数据 disconnected = not data except socket.error: disconnected = True if disconnected: print r.getpeername(), 'disconnected' inputs.remove(r) else: print data #打印接收到的数据客户端测试代码:
#!/usr/bin/env python from socket import * HOST = 'localhost' PORT = 1234 BUFSIZ = 1024 ADDR = (HOST, PORT) tcpCliSock = socket(AF_INET, SOCK_STREAM) tcpCliSock.connect(ADDR) while True: data = raw_input('> ') if not data: break tcpCliSock.send(data) tcpCliSock.close()运行结果:
#!/usr/bin/python #encoding=utf-8 import select import socket import Queue #create a socket server = socket.socket(socket.AF_INET,socket.SOCK_STREAM) server.setblocking(False) #set option reused server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR , 1) server_address= ('192.168.1.102',10001) server.bind(server_address) server.listen(10) #sockets from which we except to read inputs = [server] #sockets from which we expect to write outputs = [] #Outgoing message queues (socket:Queue) message_queues = {} #A optional parameter for select is TIMEOUT timeout = 20 while inputs: print "waiting for next event" readable , writable , exceptional = select.select(inputs, outputs, inputs, timeout) # When timeout reached , select return three empty lists if not (readable or writable or exceptional) : print "Time out ! " break; for s in readable : if s is server: # A "readable" socket is ready to accept a connection connection, client_address = s.accept() print " connection from ", client_address connection.setblocking(0) inputs.append(connection) message_queues[connection] = Queue.Queue() else: data = s.recv(1024) if data : print " received " , data , "from ",s.getpeername() message_queues[s].put(data) # Add output channel for response if s not in outputs: outputs.append(s) else: #Interpret empty result as closed connection print " closing", client_address if s in outputs : outputs.remove(s) inputs.remove(s) s.close() #remove message queue del message_queues[s] for s in writable: try: next_msg = message_queues[s].get_nowait() except Queue.Empty: print " " , s.getpeername() , 'queue empty' outputs.remove(s) else: print " sending " , next_msg , " to ", s.getpeername() s.send(next_msg) for s in exceptional: print " exception condition on ", s.getpeername() #stop listening for input on the connection inputs.remove(s) if s in outputs: outputs.remove(s) s.close() #Remove message queue del message_queues[s]Client端创建多个socket进行server的测试程序:
#!/usr/bin/python #encoding=utf-8 import socket messages = ["This is the message" , "It will be sent" , "in parts "] print "Connect to the server" server_address = ("192.168.1.102",10001) #Create a TCP/IP sock socks = [] for i in range(10): socks.append(socket.socket(socket.AF_INET,socket.SOCK_STREAM)) for s in socks: s.connect(server_address) counter = 0 for message in messages : #Sending message from different sockets for s in socks: counter+=1 print " %s sending %s" % (s.getpeername(),message+" version "+str(counter)) s.send(message+" version "+str(counter)) #Read responses on both sockets for s in socks: data = s.recv(1024) print " %s received %s" % (s.getpeername(),data) if not data: print "closing socket ",s.getpeername() s.close()
(2)poll方法
poll方法应用很广泛 ,在需要同时为很多连接服务的时候比较有用。以为select方法采用的是一种位图索引的方式来处理文件描述符而poll方法则仅仅只需要处理感兴趣的文件描述符,所以select与最大的文件描述符是一致的而poll与文件描述符的个数是一致的,poll方法的这种特点可以有效的降低服务器的处理负担。 pool方法在select模块中,当调用poll方法时将得到一个Polling类对象,该对象有register、unregister和poll三个方法,poll方法有一个可选的超时参数,若被忽略、为负数或为0,则调用此方法将阻塞到至少有一个事件到达。poll方法将返回(fd,event)对的列表,其中fd为文件描述符,event用来指示发生的事件,event是一个位掩码通过一个整数的位来对应特定的事件信息,若需要知道特定的事件是否发生,可以使用&操作符。
poll方法的事件信息:
#!/usr/bin/python #encoding=utf-8 import socket, select s = socket.socket() #生成socket对象 host = socket.gethostname() port = 1235 s.bind((host, port)) #绑定套接字接口地址 fd_dict = {s.fileno(): s} s.listen(5) #开始服务器端监听 p = select.poll() #生成Polling对象 p.register(s) #注册socket对象 while True: events = p.poll() #获取准备好的文件对象 for fd, event in events: st = fd_dict[fd] if st is s: c, addr = s.accept() #处理连接 print 'Got connection from', addr p.register(c) fd_dict[c.fileno()] = c #加入连接socket elif event & select.POLLIN: data = fd_dict[fd].recv(1024) #接收时间 if not data: print fd_dict[fd].getpeername(), 'disconnected' p.unregister(fd) #取消注册 del fd_dict[fd] else: print data #打印数据
#!/usr/bin/python #encoding=utf-8 import socket import select import Queue # Create a TCP/IP socket, and then bind and listen server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.setblocking(False) server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) server_address = ("192.168.1.102", 10001) print "Starting up on %s port %s" % server_address server.bind(server_address) server.listen(5) message_queues = {} #The timeout value is represented in milliseconds, instead of seconds. timeout = 1000 # Create a limit for the event READ_ONLY = ( select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR) READ_WRITE = (READ_ONLY|select.POLLOUT) # Set up the poller poller = select.poll() poller.register(server,READ_ONLY) #Map file descriptors to socket objects fd_to_socket = {server.fileno():server,} while True: print "Waiting for the next event" events = poller.poll(timeout) print "*"*20 print len(events) print events print "*"*20 for fd ,flag in events: s = fd_to_socket[fd] if flag & (select.POLLIN | select.POLLPRI) : if s is server : # A readable socket is ready to accept a connection connection , client_address = s.accept() print " Connection " , client_address connection.setblocking(False) fd_to_socket[connection.fileno()] = connection poller.register(connection,READ_ONLY) #Give the connection a queue to send data message_queues[connection] = Queue.Queue() else : data = s.recv(1024) if data: # A readable client socket has data print " received %s from %s " % (data, s.getpeername()) message_queues[s].put(data) poller.modify(s,READ_WRITE) else : # Close the connection print " closing" , s.getpeername() # Stop listening for input on the connection poller.unregister(s) s.close() del message_queues[s] elif flag & select.POLLHUP : #A client that "hang up" , to be closed. print " Closing ", s.getpeername() ,"(HUP)" poller.unregister(s) s.close() elif flag & select.POLLOUT : #Socket is ready to send data , if there is any to send try: next_msg = message_queues[s].get_nowait() except Queue.Empty: # No messages waiting so stop checking print s.getpeername() , " queue empty" poller.modify(s,READ_ONLY) else : print " sending %s to %s" % (next_msg , s.getpeername()) s.send(next_msg) elif flag & select.POLLERR: #Any events with POLLERR cause the server to close the socket print " exception on" , s.getpeername() poller.unregister(s) s.close() del message_queues[s]
#!/usr/bin/python #encoding=utf-8 import asyncore, socket class HttpClient (asyncore.dispatcher): #定义了一个HttpClient类 def __init__(self, host, path): #类的构造函数 asyncore.dispatcher.__init__(self) self.create_socket(socket.AF_INET, socket.SOCK_STREAM) #创建socket对象 self.connect( (host, 80) ) self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path def handle_connect(self): #连接调用接口 pass def handle_close(self): #接口关闭函数 self.close() def handle_read(self): #读取数据 print self.recv(1024) def handle_write(self): #写入数据 sent = self.send(self.buffer) self.buffer = self.buffer[sent:] def writable(self): #判断是否写入数据 return (len(self.buffer) > 0) if __name__ == ‘__main__’: c = HttpClient('www.python.org', '/') asyncore.loop() #开始异步通信处理方式
#!/usr/bin/python #encoding=utf-8 from twisted.internet import reactor from twisted.internet.protocol import Protocol, Factory class EchoServer(Protocol): def connectionMade(self): #连接建立的时候 print 'Get connection from', self.transport.client self.factory.numProtocols = self.factory.numProtocols+1 if self.factory.numProtocols > 2: #当连接超过2个的时候,断开连接 self.transport.write("Too many connections, try later\n") self.transport.loseConnection() return print 'Get connection from', self.transport.client def connectionLost(self, reason): #断开连接 self.factory.numProtocols = self.factory.numProtocols-1 def dataReceived (self, data): #将收到的数据返回给客户端 self.transport.write(data) print data factory = Factory() factory.numProtocols = 0 factory.protocol = EchoServer port = 1200 reactor.listenTCP(port, factory) reactor.run() #进入循环使用命令#telnet 127.0.0.1 1200为客户端进行测试:
(2)Twisted框架的官方例子:
#!/usr/bin/env python # coding: utf-8 from twisted.internet.protocol import Protocol from twisted.internet.protocol import Factory from twisted.internet import reactor class Echo(Protocol): '''协议类实现用户的服务协议,例如 http,ftp,ssh 等''' def __init__(self, factory): self.factory = factory def connectionMade(self): '''连接建立时被回调的方法''' self.factory.numProtocols = self.factory.numProtocols + 1 self.transport.write("Welcome! There are currently %d open connections.\n" %(self.factory.numProtocols,)) def connectionLost(self, reason): '''连接关闭时被回调的方法''' self.factory.numProtocols = self.factory.numProtocols - 1 def dataReceived(self, data): '''接收数据的函数,当有数据到达时被回调''' self.transport.write(data) class EchoFactory(Factory): '''协议工厂类,当客户端建立连接的时候,创建协议对象,协议对象与客户端连接一一对应''' numProtocols = 0 #protocol = Echo def buildProtocol(self, addr): return Echo(self) if __name__ == '__main__': # 创建监听端口 FACTORY = EchoFactory() reactor.listenTCP(8007, FACTORY) # 开始监听事件 reactor.run()协议工厂继承自twisted.internet.protocol.Factory,需实现buildProtocol方法,协议工厂负责实例化协议类,不应该保存于连接相关的状态信息,因为协议工厂类仅创建一个。 协议类继承自twisted.internet.protocol.Protocol,需实现dataReceived等方法,在协议类中实现应用协议,每一个客户端连接都会创建一个新的协议类对象。transport就是连接对象,通过它进行网络写数据。
#!/usr/bin/env python # coding: utf-8 from twisted.application import service, internet from echoServ import EchoFactory # 创建应用程序对象 application = service.Application('Echo 服务程序') # 创建 service 对象 myServices = internet.TCPServer(8007, EchoFactory()) # 设置 application 为 service 的父元素 myservices.setServiceParent(application)然后用守护进程方式运行服务,运行命令:#twistd -y echo.tac。
#!/usr/bin/python #encoding=utf-8 from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver from twisted.internet import reactor class Chat(LineReceiver): def __init__(self, users): self.users = users self.name = None self.state = "GETNAME" def connectionMade(self): self.sendLine("What's your name?") def connectionLost(self, reason): if self.users.has_key(self.name): del self.users[self.name] def lineReceived(self, line): if self.state == "GETNAME": self.handle_GETNAME(line) else: self.handle_CHAT(line) def handle_GETNAME(self, name): if self.users.has_key(name): self.sendLine("Name taken, please choose another.") return self.sendLine("Welcome, %s!" % (name,)) self.name = name self.users[name] = self self.state = "CHAT" def handle_CHAT(self, message): message = "<%s> %s" % (self.name, message) for name, protocol in self.users.iteritems(): if protocol != self: protocol.sendLine(message) class ChatFactory(Factory): def __init__(self): self.users = {} # maps user names to Chat instances def buildProtocol(self, addr): return Chat(self.users) if __name__ == '__main__': reactor.listenTCP(8123, ChatFactory()) reactor.run()telnet下的运行结果:
上述是聊天记录,第三个用户是中途加入的,只能接受到加入后的聊天记录。
(4)Twisted框架中对文件的操作
工厂有startFactory和stopFactory两种方式来执行相关应用的创建与销毁,下述代码从客户端接收到的信息都会被写入文件中。
#!/usr/bin/env python # coding: utf-8 from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver from twisted.internet import reactor class LoggingProtocol(LineReceiver): def lineReceived(self, line): self.factory.fp.write(line+'\n') self.factory.fp.flush() class LogfileFactory(Factory): protocol = LoggingProtocol def __init__(self, fileName): self.file = fileName def startFactory(self): self.fp = open(self.file, 'a') def stopFactory(self): self.fp.close() if __name__ == '__main__': # 创建监听端口 FACTORY = LogfileFactory("/tmp/log.file") reactor.listenTCP(8007, FACTORY) # 开始监听事件 reactor.run()
模拟ssh协议,服务器端:
import socket import os import commands HOST = '127.0.0.1' PORT = 50007 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((HOST, PORT)) s.listen(1) while True: conn, addr = s.accept() print 'connected by', addr while True: data = conn.recv(1024) if not data: break; print 'command reveived from:', addr, data #cmd_result=os.popen('data;echo $?').read() status, cmd_result = commands.getstatusoutput(data) if len(cmd_result.strip()) != 0: conn.sendall(cmd_result) else: conn.sendall("DONE") conn.close()
客户端:
import socket import time HOST='127.0.0.1' PORT=50007 s=socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((HOST, PORT)) while True: cmd = raw_input("your command:").strip() if len(cmd) == 0 : continue s.sendall(cmd) data=s.recv(8096) print 'client reveived:', data s.close()
类似FTP软件的发送命令和传递文件,服务器:
#!/usr/bin/python #-*- coding: utf-8 -*- import SocketServer import commands import time class MyTCPHandler(SocketServer.BaseRequestHandler): #并发 def handle(self): while True: cmd_result = '' self.data = self.request.recv(1024).strip() if not self.data: #客户端离开 print "client:%s leave!" % self.client_address[0] break user_input = self.data.strip().split() if user_input[0] == 'get': with open(user_input[1], 'rb') as f: self.request.sendall(f.read()) #发送文件 time.sleep(0.5) #sleep一段时间 self.request.send("FILETRANSFERDONE") #发送文件结束标志 continue print "%s wrote:%s" % (self.client_address[0], self.data) status, cmd_result = commands.getstatusoutput(self.data) #处理命令 if len(cmd_result.strip()) != 0: self.request.sendall(cmd_result) else: self.request.sendall('Done') if __name__ == "__main__": HOST, PORT = "localhost", 9999 server = SocketServer.ThreadingTCPServer((HOST, PORT), MyTCPHandler) server.serve_forever()类似FTP软件的发送命令和传递文件,客户端:
import socket import time HOST='127.0.0.1' PORT=9999 s=socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((HOST, PORT)) while True: cmd = raw_input("your command:").strip() if len(cmd) == 0: continue s.sendall(cmd) if (cmd == 'quit' or cmd == 'exit'): break; if cmd.split()[0] == 'get': #transfer file with open(cmd.split()[1].split("/")[-1], 'wb') as f: #use os.path.basename while True: data = s.recv(1024) #if not data: break #different from server if data == 'FILETRANSFERDONE': break f.write(data) continue else: #send cmd or data data=s.recv(8096) print 'client reveived:', data s.close()
#!/usr/bin/env python # -*- coding:utf-8 -*- import socket import threading import SocketServer def client(ip, port, message): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((ip, port)) try: sock.sendall(message) response = sock.recv(1024) print "Received: {0}".format(response) finally: sock.close() if __name__ == "__main__": # Port 0 means to select an arbitrary unused port HOST, PORT = "localhost", 0 th1 = threading.Thread(target=client, args=(ip, port, "Hello World 1",)) th2 = threading.Thread(target=client, args=(ip, port, "Hello World 2",)) th3 = threading.Thread(target=client, args=(ip, port, "Hello World 3",)) th1.start() th2.start() th3.start() th1.join() th2.join() th3.join()