Windows Sockets网络编程（4）套接字重叠IO模型
摘要：上一篇文章《Windows Sockets网络编程（3）WSAEventSelect模型开发》事件通知的Select模型，较之该文《Windows Sockets网络编程（1）TCP select & thread》中单纯的select模型有了很大的改进，其中一个最大的优点就是解决了Select不能被用户主动触发的问题。但是，还是存在不少缺陷。试想这样的情景：一般网络通信的这样的，①首先网卡收到数据，②然后Socket阻塞事件被触发，③接着开始读取网卡数据，④读取完毕开始使用相关数据。其中，将数据从网卡读取到内存中的操作，又称作IO操作。这种操作一般是耗时的。在学习《操作系统》课程中，学习过一种叫做“DMA的直接内存访问机制”，这种机制主要是将IO数据直接送往内存中某处，而基本不需要CPU干预，当IO操作完毕时再通知CPU。这是一个操作系统领域的突破性进展，极大的解放了CPU的工作强度。本文即将介绍的套接字重叠IO模型，是一种几乎不需要CPU参与，就能将数据从网卡输送到内存相应位置的技术，它主要有两种实现方式：事件通知和完成例程。
目录：
-----------------------------------------------
- 事件通知
- WSARecv与LPWSAOVERLAPPED
- 立即数据与异步问题
- WSAGetOverlappedResult函数
- 绑定SOCKET与EVENT
- 完成例程
- 何为APC函数？
- 完成例程的原型
- 线程可警告状态SleepEx
- 实践1：事件通知模型
- 实践2：完成例程模型
- 实践3：TCPClient
1.事件通知
〇 WSARecv与LPWSAOVERLAPPED
事件通知技术的故事要从WSARecv函数说起，这里只关注事件通知是如何完成的，而不去关注其他细节。观察一下函数原型，
WSARecv(
_In_ SOCKETs,
_In_reads_(dwBufferCount)__out_data_source(NETWORK)LPWSABUFlpBuffers,
_In_ DWORDdwBufferCount,
_Out_opt_ LPDWORDlpNumberOfBytesRecvd,
_Inout_ LPDWORDlpFlags,
_Inout_opt_ LPWSAOVERLAPPED lpOverlapped,
_In_opt_ LPWSAOVERLAPPED_COMPLETION_ROUTINElpCompletionRoutine
);
从上述原型中可以看到倒数第二个参数LPWSAOVERLAPPED，这并不是一个系统变量类型，而是一个结构体，原型如下，
typedef struct _OVERLAPPED {
ULONG_PTR Internal;
ULONG_PTR InternalHigh;
union {
struct {
DWORD Offset;
DWORD OffsetHigh;
} DUMMYSTRUCTNAME;
PVOID Pointer;
} DUMMYUNIONNAME;
HANDLE hEvent;
} OVERLAPPED, *LPOVERLAPPED;同样，这里只关注与事件相关的最重要的参数HANDLE hEvent。文章讲到这里，基本上已经梳理清楚了事件通知模型的条理了——就是WSACreateEvent创建一个事件，将其绑定在结构体中的hEvent当中，然后将结构体整体作为WSARecv的一个参数。这样，当“网卡数据传输到内存指定位置时”，hEvent事件就会被激发（又叫做已触发状态），这时候直接去读取内存数据就可以了。
〇 立即数据与异步问题
那么，问题来了。WSARecv是非阻塞函数，数据要如何读取呢？
那么要继续分析一下这个函数，①非阻塞函数也是可以返回读取到的网卡数据的。②如果一时间获取不到网卡数据该怎么办？
针对问题1：WSARecv函数直接获取网卡数据。
这个毫无疑问，需要获取的数据量极其小，能在WSARecv被调用的一瞬间完成。那么，此时WSARecv函数返回值将为0，同时参数lpNumberOfBytesRecvd将被置为获取到的字节数。
针对问题2：一时间获取不到网卡数据。
WSARecv函数为非阻塞函数，它不会傻傻的等待网卡数据运输到内存。这时候，WSARecv函数会返回SOCKET_ERROR，一旦检测到该返回值，应该马上调用WSAGetLastError()获取此时的错误码——ERROR_IO_PENDING（一般是这个错误码，它表示recv操作正在异步执行中）。至于何时获取完毕？这就是上文说道的Event机制了。
BOOL EventNotification::recvAsynData()
{
DWORD recv_length = 0L, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_op_type = RECV_FLAG;
m_io.hEvent = m_we;
m_wsa_recv_buf.buf = m_recv_buffer;
m_wsa_recv_buf.len = sizeof(m_recv_buffer) / sizeof(char);
if (SOCKET_ERROR == WSARecv(m_s, &m_wsa_recv_buf, 1, &recv_length, &flag, &m_io, NULL)){
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
上述代码，基本上就是对前文描述的实现。至于WSARecv最后一个参数lpCompletionRoutine为什么置空，从参数名字也可以猜到，它的作用是“完成例程”，它是套接字重叠IO模型的另外一种实现方式，此处谈论的是“事件通知”方式。
〇 WSAGetOverlappedResult函数
立即完成的WSARecv可以直接读取到数据，那么事件通知的数据该如何获得呢？一个新的函数WSAGetOverlappedResult将被介绍，
WSAGetOverlappedResult(
_In_ SOCKET s,
_In_ LPWSAOVERLAPPED lpOverlapped,
_Out_ LPDWORD lpcbTransfer,
_In_ BOOL fWait,
_Out_ LPDWORD lpdwFlags
);
s：发起重叠操作的套接字；
lpOverlapped：发起重叠操作的LPWSAOVERLAPPED结构体指针（里面有Event还记得吗？）；
lpcbTransfer：实际发送或者接受到的字节数（和send()/recv()返回值有些像了）；
fWait：函数返回方式。当为TRUE时，该函数直到重叠操作完成才返回。当为FALSE时，如果网卡数据尚未被输送到内存或过程中发生了意外，函数将返回FALSE，这时候如果立即获取错误码将会是——WSA_IO_INCOMPLETE；
lpdwFlags：完成状态的附加标志。
也就是说，只有当WSAGetOverlappedResult函数的返回值不为FALSE，同时lpcbTransfer字节长度不为0时，才表示一次成功的操作。
BOOL ret = WSAGetOverlappedResult(eNty->m_s, &eNty->m_io, &length, TRUE, &flags);
if (ret == FALSE || length == 0){
eraseNode(i);
}
〇 绑定SOCKET与EVENT
这里还有一个事情没有提到，就是“绑定每个SOCKET和EVENT”，这个操作十分重要。最简单的方式就是直接将两者进行一种一一对应的映射关系。（可以简单用两个数组来表示，这样的话下标为X处的EVENT被触发时就表示下标为X的SOCKET有数据到来。）
typedef structEventNtf
{
WSAEVENT eventArray[WSA_MAXIMUM_WAIT_EVENTS]; //WSA_MAXIMUM_WAIT_EVENTS = 64
EventNotification* ioArray[WSA_MAXIMUM_WAIT_EVENTS];
unsignedint endNullIndex = 0;
} EVENTNTY;
2.完成例程
〇 何为APC函数？
完成例程是Windows Sockets提供的另外一种管理完成的重叠IO方法。完成例程其实就是一个APC函数，当发起重叠IO操作时，将该函数传递给发起操作的函数，当重叠IO完成时且线程处于可警告状态时，将由系统调用完成例程这个APC函数。通俗一点说，完成例程其实就是一个APC函数，被挂载在线程的APC队列上，当线程处于可警告状态时，线程会自动调用APC队列，执行里面的所有函数。对于这段话觉得很难理解，或者不知道什么是APC函数，什么是线程可警告状态的可以先去阅读《Windows APC机制 & 可警告alertable的线程等待状态》一文。
完成例程的WSARecv函数和事件通知模型有何不同呢？不同处主要有两点：①hEvent被废弃，不再作为事件通知绑定；②WSARecv最后一个参数不再为NULL，而是要传入一个完成例程APC函数。（这里对于hEvent要注意，虽然被废弃，但是此处一般用于传输上下文（context）“如下述源码”，以便在完成例程APC函数被触发时，能知道是谁完成了IO例程）
BOOL CompletionRoutine::recvAsynData()
{
DWORD recv_length = 0L, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_io.hEvent = WSAEVENT(this); //this
m_wsa_recv_buf.buf = m_recv_buffer;
m_wsa_recv_buf.len = sizeof(m_recv_buffer) / sizeof(char);
m_op_type = RECV_FLAG;
if (SOCKET_ERROR == WSARecv(m_s, &m_wsa_recv_buf, 1, &recv_length, &flag,
&m_io, ioRoutine)){//ioRoutine
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
〇 完成例程的原型
那么，完成例程APC函数到底是什么样子呢？如果你阅读了上面推荐的那篇文章，相信你已经知道了，就是这个样子。
void CALLBACK CompletionRoutine::ioRoutine(DWORD err, DWORD length, LPWSAOVERLAPPED overlapped, DWORD flags)
{
if (err != 0 || length == 0){
//err
}
CompletionRoutine* cr = (CompletionRoutine*)overlapped->hEvent;
switch (cr->m_op_type)
{
case RECV_FLAG:
cr->m_recv_buffer[length] = '\0';
printf("recv[%s]\n", cr->m_recv_buffer);
strcpy_s(cr->m_send_buffer, CompletionRoutine::SEND_BUFFER, "hello completion routine.");
cr->sendAsynData();
break;
case SEND_FLAG: break;
default:
break;
}
}
〇 线程可警告状态SleepEx
对于完成例程IO一定要注意，这种实现机理是利用线程在可警告状态主动调用APC函数来实现的，而accept并不能使线程进入可警告状态，就如同Sleep函数一样，也不能。所以，为了使完成例程一定能被触发，SleepEx(1,TRUE)必不可少。对于为什么这里用SleepEx有兴趣的可以去阅读上面推荐的那篇关于APC函数的文章。
while (TRUE){
SOCKET sAccept = INVALID_SOCKET;
if ((sAccept = accept(sListen, NULL, NULL)) == INVALID_SOCKET){
break;
}
CompletionRoutine* cr = new CompletionRoutine(sAccept);
if (FALSE == cr->recvAsynData()){
eraseNode(global.endNullIndex);
continue;
}
global.ioArray[global.endNullIndex] = cr;
InterlockedIncrement(&global.endNullIndex);//++classTotal
SleepEx(1, TRUE);
}
实践1：事件通知模型
//EventNotification.h
#pragma once
#include
#pragma comment(lib,"ws2_32.lib")
class EventNotification
{
public:
enum{ RECV_BUFFER = 1024, SEND_BUFFER = 1024 };
EventNotification(SOCKET s, WSAEVENT we);
virtual ~EventNotification();
public:
public:
BOOL recvAsynData();
BOOL sendAsynData();
public:
SOCKET m_s;
WSAOVERLAPPED m_io;
WSAEVENT m_we;
char m_recv_buffer[RECV_BUFFER];
char m_send_buffer[SEND_BUFFER];
WSABUF m_wsa_recv_buf;
WSABUF m_wsa_send_buf;
int m_op_type;
};
//EventNotification.cpp
#include
#include "EventNotification.h"
HANDLE hThread;
DWORD WINAPI serviceThread(LPVOID context);
enum { RECV_FLAG, SEND_FLAG, ACCEPT_FLAG };
/*
Date |Change
-----------------------------------------------------------
2017-7-27 |关联Event与EventNotification类，当事件发生时便于查找对象
*/
typedef struct EventNtf
{
WSAEVENT eventArray[WSA_MAXIMUM_WAIT_EVENTS];
EventNotification* ioArray[WSA_MAXIMUM_WAIT_EVENTS];
unsigned int endNullIndex = 0;
} EVENTNTY;
EVENTNTY global;
EventNotification::EventNotification(SOCKET s, WSAEVENT we)
{
m_s = s;
m_we = we;
}
EventNotification::~EventNotification()
{
closesocket(m_s);
}
BOOL EventNotification::recvAsynData()
{
DWORD recv_length = 0L, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_op_type = RECV_FLAG;
m_io.hEvent = m_we;
m_wsa_recv_buf.buf = m_recv_buffer;
m_wsa_recv_buf.len = sizeof(m_recv_buffer) / sizeof(char);
if (SOCKET_ERROR == WSARecv(m_s, &m_wsa_recv_buf, 1, &recv_length, &flag, &m_io, NULL)){
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
BOOL EventNotification::sendAsynData()
{
DWORD send_length, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_op_type = SEND_FLAG;
m_io.hEvent = m_we;
m_wsa_send_buf.buf = m_send_buffer;
m_wsa_send_buf.len = strlen(m_wsa_send_buf.buf);
if (SOCKET_ERROR == WSASend(m_s, &m_wsa_send_buf, 1, &send_length, flag, &m_io, NULL)){
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
EventNotification* getOverlappingIO(unsigned int index){
return global.ioArray[index];
}
void eraseNode(unsigned int index)
{
if (global.ioArray[index]){
delete global.ioArray[index];
global.ioArray[index] = NULL;
}
for (unsigned int i = index; i < global.endNullIndex; ++i){
global.eventArray[i] = global.eventArray[i + 1];
global.ioArray[i] = global.ioArray[i + 1];
}
InterlockedDecrement(&global.endNullIndex);//--eventTotal
}
void eraseAllNode()
{
for (unsigned int i = 0; i < global.endNullIndex; ++i){
if (global.ioArray[i]){
delete global.ioArray[i];
global.ioArray[i] = NULL;
}
}
InterlockedExchange(&global.endNullIndex, 0);//eventTotal=0
}
void startListener(unsigned int port){
WSADATA wsaData;
if (WSAStartup(0x0202, &wsaData) != 0){
return;
}
SOCKET sListen = INVALID_SOCKET;
if ((sListen = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET){
WSACleanup();
return;
}
SOCKADDR_IN sin;
sin.sin_family = AF_INET;
sin.sin_addr.S_un.S_addr = htonl(INADDR_ANY);
sin.sin_port = htons(port);
if (bind(sListen, (SOCKADDR*)&sin, sizeof(sin)) == SOCKET_ERROR){
closesocket(sListen);
WSACleanup();
return;
}
if (listen(sListen, SOMAXCONN)){
closesocket(sListen);
WSACleanup();
return;
}
if ((global.eventArray[global.endNullIndex] = WSACreateEvent()) == WSA_INVALID_EVENT){
return;
}
EventNotification* eNty = new EventNotification(sListen, global.eventArray[global.endNullIndex]);
ZeroMemory(&eNty->m_io, sizeof(eNty->m_io));
eNty->m_op_type = ACCEPT_FLAG;
eNty->m_io.hEvent = global.eventArray[global.endNullIndex];
global.ioArray[global.endNullIndex] = eNty;
InterlockedIncrement(&global.endNullIndex);
hThread = CreateThread(NULL, 0, serviceThread, NULL, 0, NULL);
while (TRUE){
SOCKET sAccept = INVALID_SOCKET;
if ((sAccept = accept(sListen, NULL, NULL)) == INVALID_SOCKET){
break;
}
if (global.endNullIndex >= WSA_MAXIMUM_WAIT_EVENTS){
break;
}
if ((global.eventArray[global.endNullIndex] = WSACreateEvent()) == WSA_INVALID_EVENT){
break;
}
eNty = new EventNotification(sAccept, global.eventArray[global.endNullIndex]);
global.ioArray[global.endNullIndex] = eNty;
if (FALSE == eNty->recvAsynData()){
eraseNode(global.endNullIndex);
continue;
}
InterlockedIncrement(&global.endNullIndex);//++eventTotal
WSASetEvent(global.eventArray[0]);//notify accept, rebuild event...
}
eraseAllNode();
WSACleanup();
}
DWORD WINAPI serviceThread(LPVOID context)
{
DWORD index = 0L, flags, length;
while (TRUE){
if ((index = WSAWaitForMultipleEvents(global.endNullIndex, global.eventArray, FALSE,
WSA_INFINITE, FALSE)) == WSA_WAIT_FAILED){
break;
}
for (DWORD i = index; i < global.endNullIndex; ++i){
index = WSAWaitForMultipleEvents(1, &global.eventArray[i], TRUE, 0L, FALSE);
if (index == WSA_WAIT_FAILED || index == WSA_WAIT_TIMEOUT){
continue;
}
WSAResetEvent(global.eventArray[i]);
EventNotification* eNty = getOverlappingIO(i);
if (eNty->m_op_type == ACCEPT_FLAG){
continue;
}
BOOL ret = WSAGetOverlappedResult(eNty->m_s, &eNty->m_io, &length, TRUE, &flags);
if (ret == FALSE || length == 0){
eraseNode(i);
}
else{
switch (eNty->m_op_type){
case RECV_FLAG:
eNty->m_recv_buffer[length] = '\0';
printf("recv:[%s]\n",eNty->m_recv_buffer);
strcpy_s(eNty->m_send_buffer, EventNotification::SEND_BUFFER, "recv ack.");
eNty->sendAsynData();
eNty->recvAsynData();
break;
case SEND_FLAG:
printf("send success!\n");
break;
default: break;
}
}
}
}
return 0L;
}
int main(int argc, char* argv[])
{
printf("overlapping eNty.\n");
startListener(8086);
return 0;
}
实践2：完成例程模型
//CompletionRoutine.h
#pragma once
#include
#pragma comment(lib,"ws2_32.lib")
class CompletionRoutine
{
public:
CompletionRoutine(SOCKET s);
virtual ~CompletionRoutine();
enum{ RECV_BUFFER = 1024, SEND_BUFFER = 1024 };
public:
public:
BOOL recvAsynData();
BOOL sendAsynData();
static void CALLBACK ioRoutine(DWORD err, DWORD length, LPWSAOVERLAPPED overlapped, DWORD flags);
public:
SOCKET m_s;
WSAOVERLAPPED m_io;
char m_recv_buffer[RECV_BUFFER];
char m_send_buffer[SEND_BUFFER];
WSABUF m_wsa_recv_buf;
WSABUF m_wsa_send_buf;
int m_op_type;
};
//CompletionRoutine.cpp
#include
#include "CompletionRoutine.h"
enum { RECV_FLAG, SEND_FLAG };
typedef struct ComplRoutine
{
CompletionRoutine* ioArray[WSA_MAXIMUM_WAIT_EVENTS];
unsigned int endNullIndex = 0;
} COMPLROUTINE;
COMPLROUTINE global;
CompletionRoutine::CompletionRoutine(SOCKET s)
{
m_s = s;
}
CompletionRoutine::~CompletionRoutine()
{
closesocket(m_s);
}
BOOL CompletionRoutine::recvAsynData()
{
DWORD recv_length = 0L, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_io.hEvent = WSAEVENT(this); //this
m_wsa_recv_buf.buf = m_recv_buffer;
m_wsa_recv_buf.len = sizeof(m_recv_buffer) / sizeof(char);
m_op_type = RECV_FLAG;
if (SOCKET_ERROR == WSARecv(m_s, &m_wsa_recv_buf, 1, &recv_length, &flag,
&m_io, ioRoutine)){//ioRoutine
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
BOOL CompletionRoutine::sendAsynData()
{
DWORD send_length, flag = 0L;
ZeroMemory(&m_io, sizeof(m_io));
m_io.hEvent = WSAEVENT(this); //this
m_wsa_send_buf.buf = m_send_buffer;
m_wsa_send_buf.len = strlen(m_wsa_send_buf.buf);
m_op_type = SEND_FLAG;
if (SOCKET_ERROR == WSASend(m_s, &m_wsa_send_buf, 1, &send_length, flag,
&m_io, ioRoutine)){//ioRoutine
if (WSAGetLastError() != ERROR_IO_PENDING){
return FALSE;
}
}
return TRUE;
}
void CALLBACK CompletionRoutine::ioRoutine(DWORD err, DWORD length, LPWSAOVERLAPPED overlapped, DWORD flags)
{
if (err != 0 || length == 0){
//err
}
CompletionRoutine* cr = (CompletionRoutine*)overlapped->hEvent;
switch (cr->m_op_type)
{
case RECV_FLAG:
cr->m_recv_buffer[length] = '\0';
printf("recv[%s]\n", cr->m_recv_buffer);
strcpy_s(cr->m_send_buffer, CompletionRoutine::SEND_BUFFER, "hello completion routine.");
cr->sendAsynData();
break;
case SEND_FLAG: break;
default:
break;
}
}
void eraseNode(unsigned int index)
{
if (global.ioArray[index]){
delete global.ioArray[index];
global.ioArray[index] = NULL;
}
for (unsigned int i = index; i < global.endNullIndex; ++i){
global.ioArray[i] = global.ioArray[i + 1];
}
InterlockedDecrement(&global.endNullIndex);//--classTotal
}
void eraseAllNode()
{
for (unsigned int i = 0; i < global.endNullIndex; ++i){
if (global.ioArray[i]){
delete global.ioArray[i];
global.ioArray[i] = NULL;
}
}
InterlockedExchange(&global.endNullIndex, 0);//classTotal=0
}
void startListener(unsigned int port){
WSADATA wsaData;
if (WSAStartup(0x0202, &wsaData) != 0){
return;
}
SOCKET sListen = INVALID_SOCKET;
if ((sListen = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET){
WSACleanup();
return;
}
SOCKADDR_IN sin;
sin.sin_family = AF_INET;
sin.sin_addr.S_un.S_addr = htonl(INADDR_ANY);
sin.sin_port = htons(port);
if (bind(sListen, (SOCKADDR*)&sin, sizeof(sin)) == SOCKET_ERROR){
closesocket(sListen);
WSACleanup();
return;
}
if (listen(sListen, SOMAXCONN)){
closesocket(sListen);
WSACleanup();
return;
}
while (TRUE){
SOCKET sAccept = INVALID_SOCKET;
if ((sAccept = accept(sListen, NULL, NULL)) == INVALID_SOCKET){
break;
}
CompletionRoutine* cr = new CompletionRoutine(sAccept);
if (FALSE == cr->recvAsynData()){
eraseNode(global.endNullIndex);
continue;
}
global.ioArray[global.endNullIndex] = cr;
InterlockedIncrement(&global.endNullIndex);//++classTotal
SleepEx(1, TRUE);
}
eraseAllNode();
WSACleanup();
}
int main(int argc, char* argv[])
{
printf("completion routine cr.\n");
startListener(8086);
return 0;
}
实践3：TCPClient
//TCPClient.cpp
#include
#pragma comment(lib,"ws2_32.lib")
#include
/*
Date |Change
-----------------------------------------
2017-7-27 |SOCKET TCP测试客户端
*/
void tcp_client()
{
SOCKET sClient = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
sockaddr_in sin;
sin.sin_family = AF_INET;
sin.sin_port = htons(8086);
sin.sin_addr.S_un.S_addr = inet_addr("127.0.0.1");
if (connect(sClient, (sockaddr *)&sin, sizeof(sin)) == SOCKET_ERROR){
closesocket(sClient);
return;
}
char buffer[1024];
sprintf_s(buffer, 1024, "hi overlapping message.(%d)", sClient);
//Sleep(1000);
int ret = send(sClient, buffer, strlen(buffer), 0);
ret = recv(sClient, buffer, sizeof(buffer), 0);
buffer[ret] = '\0';
printf("%s\n", buffer);
closesocket(sClient);
}
int main(int argc, char* argv[])
{
printf("tcp client.\n");
WSADATA wsa;
WSAStartup(MAKEWORD(2, 2), &wsa);
int i = 5;
while (i--){
tcp_client();
Sleep(2000);
}
WSACleanup();
return 0;
}
参考文献：
[1] 孙海民.精通Windows Sockets网络开发——基于Visual C++实现[M]. 北京:人民邮电出版社, 2008. 327-340
@qingdujun
2017-7-28 in Xi'An