Redis数据库使用总结

Redis是一个高效的内存数据库， 根据多方面了解，
我们计划选用Redis数据库作为CSMD3.0的数据存储方案

欢迎加入QQ群:CTP/易盛(行情交易)开发

测试环境搭建:

centos 安装redis:

#yum install redis

启动redis服务:

#service redis start

使用Redis客户端测试Redis服务器工作是否正常：

#redis-cli
127.0.0.1:6379> set string1 "CCCCCCCCCC"
OK
127.0.0.1:6379> get string1
"CCCCCCCCCC"
127.0.0.1:6379>

把字符串"CCCCCCCCCC" 写入 string1, 形成 string1 ==> "CCCCCCCCCC" 的键值对(key-value),
可以判断Redis服务器工作正常。

安装官方C++编程接口hiredis:

yum install hiredis-devel.x86_64

在线保存CTP数据，或者离线存储深度行情数据到Redis:

//写数据到Redis
int insert_to_redis(char* key, LXTick* value, CThostFtdcDepthMarketDataField* pDepthMarketData)
{
char score[128];
snprintf(score, sizeof(score)-1, "%ld", value->ticktimestamp);
redisReply* reply = (redisReply*)redisCommand(redis, "zadd %s %s %b", key, score, value, sizeof(LXTick));
freeReplyObject(reply);
return 0;
}

我们使用python程序从Cassandra数据库取出两年数据存入Redis数据库，用于测试。

安装apache2:

yum install httpd

安装php5:

yum install php5

安装phpredis:

yum install php5-redis

首先测试php是否功能正常：

#vi info.php
phpinfo();
?>

然后在shell执行: php info.php 看输出是否正常, 如果正常代表php已经工作。

接下来把 info.php 复制到 /var/www/html/
然后浏览器打开 http://192.168.1.207:8080/info.php
如果正常显示php和apache信息代表 apache与php均工作正常。

列出所有key getkeys.php (http://192.168.1.207:8080/getkeys.php?key=2015&db=2):

$redis = new Redis();
$redis->connect('192.168.1.207', 6379);
$redis->select($_GET["db"]);
$ticks = $redis->keys($_GET["key"]);
foreach($ticks as $i => $tick)
{
echo "$tick ";
echo $redis->zcount($tick, 0, 99999999999999999); //显示所有key
echo "
";
}
?>

点击某合约获取数据 getdata1.php (http://192.168.1.207:8080/getdata1.php?cc=CZCE:RM:RM509:20150402):

$redis = new Redis();
$redis->connect('192.168.1.207', 6379);
$redis->select(2);
$ticks = $redis->zrange($_GET["cc"], 0, -1);
foreach($ticks as $i => $tick)
{
echo $tick;

}
?>

使用python测试获取一个合约一个月内所有Tick数据,
以下测试结果的前一项是nginx+文件, 后一项是 apache + redis

nginx和apache2均启用gzip压缩的测试结果:
总文件大小 耗时 速率(解压后文件体积/时间)
49093660kB, 0.550s 679.773Mbps (win2008 + nginx + 文件)
97384056kB, 1.563s 475.355Mbps (centos + apache + phpredis + redis)

均不启用gzip压缩的测试结果:
49093660KB, 4.1779999733s 89.6493306647Mbps (win2008 + nginx + 文件)
97384056kB, 8.61899995804s 86.2027367322Mbps (centos + apache + phpredis + redis)

由上表可见, 在不启用压缩条件下php + redis速率基本与nginx+文件方案相当，

附python测试脚本:

#--encoding=utf-8--
import requests
from Queue import Queue
from time import time
from threading import Thread
class HttpThread(Thread):
def __init__(self,url,queue,session):
Thread.__init__(self)
self.url=url
self.queue=queue
self.session=session
def run(self):
rep=self.session.get(self.url)
self.queue.put(rep.content)

def multithread_download(baseurl,batchparams,session):
st=time()
threads = []
downloadq=Queue()
for p in batchparams:
t=HttpThread(baseurl %p,downloadq,session)
t.start()
threads.append(t)
for t in threads:
t.join()
size=0
while not downloadq.empty():
t=downloadq.get()
size+=len(t)
tt=time()-st
print 'Http download data,Total size:%sKB,total time:%s '%(size,tt),'speed:%sMbps' %str(size8/(10241024*tt))

def test_http_download():
s=requests.Session()
s.get("http://192.168.1.206:8081/")
# s.headers['Accept-Encoding']='gzip,deflate'
multithread_download("http://192.168.1.206:8081/CTP/CFFEX/IF/IF1606/2016-05-%s/IF1606-P-tick-2016-05-%s.dat",
[(i,i) for i in ['03','04','05','06','09','10','11','12','13','16','17','18','19','20','23','24','25','26','27','30','31']], s)

def test_redishttp_download():
from cStringIO import StringIO
from lxml import etree
from lxml.cssselect import CSSSelector
s=requests.Session()
s.get("http://192.168.1.207:8080/")
# s.headers['Accept-Encoding']='deflate'

>rep=requests.get("http://192.168.1.207:8080/getkeys.php?key=CFFEX:IF:IF1310*&db=2").content
tree=etree.parse(StringIO(rep), etree.HTMLParser())
links=[e.attrib["href"] for e in CSSSelector("a")(tree)]
multithread_download("http://192.168.1.207:8080/%s", links, s)

test_http_download()
test_redishttp_download()