总之网站上说得很好啦:http://msgpack.org/,又快,生成的文件又小等等,总之就是很牛逼的救世主咯
因为忍受不了 pickle 的速度,所以尝试切换一下。
import pickle
import msgpack
import gc
a = list(range(10000*10000))
def dump1():
with open('1', 'wb') as f:
pickle.dump(a, f)
def dump2():
with open('2', 'wb') as f:
msgpack.dump(a, f)
def dump3():
gc.disable()
with open('2', 'wb') as f:
msgpack.dump(a, f)
gc.enable()
然后测速,有 dump3
的原因是官方网站说关闭gc有奇效:
CPython's GC starts when growing allocated object. This means unpacking may cause useless GC. You can use gc.disable() when unpacking large message.
恩,先看测试结果(测试了10次):
dump1: 1 loop, best of 3: 3.98 s per loop
dump2: 1 loop, best of 10: 5.22 s per loop
dump3: 1 loop, best of 10: 5.13 s per loop
恩,彻底打脸了。关闭gc也没有奇效。
然后看dump出来的文件 (=,=b):
-rw-r--r-- 1 chendx staff 477M 8 11 18:43 1
-rw-r--r-- 1 chendx staff 477M 8 11 18:49 2
恩,浪费我的时间……
然后不死心
def load1():
with open('1', 'rb') as f:
pickle.load(f)
def load2():
with open('2', 'rb') as f:
msgpack.load(f)
结果(还是10次)
load1: 1 loop, best of 10: 6.78 s per loop
load2: 1 loop, best of 10: 5.91 s per loop
恩,真是鸡肋啊