使用pickle记录过程数据

在使用python进行人工智能训练过程中，往往要保存一些过程数据，方便对比或查看。

这里使用pickle包，将每项过程数据保存成字典的形式，存储在pickle中；

另外为了方便可视化，可将pickle数据写入到txt中。

1. pickle格式数据的读写

import pickle
import os


def save_pkl(path, key, item):
    dic = read_pkl(path)
    dic[key] = dic.get(key, [])+[item]
    file = open(path, 'wb')
    pickle.dump(dic, file)
    file.close()


def read_pkl(path):
    if not os.path.isfile(path):
        return {}
    file = open(path, 'rb')
    dic = pickle.load(file)
    file.close()
    return dic

2. 将pickle格式数据写入txt

def pickle_txt(pkl,txt):
    dic = read_pkl(pkl)
    file = open(txt, 'a')
    for key in dic.keys():
        file.writelines(f"{key}\n")
        ls = [item+'\n' for item in dic[key]]
        file.writelines(ls)
    file.close()

3. 通过修改txt来修改pickle

直接修改pickle不太方便，但是可以通过直观修改txt，再将txt内容回传给pickle，来完成修改。

def txt_pickle(txt,pkl):
    ls=[]
    with open(txt,'r') as f:
        for line in f.readlines():
            ls.append(line.strip())
    f.close()
    dic={}
    sub_ls=[ls[0]]
    for item in ls[1:]:
        if  '/' not in item and sub_ls:
            if len(sub_ls)>1:
                dic[sub_ls[0]]=sub_ls[1:]
            sub_ls=[item]
        elif '/'in item:
            sub_ls.append(item)
    if len(sub_ls)>1:
        dic[sub_ls[0]]=sub_ls[1:]

    if os.path.isfile(pkl):
        os.remove(pkl)
        
    file = open(pkl, 'wb')
    pickle.dump(dic, file)
    file.close()

4. 例子

pkl = 'E:/tem.pkl'
key, item = 'abc', "d/e/f"
save_pkl(pkl, key, item)
key, item = 'abc', "j/k/l"
save_pkl(pkl, key, item)
key, item = 'mno', "p/q/r"
save_pkl(pkl, key, item)

txt="E:/1.txt"
pickle_txt(pkl,txt)

pickle中存放内容：

 {'key1': ['item1', 'item2'], 'key2': ['item3']}

txt中存放内容：

key1
item1
item2
key2
item3