python神器-文本文件转结构化数据-Kummer话你知

用途

用户将文本文件解析为结构化的内容
https://github.com/google/textfsm/wiki/TextFSM

案例研究

比如将linux中进程内存的smaps解析为csv
每一块内存为

7fc84255c000-7fc842560000 rw-s 100573000 00:06 383                       /dev/dri/card0
Size:                 16 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
VmFlags: rd wr sh mr mw me ms io de dd sd mm 

模板例子

要解析该文本，先完成一个解析模板，一个解析模板的例子如下：

# define values
Value Filldown ProcessName (.+)
Value Begin ([0-9a-fA-F]+)
Value End ([0-9a-fA-F]+)
Value Size (\d+)
Value KernelPageSize (\d+)
Value MMUPageSize (\d+)
Value Rss (\d+)
Value Pss (\d+)
Value Shared_Clean (\d+)
Value Shared_Dirty (\d+)
Value Private_Clean (\d+)
Value Private_Dirty (\d+)
Value Referenced (\d+)
Value Anonymous (\d+)
Value LazyFree (\d+)
Value AnonHugePages (\d+)
Value ShmemPmdMapped (\d+)
Value Shared_Hugetlb (\d+)
Value Private_Hugetlb (\d+)
Value Swap (\d+)
Value SwapPss (\d+)
Value Locked (\d+)
Value VmFlags ((\w|\s)+)

# Allway starts in 'Start' state.
Start
  ^@@@@@${ProcessName}@@@@@ -> Record Item

# A state transition was not strictly necessary but helpful for the example.
Item
  ^${Begin}\-${End}
  ^MMUPageSize:\s+${MMUPageSize} kB
  ^Rss:\s+${Rss} kB
  ^Pss:\s+${Pss} kB
  ^Shared_Clean:\s+${Shared_Clean} kB
  ^Shared_Dirty:\s+${Shared_Dirty} kB
  ^Private_Clean:\s+${Private_Clean} kB
  ^Private_Dirty:\s+${Private_Dirty} kB
  ^Referenced:\s+${Referenced} kB
  ^Anonymous:\s+${Anonymous} kB
  ^LazyFree:\s+${LazyFree} kB
  ^AnonHugePages:\s+${AnonHugePages} kB
  ^ShmemPmdMapped:\s+${ShmemPmdMapped} kB
  ^Shared_Hugetlb:\s+${Shared_Hugetlb} kB
  ^Private_Hugetlb:\s+${Private_Hugetlb} kB
  ^Swap:\s+${Swap} kB
  ^SwapPss:\s+${SwapPss} kB
  ^Locked:\s+${Locked} kB
  ^VmFlags:\s*${VmFlags} -> Record Item
  

# An implicit EOF state outputs the last record.

模板-组成部分1：Value定义

该模板分为两个部分，第一部分是定义表中有哪些字段。
定义一个字段，三个部分：
①以Value开头，
②一些选项，
③字段名称
④字段匹配的正则表达式

image.png

# define values
Value Filldown ProcessName (.+)
Value Begin ([0-9a-fA-F]+)
Value End ([0-9a-fA-F]+)
Value Size (\d+)
Value KernelPageSize (\d+)
Value MMUPageSize (\d+)
Value Rss (\d+)
Value Pss (\d+)
Value Shared_Clean (\d+)
Value Shared_Dirty (\d+)
Value Private_Clean (\d+)
Value Private_Dirty (\d+)
Value Referenced (\d+)
Value Anonymous (\d+)
Value LazyFree (\d+)
Value AnonHugePages (\d+)
Value ShmemPmdMapped (\d+)
Value Shared_Hugetlb (\d+)
Value Private_Hugetlb (\d+)
Value Swap (\d+)
Value SwapPss (\d+)
Value Locked (\d+)
Value VmFlags ((\w|\s)+)

模板-组成部分2：FSM

模板第二部分是一个FSM（有限自动状态机）
①状态名称
②每一个状态下有若干条状态规则，状态规则有一个表达式组成，
③每一条规则可以填充一个或多个Value字段。填充采用${字段名称}方式
④规则后面可以跟一个->，完成状态跳转和记录操作

# Allway starts in 'Start' state.
Start
  ^@@@@@${ProcessName}@@@@@ -> Record Item

# A state transition was not strictly necessary but helpful for the example.
Item
  ^${Begin}\-${End}
  ^MMUPageSize:\s+${MMUPageSize} kB
  ^Rss:\s+${Rss} kB
  ^Pss:\s+${Pss} kB
  ^Shared_Clean:\s+${Shared_Clean} kB
  ^Shared_Dirty:\s+${Shared_Dirty} kB
  ^Private_Clean:\s+${Private_Clean} kB
  ^Private_Dirty:\s+${Private_Dirty} kB
  ^Referenced:\s+${Referenced} kB
  ^Anonymous:\s+${Anonymous} kB
  ^LazyFree:\s+${LazyFree} kB
  ^AnonHugePages:\s+${AnonHugePages} kB
  ^ShmemPmdMapped:\s+${ShmemPmdMapped} kB
  ^Shared_Hugetlb:\s+${Shared_Hugetlb} kB
  ^Private_Hugetlb:\s+${Private_Hugetlb} kB
  ^Swap:\s+${Swap} kB
  ^SwapPss:\s+${SwapPss} kB
  ^Locked:\s+${Locked} kB
  ^VmFlags:\s*${VmFlags} -> Record Item
  

# An implicit EOF state outputs the last record.

一个Python脚本

import textfsm
import os
from optparse import OptionParser

usage = '''将一个txt文件转换为csv文件'''
par = OptionParser(usage=usage)

par.add_option("-s","--src",action = "store",type = "string",dest = "src")
par.add_option("-d","--dst",action = "store",type = "string",dest = "dst")
par.add_option("-t","--template",action = "store",type = "string",dest = "template")

(opt,args) = par.parse_args()
src = opt.src
dst = opt.dst
tmp = opt.template
if src and dst and tmp and os.path.isfile(src) and os.path.isfile(tmp):
    txt=''
    with open(src,'r+') as f: txt = f.read()
    with open(tmp,'r+') as f: fsm = textfsm.TextFSM(f)
    data = fsm.ParseText(txt)
    w = open(dst,'w+')
    w.write( ', '.join(fsm.header)+'\n' )
    for row in data:
        w.write( ', '.join(row) +'\n')
    w.close()
else:
    print('src or tmplate not exist!')