基于词表的切词——最短路径方法

最短路径方法的目的是使得分词后得到的词最少，实现的方法是从句子中匹配出所有在词表中的词，以词为边(边的权重为1)、词与词的间隔(切分点)为节点构造出一个有向无环图(DAG)，有唯一的起点(句子的开始)和唯一的终点(句子的结束)，找到一条最短路径，即切分最少。

 

和正向最大匹配一样最短路径方法只需要一个词表即可进行切词，但得到的结果要更好，主要原因是考虑到了上下文的衔接性，把切分过程从一维扩展到了二维。但是这种方法时间复杂性更高，而且为了保证图的连通性需要在词匹配时要分割到字。

 

最短路径方法可以使用词频方便的扩展。将词频作为边的权重，将2-gram词频作为节点权重，这样可以容易的得到加入了词频信息的最短路径切分。如果有可能的话，可以应用N-Gram词频信息在路径选择上，甚至可以将词法信息加入进来，通过于词法图的匹配提取最佳路径。

 

最短路径路径方法还有助于对专有名词的切分。由于在切分专有名词时，经常出现短词或单字，可以在进行路径选择之前先对这样的边进行识别并赋予较低的权重使得在路径选择时能够倾向于这些边。

 

最短路径算法只能得到一条结果路径，和结果路径权重相同的其它路径都被舍弃了，而且接近最短路径的k最短路径均被舍弃，这样往往会失去正确切分。这些问题可以通过使用k最短路径的其它算法得到解决。

 

下面是一个简单的最短路径法的Python实现：

  1  ''' Implements SPM(Shortest Path Matching) Method
  2  '''
  3 
  4  import  string
  5  import  codecs
  6  import  re
  7 
  8  # {entry1:cateory1, entry2:category2, ..., entryN:categoryM}
  9  dict  =  {}
10  # a string contains delimiting punctuations
11  punc  =   ''
12  max_len  =  0
13 
14  def  segment(str):
15       ''' segment the given string in a method which made the number of tokens after
16      segmentation is minium. The algorithm used here can bu summaried as following:
17      1. use delimiting punctuations to segment the given string into short sentences.
18      2. pick the first sentence
19      3. find all known words in this picked sentence.
20      4. organize all words into DAG
21      5. find the shortest path from the start to the end, which is the segmentation we want
22      6. pick the next sentence and repeat from 3 until all sentences has been processed.
23       '''
24       global  punc
25      ret  =  []
26 
27      re_sent  =  re.compile( ' ([^%s]+)([%s])+ '   %  (punc, punc), re.MULTILINE)
28      cnt  =  0
29       for  match  in  re_sent.finditer(str):
30          sen  =  match.group( 1 )
31           # print sen
32          dag  =  organize(sen)
33           # print dag
34          path  =  find_path(dag)
35           for  i, l  in  path:
36               # print sen[i:i+l],
37              ret.append(sen[i:i + l])
38           # print
39           #  append a punctuation after the sentence
40           #  NOTICE: multiple punctuations is not supported
41          ret.append(match.group( 2 ))
42       return  ret
43 
44  def  organize(sentence):
45       ''' find all known words in the given sentence and organize it into a DAG
46      To represent nodes in a DAG, here a data structure of node is used as following:
47      [hop1, hop2]
48      hop is the distence from this node to the next one. On one char in the sentence
49      there could be more than one node structures that represent the multiple ways to
50      the segment the chars after this one. There is an ending node, [0], for easily
51      traversing the DAG.
52      And to represent the DAG, the following structure is used:
53      [[2,5], ..., [0]]
54       '''
55       global  dict
56      dag  =  []
57       # find all known words
58      n  =  l  =  len(sentence)
59       if  l > max_len:
60          l  =  max_len
61      c  =  0
62       while  c < len(sentence):
63          tl  =  l
64           if  c + tl > len(sentence):
65              tl  =  len(sentence) - c
66           while  tl > 1 :
67              t  =  sentence[c:c + tl]
68               if  dict.has_key(t):  #  find
69                   if  len(dag) == c:  #  first time to reach a node
70                      dag.append([len(t)])
71                   else :
72                      dag[c].append(len(t))
73               #  truncate one and retry
74              tl  -=   1
75           else :  # only one char left
76               if  len(dag) == c:
77                  dag.append([ 1 ])
78               else :
79                  dag[c].append( 1 )
80          c  +=   1   # try from next char
81      
82      dag.append([0])
83       return  dag
84 
85  def  find_path(dag):
86       ''' uses statnd Dijkstra algorithm to find the shortest path
87      from in the given dag. returns the path in such a format:
88      [(0,2), (2, 3), (5, 1), (6,4)]
89      the format of tuples in above sequence is (n, l), in which n
90      represent the index of this token and l is the length of this
91      token.
92       '''
93      wt  =  []
94      rc  =  []
95      pre  =  []
96      es  =  []
97       for  i  in  range(0, len(dag)):
98          wt.append(len(dag) + 1 )
99          rc.append(0)
100          pre.append(i - 1 )
101      rc[0]  =   1
102      wt[0]  =  0
103      es.append(0)
104       while   1 :
105           if  len(es) == 0:
106               break
107          min_node  =   - 1
108          min  =  len(dag) + 1
109           for  e  in  es:
110               if  wt[e] < min:
111                  min_node  =  e
112                  min  =  e
113          c  =  min_node
114          es.remove(c)
115          
116           for  e  in  dag[c]:
117              t  =  e + c
118               if   not  rc[t]:
119                  d  =  wt[c]  +   1
120                   if  d < wt[t]:
121                      wt[t]  =  d
122                      pre[t]  =  c
123                      es.append(t)
124      c  =  len(dag) - 1
125      path  =  []
126       while  pre[c] !=- 1 :
127          path.append((pre[c], c - pre[c]))
128          c  =  pre[c]
129      path.reverse()
130       return  path

MSN Space Link: http://spaces.msn.com/vanzolo/blog/cns!4A43F3D396FBF12F!1198.entry?_c11_blogpart_blogpart=blogview&_c=blogpart#permalink