源码地址:https://github.com/XBWer/WordSimilarity
1.确定要解决的问题及意义
在基于代码片段的分类过程中,由于程序员对数据变量名的选取可能具有一定的规范性,在某一特定业务处理逻辑代码中,可能多个变量名之间具有关联性或相似性(如“trade”(商品交易)类中,可能存在“business”,“transaction”,“deal”等同义词),在某些情况下,它们以不同的词语表达了相同的含义。因此,为了能够对代码片段做出更加科学的类别判断,更好地识别这些同义词,我们有必要寻找一种能够解决避免由于同义词的存在而导致误分类的方法。说白了,就是要去判断词语之间的相似度(即确定是否为近义词),并找出代码段中出现次数最多的一组语义。
2.要达到的效果
即在给定的代码段中,能够发现哪些词是属于同义词,并且能够实现分类。
Eg.public static void function(){
String trade=”money”;
Int deal=5;
Long long business=0xfffffff;
Boolen transaction=TRUE;
……
}
Output:同义词有:trade,deal,business,transaction
这段代码很可能与trade有关
3.初识WordNet
问题确定了之后,通过网上的搜索,发现了WordNet和word2vec这两个相关的词汇。(后知后觉,这本身就是一个找近义词的过程)
3.1 WordNet是什么
首先,来看WordNet。搜了一下相关介绍:
WordNet是一个由普林斯顿大学认识科学实验室在心理学教授乔治·A·米勒的指导下建立和维护的英语字典。开发工作从1985年开始,从此以后该项目接受了超过300万美元的资助(主要来源于对机器翻译有兴趣的政府机构)。
由于它包含了语义信息,所以有别于通常意义上的字典。WordNet根据词条的意义将它们分组,每一个具有相同意义的字条组称为一个synset(同义词集合)。WordNet为每一个synset提供了简短,概要的定义,并记录不同synset之间的语义关系。
WordNet的开发有两个目的:
它既是一个字典,又是一个辞典,它比单纯的辞典或词典都更加易于使用。
支持自动的文本分析以及人工智能应用。
WordNet内部结构
在WordNet中,名词,动词,形容词和副词各自被组织成一个同义词的网络,每个同义词集合都代表一个基本的语义概念,并且这些集合之间也由各种关系连接。(一个多义词将出现在它的每个意思的同义词集合中)。在WordNet的第一版中(标记为1.x),四种不同词性的网络之间并无连接。WordNet的名词网络是第一个发展起来的。
名词网络的主干是蕴涵关系的层次(上位/下位关系),它占据了关系中的将近80%。层次中的最顶层是11个抽象概念,称为基本类别始点(unique beginners),例如实体(entity,“有生命的或无生命的具体存在”),心理特征(psychological feature,“生命有机体的精神上的特征)。名词层次中最深的层次是16个节点。
(wikipedia)
通俗地来说,WordNet是一个结构化很好的知识库,它不但包括一般的词典功能,另外还有词的分类信息。目前,基于WordNet的方法相对来说比较成熟,比如路径方法 (lch)、基于信息论方法(res)等。(详见参考文献)
3.2 WordNet的安装与配置
有了WordNet ,也就等于是有了我们所要的单词库。所以,暂时先不考虑相似度的计算,把WordNet下载下来再说。
参考http://hi.baidu.com/buptyoyo/item/f13dfe463c061e3afb896028。顺利地下载,安装以及跑demo。
之后,一起来看一下WordNet的文件结构:
bin目录下,有可执行文件WordNet 2.1.exe:
可以看到,WordNet对所有的英文单词都进行的分类,并且形成了一棵语义树。在本例中,entity——>abstract entity——>abstraction——>attribute——>state——>feeling——> emotion——>love;
从叶子节点到根节点
WordNet名次分类中的25个基本类:
dict目录里面存放的就是资源库了,可以看到,它以形容词,副词,名词,动词来分类:
doc为WordNet的用户手册文件文件夹
lib为WordNet软件使用Windows资源的函数库
src为源码文件夹
4.解决问题的大致思路
我们首先以 WordNet 的词汇语义分类作为基础,抽取出其中的同义词,然后采用基于向量空间的方法计算出相似度。工作流程如下:
5.基于WordNet的相似度计算
以下摘自:《基于WordNet的英语词语相似度计算》
5.1 特征提取
5.2 意义相似度和词语相似度的计算
6.实现效果
与“trade”的相似度比较:
先看第一组:trade vs trade
自己和自己当然是相似度100%
再看第二组:trade#n#5 vs deal#n#1
相似度竟然和第一组是一样的!根据结果,trade作为名词时,它的第5种含义和deal作为名词时的第1种含义是完全相似的。让我们去库里看个究竟:
trade#n#5:
再来看一组不是很好理解的:
trade#n#7 vs deal#n#2
他们的相似度达到了0.14+,算是比较高的了,这是为什么呢?
trade#n#7:
sunshine#n#2:
相信聪明的你一定明白了为什么。
与“cat”的相似度比较:
7.代码分析
工程结构图:
test.java
1 package JWordNetSim.test; 2 3 import java.io.FileInputStream; 4 import java.util.HashMap; 5 import java.util.Map; 6 7 import net.didion.jwnl.JWNL; 8 import net.didion.jwnl.data.IndexWord; 9 import net.didion.jwnl.data.POS; 10 import net.didion.jwnl.dictionary.Dictionary; 11 import shef.nlp.wordnet.similarity.SimilarityMeasure; 12 13 /** 14 * A simple test of this WordNet similarity library. 15 * @author Mark A. Greenwood 16 */ 17 public class Test 18 { 19 public static void main(String[] args) throws Exception 20 { 21 //在运行代码前,必须在本机上安装wordnet2.0,只能装2.0,装了2.1会出错 22 JWNL.initialize(new FileInputStream("D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\wordnet.xml")); 23 24 //建议一个映射去配置相关参数 25 Mapparams = new HashMap (); 26 27 //the simType parameter is the class name of the measure to use 28 params.put("simType","shef.nlp.wordnet.similarity.JCn"); 29 30 //this param should be the URL to an infocontent file (if required 31 //by the similarity measure being loaded) 32 params.put("infocontent","file:D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\ic-bnc-resnik-add1.dat"); 33 34 //this param should be the URL to a mapping file if the 35 //user needs to make synset mappings 36 params.put("mapping","file:D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\domain_independent.txt"); 37 38 //create the similarity measure 39 SimilarityMeasure sim = SimilarityMeasure.newInstance(params); 40 41 //取词 42 // Dictionary dict = Dictionary.getInstance(); 43 // IndexWord word1 = dict.getIndexWord(POS.NOUN, "trade"); //这里把trade和dog完全定义为名词来进行处理 44 // IndexWord word2 = dict.getIndexWord(POS.NOUN,"dog"); // 45 // 46 // //and get the similarity between the first senses of each word 47 // System.out.println(word1.getLemma()+"#"+word1.getPOS().getKey()+"#1 " + word2.getLemma()+"#"+word2.getPOS().getKey()+"#1 " + sim.getSimilarity(word1.getSense(1), word2.getSense(1))); 48 49 // //get similarity using the string methods (note this also makes use 50 // //of the fake root node) 51 // System.out.println(sim.getSimilarity("trade#n","deal#n")); 52 53 //get a similarity that involves a mapping 54 System.out.println(sim.getSimilarity("trade", "trade")); 55 System.out.println(sim.getSimilarity("trade", "deal")); 56 System.out.println(sim.getSimilarity("trade", "commerce")); 57 System.out.println(sim.getSimilarity("trade", "transaction")); 58 System.out.println(sim.getSimilarity("trade", "finance")); 59 System.out.println(sim.getSimilarity("trade", "financial")); 60 System.out.println(sim.getSimilarity("trade", "business")); 61 System.out.println(sim.getSimilarity("trade", "economy")); 62 System.out.println(sim.getSimilarity("trade", "school")); 63 System.out.println(sim.getSimilarity("trade", "dog")); 64 System.out.println(sim.getSimilarity("trade", "cat")); 65 System.out.println(sim.getSimilarity("trade", "book")); 66 System.out.println(sim.getSimilarity("trade", "sunshine")); 67 System.out.println(sim.getSimilarity("trade", "smile")); 68 System.out.println(sim.getSimilarity("trade", "nice")); 69 System.out.println(sim.getSimilarity("trade", "hardly")); 70 System.out.println(sim.getSimilarity("trade", "beautiful")); 71 } 72 }
SimilarityMeasure.java
1 package shef.nlp.wordnet.similarity; 2 3 import java.io.BufferedReader; 4 import java.io.InputStreamReader; 5 import java.net.URL; 6 import java.util.Arrays; 7 import java.util.HashMap; 8 import java.util.HashSet; 9 import java.util.LinkedHashMap; 10 import java.util.Map; 11 import java.util.Set; 12 13 import net.didion.jwnl.JWNLException; 14 import net.didion.jwnl.data.IndexWord; 15 import net.didion.jwnl.data.POS; 16 import net.didion.jwnl.data.Synset; 17 import net.didion.jwnl.dictionary.Dictionary; 18 19 /** 20 * An abstract notion of a similarity measure that all provided 21 * implementations extend. 22 * @author Mark A. Greenwood 23 */ 24 public abstract class SimilarityMeasure 25 { 26 /** 27 * A mapping of terms to specific synsets. Usually used to map domain 28 * terms to a restricted set of synsets but can also be used to map 29 * named entity tags to appropriate synsets. 30 */ 31 private Map> domainMappings = new HashMap >(); 32 33 /** 34 * The maximum size the cache can grow to 35 */ 36 private int cacheSize = 5000; 37 38 /** 39 * To speed up computation of the similarity between two synsets 40 * we cache each similarity that is computed so we only have to 41 * do each one once. 42 */ 43 private Map cache = new LinkedHashMap (16,0.75f,true) 44 { 45 public boolean removeEldestEntry(Map.Entry eldest) 46 { 47 //if the size is less than zero then the user is asking us 48 //not to limit the size of the cache so return false 49 if (cacheSize < 0) return false; 50 51 //if the cache has crown bigger than it's max size return true 52 return size() > cacheSize; 53 } 54 }; 55 56 /** 57 * Get a previously computed similarity between two synsets from the cache. 58 * @param s1 the first synset between which we are looking for the similarity. 59 * @param s2 the other synset between which we are looking for the similarity. 60 * @return The similarity between the two sets or null 61 * if it is not in the cache. 62 */ 63 protected final Double getFromCache(Synset s1, Synset s2) 64 { 65 return cache.get(s1.getKey()+"-"+s2.getKey()); 66 } 67 68 /** 69 * Add a computed similarity between two synsets to the cache so that 70 * we don't have to compute it if it is needed in the future. 71 * @param s1 one of the synsets between which we are storring a similarity. 72 * @param s2 the other synset between which we are storring a similarity. 73 * @param sim the similarity between the two supplied synsets. 74 * @return the similarity score just added to the cache. 75 */ 76 protected final double addToCache(Synset s1, Synset s2, double sim) 77 { 78 cache.put(s1.getKey()+"-"+s2.getKey(),sim); 79 80 return sim; 81 } 82 83 /** 84 * Configures the similarity measure using the supplied parameters. 85 * @param params a set of key-value pairs that are used to configure 86 * the similarity measure. See concrete implementations for details 87 * of expected/possible parameters. 88 * @throws Exception if an error occurs while configuring the similarity measure. 89 */ 90 protected abstract void config(Map params) throws Exception; 91 92 /** 93 * Create a new instance of a similarity measure. 94 * @param confURL the URL of a configuration file. Parameters are specified 95 * one per line as key:value pairs. 96 * @return a new instance of a similairy measure as defined by the 97 * supplied configuration URL. 98 * @throws Exception if an error occurs while creating the similarity measure. 99 */ 100 public static SimilarityMeasure newInstance(URL confURL) throws Exception 101 { 102 //create map to hold the key-value pairs we are going to read from 103 //the configuration file 104 Map params = new HashMap (); 105 106 //create a reader for the config file 107 BufferedReader in = null; 108 109 try 110 { 111 //open the config file 112 in = new BufferedReader(new InputStreamReader(confURL.openStream())); 113 114 String line = in.readLine(); 115 while (line != null) 116 { 117 line = line.trim(); 118 119 if (!line.equals("")) 120 { 121 //if the line contains something then 122 123 //split the data so we get the key and value 124 String[] data = line.split("\\s*:\\s*",2); 125 126 if (data.length == 2) 127 { 128 //if the line is valid add the two parts to the map 129 params.put(data[0], data[1]); 130 } 131 else 132 { 133 //if the line isn't valid tell the user but continue on 134 //with the rest of the file 135 System.out.println("Config Line is Malformed: " + line); 136 } 137 } 138 139 //get the next line ready to process 140 line = in.readLine(); 141 } 142 } 143 finally 144 { 145 //close the config file if it got opened 146 if (in != null) in.close(); 147 } 148 149 //create and return a new instance of the similarity measure specified 150 //by the config file 151 return newInstance(params); 152 } 153 154 /** 155 * Creates a new instance of a similarity measure using the supplied parameters. 156 * @param params a set of key-value pairs which define the similarity measure. 157 * @return the newly created similarity measure. 158 * @throws Exception if an error occurs while creating the similarity measure. 159 */ 160 public static SimilarityMeasure newInstance(Map params) throws Exception 161 { 162 //get the class name of the implementation we need to load 163 String name = params.remove("simType"); 164 165 //if the name hasn't been specified then throw an exception 166 if (name == null) throw new Exception("Must specifiy the similarity measure to use"); 167 168 //Get hold of the class we need to load 169 @SuppressWarnings("unchecked") Class c = (Class )Class.forName(name); 170 171 //create a new instance of the similarity measure 172 SimilarityMeasure sim = c.newInstance(); 173 174 //get the cache parameter from the config params 175 String cSize = params.remove("cache"); 176 177 //if a cache size was specified then set it 178 if (cSize != null) sim.cacheSize = Integer.parseInt(cSize); 179 180 //get the url of the domain mapping file 181 String mapURL = params.remove("mapping"); 182 183 if (mapURL != null) 184 { 185 //if a mapping file has been provided then 186 187 //open a reader over the file 188 BufferedReader in = new BufferedReader(new InputStreamReader((new URL(mapURL)).openStream())); 189 190 //get the first line ready for processing 191 String line = in.readLine(); 192 193 while (line != null) 194 { 195 if (!line.startsWith("#")) 196 { 197 //if the line isn't a comment (i.e. it doesn't start with #) then... 198 199 //split the line at the white space 200 String[] data = line.trim().split("\\s+"); 201 202 //create a new set to hold the mapped synsets 203 Set mappedTo = new HashSet (); 204 205 for (int i = 1 ; i < data.length ; ++i) 206 { 207 //for each synset mapped to get the actual Synsets 208 //and store them in the set 209 mappedTo.addAll(sim.getSynsets(data[i])); 210 } 211 212 //if we have found some actual synsets then 213 //store them in the domain mappings 214 if (mappedTo.size() > 0) sim.domainMappings.put(data[0], mappedTo); 215 } 216 217 //get the next line from the file 218 line = in.readLine(); 219 } 220 221 //we have finished with the mappings file so close it 222 in.close(); 223 } 224 225 //make sure it is configured properly 226 sim.config(params); 227 228 //then return it 229 return sim; 230 } 231 232 /** 233 * This is the method responsible for computing the similarity between two 234 * specific synsets. The method is implemented differently for each 235 * similarity measure so see the subclasses for detailed information. 236 * @param s1 one of the synsets between which we want to know the similarity. 237 * @param s2 the other synset between which we want to know the similarity. 238 * @return the similarity between the two synsets. 239 * @throws JWNLException if an error occurs accessing WordNet. 240 */ 241 public abstract double getSimilarity(Synset s1, Synset s2) throws JWNLException; 242 243 /** 244 * Get the similarity between two words. The words can be specified either 245 * as just the word or in an encoded form including the POS tag and possibly 246 * the sense number, i.e. cat#n#1 would specifiy the 1st sense of the noun cat. 247 * @param w1 one of the words to compute similarity between. 248 * @param w2 the other word to compute similarity between. 249 * @return a SimilarityInfo instance detailing the similarity between the 250 * two words specified. 251 * @throws JWNLException if an error occurs accessing WordNet. 252 */ 253 public final SimilarityInfo getSimilarity(String w1, String w2) throws JWNLException 254 { 255 //Get the (possibly) multiple synsets associated with each word 256 Set ss1 = getSynsets(w1); 257 Set ss2 = getSynsets(w2); 258 259 //assume the words are not at all similar 260 SimilarityInfo sim = null; 261 262 for (Synset s1 : ss1) 263 { 264 for (Synset s2 : ss2) 265 { 266 //for each pair of synsets get the similarity 267 double score = getSimilarity(s1, s2); 268 269 if (sim == null || score > sim.getSimilarity()) 270 { 271 //if the similarity is better than we have seen before 272 //then create and store an info object describing the 273 //similarity between the two synsets 274 sim = new SimilarityInfo(w1, s1, w2, s2, score); 275 } 276 } 277 } 278 279 //return the maximum similarity we have found 280 return sim; 281 } 282 283 /** 284 * Finds all the synsets associated with a specific word. 285 * @param word the word we are interested. Note that this may be encoded 286 * to include information on POS tag and sense index. 287 * @return a set of synsets that are associated with the supplied word 288 * @throws JWNLException if an error occurs accessing WordNet 289 */ 290 private final Set getSynsets(String word) throws JWNLException 291 { 292 //get a handle on the WordNet dictionary 293 Dictionary dict = Dictionary.getInstance(); 294 295 //create an emptuy set to hold any synsets we find 296 Set synsets = new HashSet (); 297 298 //split the word on the # characters so we can get at the 299 //upto three componets that could be present: word, POS tag, sense index 300 String[] data = word.split("#"); 301 302 //if the word is in the domainMappings then simply return the mappings 303 if (domainMappings.containsKey(data[0])) return domainMappings.get(data[0]); 304 305 if (data.length == 1) 306 { 307 //if there is just the word 308 309 for (IndexWord iw : dict.lookupAllIndexWords(data[0]).getIndexWordArray()) 310 { 311 //for each matching word in WordNet add all it's senses to 312 //the set we are building up 313 synsets.addAll(Arrays.asList(iw.getSenses())); 314 } 315 316 //we have finihsed so return the synsets we found 317 return synsets; 318 } 319 320 //the calling method specified a POS tag as well so get that 321 POS pos = POS.getPOSForKey(data[1]); 322 323 //if the POS tag isn't valid throw an exception 324 if (pos == null) throw new JWNLException("Invalid POS Tag: " + data[1]); 325 326 //get the word with the specified POS tag from WordNet 327 IndexWord iw = dict.getIndexWord(pos, data[0]); 328 329 if (data.length > 2) 330 { 331 //if the calling method specified a sense index then 332 //add just that sysnet to the set we are creating 333 synsets.add(iw.getSense(Integer.parseInt(data[2]))); 334 } 335 else 336 { 337 //no sense index was specified so add all the senses of 338 //the word to the set we are creating 339 synsets.addAll(Arrays.asList(iw.getSenses())); 340 } 341 342 //return the set of synsets we found for the specified word 343 return synsets; 344 } 345 }
每个函数都有详细注解,大家应该都看的明白。
262~277的循环过程如下:
JCN.java
1 /************************************************************************ 2 * Copyright (C) 2006-2007 The University of Sheffield * 3 * Developed by Mark A. Greenwood* 4 * * 5 * This program is free software; you can redistribute it and/or modify * 6 * it under the terms of the GNU General Public License as published by * 7 * the Free Software Foundation; either version 2 of the License, or * 8 * (at your option) any later version. * 9 * * 10 * This program is distributed in the hope that it will be useful, * 11 * but WITHOUT ANY WARRANTY; without even the implied warranty of * 12 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * 13 * GNU General Public License for more details. * 14 * * 15 * You should have received a copy of the GNU General Public License * 16 * along with this program; if not, write to the Free Software * 17 * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. * 18 ************************************************************************/ 19 20 package shef.nlp.wordnet.similarity; 21 22 import net.didion.jwnl.JWNLException; 23 import net.didion.jwnl.data.Synset; 24 25 /** 26 * An implementation of the WordNet similarity measure developed by Jiang and 27 * Conrath. For full details of the measure see: 28 *Jiang J. and Conrath D. 1997. Semantic similarity based on corpus 29 * statistics and lexical taxonomy. In Proceedings of International 30 * Conference on Research in Computational Linguistics, Taiwan.31 * @author Mark A. Greenwood 32 */ 33 public class JCn extends ICMeasure 34 { 35 /** 36 * Instances of this similarity measure should be generated using the 37 * factory methods of {@link SimilarityMeasure}. 38 */ 39 protected JCn() 40 { 41 //A protected constructor to force the use of the newInstance method 42 } 43 44 @Override public double getSimilarity(Synset s1, Synset s2) throws JWNLException 45 { 46 //if the POS tags are not the same then return 0 as this measure 47 //only works with 2 nouns or 2 verbs. 48 if (!s1.getPOS().equals(s2.getPOS())) return 0; 49 50 //see if the similarity is already cached and... 51 Double cached = getFromCache(s1, s2); 52 53 //if it is then simply return it 54 if (cached != null) return cached.doubleValue(); 55 56 //Get the Information Content (IC) values for the two supplied synsets 57 double ic1 = getIC(s1); 58 double ic2 = getIC(s2); 59 60 //if either IC value is zero then cache and return a sim of 0 61 if (ic1 == 0 || ic2 == 0) return addToCache(s1,s2,0); 62 63 //Get the Lowest Common Subsumer (LCS) of the two synsets 64 Synset lcs = getLCSbyIC(s1,s2); 65 66 //if there isn't an LCS then cache and return a sim of 0 67 if (lcs == null) return addToCache(s1,s2,0); 68 69 //get the IC valueof the LCS 70 double icLCS = getIC(lcs); 71 72 //compute the distance between the two synsets 73 //NOTE: This is the original JCN measure 74 double distance = ic1 + ic2 - (2 * icLCS); 75 76 //assume the similarity between the synsets is 0 77 double sim = 0; 78 79 if (distance == 0) 80 { 81 //if the distance is 0 (i.e. ic1 + ic2 = 2 * icLCS) then... 82 83 //get the root frequency for this POS tag 84 double rootFreq = getFrequency(s1.getPOS()); 85 86 if (rootFreq > 0.01) 87 { 88 //if the root frequency has a value then use it to generate a 89 //very large sim value 90 sim = 1/-Math.log((rootFreq - 0.01) / rootFreq); 91 } 92 } 93 else 94 { 95 //this is the normal case so just convert the distance 96 //to a similarity by taking the multiplicative inverse 97 sim = 1/distance; 98 } 99 100 //cache and return the calculated similarity 101 return addToCache(s1,s2,sim); 102 } 103 }
LIN.java
1 package shef.nlp.wordnet.similarity; 2 3 import net.didion.jwnl.JWNLException; 4 import net.didion.jwnl.data.Synset; 5 6 /** 7 * An implementation of the WordNet similarity measure developed by Lin. For 8 * full details of the measure see: 9 *Lin D. 1998. An information-theoretic definition of similarity. In10 * Proceedings of the 15th International Conference on Machine 11 * Learning, Madison, WI. 12 * @author Mark A. Greenwood 13 */ 14 public class Lin extends ICMeasure 15 { 16 /** 17 * Instances of this similarity measure should be generated using the 18 * factory methods of {@link SimilarityMeasure}. 19 */ 20 protected Lin() 21 { 22 //A protected constructor to force the use of the newInstance method 23 } 24 25 @Override public double getSimilarity(Synset s1, Synset s2) throws JWNLException 26 { 27 //if the POS tags are not the same then return 0 as this measure 28 //only works with 2 nouns or 2 verbs. 29 if (!s1.getPOS().equals(s2.getPOS())) return 0; 30 31 //see if the similarity is already cached and... 32 Double cached = getFromCache(s1, s2); 33 34 //if it is then simply return it 35 if (cached != null) return cached.doubleValue(); 36 37 //Get the Information Content (IC) values for the two supplied synsets 38 double ic1 = getIC(s1); 39 double ic2 = getIC(s2); 40 41 //if either IC value is zero then cache and return a sim of 0 42 if (ic1 == 0 || ic2 == 0) return addToCache(s1,s2,0); 43 44 //Get the Lowest Common Subsumer (LCS) of the two synsets 45 Synset lcs = getLCSbyIC(s1,s2); 46 47 //if there isn't an LCS then cache and return a sim of 0 48 if (lcs == null) return addToCache(s1,s2,0); 49 50 //get the IC valueof the LCS 51 double icLCS = getIC(lcs); 52 53 //caluclaue the similarity score 54 double sim = (2*icLCS)/(ic1+ic2); 55 56 //cache and return the calculated similarity 57 return addToCache(s1,s2,sim); 58 } 59 }
参考文献:
《基于维基百科的语义相似度计算》盛志超,陶晓鹏(复旦大学计算机科学技术学院);
《基于WordNet的英语词语相似度计算》颜伟,荀恩东(北京语言大学 语言信息处理研究所)
WordNet中的名词:http://ccl.pku.edu.cn/doubtfire/semantics/wordnet/c-wordnet/nouns-in-wordnet.htm
MIT的JWI(Java WordNet Interface)和JWNL(Java WordNet Library)比较
http://jxr19830617.blog.163.com/blog/static/163573067201301985219857/
http://jxr19830617.blog.163.com/blog/static/1635730672013019105255295/