A. Covert text file to sequeces file
Class: SequenceFilesFromDirectory
Functions: Converts a directory of text documents into SequenceFiles of Specified chunkSize. This class takes in a parent directory containing sub folders of text documents and recursively reads the files and creates the {@link org.apache.hadoop.io.SequenceFile}s of docid => content. The docid is set as the relative path of the document from the parent directory prepended with a specified prefix. You can also specify the input encoding of the text files. The content of the output SequenceFiles are encoded as UTF-8 text.
B.Covert sequences file to Document Vectors
Class: SparseVectorsFromSequenceFiles
Functions: Converts a given set of sequence files into SparseVectors
1. tokenize the documents in sequence file
DocumentProcessor.tokenizeDocuments(inputDir, analyzerClass, tokenizedPath, conf);
Key: /2435: Value: 为什么很多看起来不是很复杂的网站，比如 Facebook 需要大量顶尖高手来开发？
Key: /2436: Value: iNOKNOK敲门网络是怎样的一个网站？
Key: /2437: Value: 北京上海生活成本为何超过了巴黎和纽约？
======>
Key: /2435: Value: [为什么, 很多, 看起来, 不是, 很复杂, 网站, 比如, facebook, 需要, 大量, 顶尖, 高手, 开发]
Key: /2436: Value: [inoknok, 敲门, 网络, 怎样, 一个, 网站]
Key: /2437: Value: [北京, 上海, 生活, 成本, 为何, 超过, 巴黎, 纽约]
2.Create Term Frequency (Tf) Vectors from the input set of documents
DictionaryVectorizer.createTermFrequencyVectors
2-1 startWordCounting(input, dictionaryJobPath, baseConf, minSupport);
Count the frequencies of words in parallel using Map/Reduce. If term frequency in the corpus is smaller than
minSupport, then the term will be stripped, which means that the word or term will not be included in the dictionary.
tokenzied-documents:
Key: /2435: Value: [为什么, 很多, 看起来, 不是, 很复杂, 网站, 比如, facebook, 需要, 大量, 顶尖, 高手, 开发]
Key: /2436: Value: [inoknok, 敲门, 网络, 怎样, 一个, 网站]
Key: /2437: Value: [北京, 上海, 生活, 成本, 为何, 超过, 巴黎, 纽约]
.......
=============>
wordcount:
Key: 专用: Value: 1
Key: 世外桃源: Value: 1
Key: 世界: Value: 13
Key: 世界上: Value: 6
Key: 世界杯: Value: 20
Key: 世界观: Value: 1
2-2 Read the feature frequency List which is built at the end of the Word Count Job and assign ids to them
createDictionaryChunks(dictionaryJobPath, output, baseConf, chunkSizeInMegabytes, maxTermDimension);
wordcount:
Key: 80、90后最大疑惑: Value: 1
Key: -727379968: Value: 1
Key: 0.25%: Value: 1
Key: 0.9秒: Value: 1
Key: 00: Value: 1
Key: 001: Value: 1
Key: 0day: Value: 1
Key: 1.8: Value: 1
Key: 1/100: Value: 1
Key: 10: Value: 4
......
===========>
dictionary:
Key: 80、90后最大疑惑: Value: 0
Key: -727379968: Value: 1
Key: 0.25%: Value: 2
Key: 0.9秒: Value: 3
Key: 00: Value: 4
Key: 001: Value: 5
Key: 0day: Value: 6
Key: 1.8: Value: 7
Key: 1/100: Value: 8
Key: 10: Value: 9
Key: 100: Value: 10
.........
2-3 Create a partial vector using a chunk of features from the input documents. And merge partial vector to complete document vector
makePartialVectors
PartialVectorMerger.mergePartialVectors
dictionary
Key: 80、90后最大疑惑: Value: 0Key: -727379968: Value: 1
Key: 0.25%: Value: 2
Key: 0.9秒: Value: 3
tokenized-documents
Key: /2435: Value: [为什么, 很多, 看起来, 不是, 很复杂, 网站, 比如, facebook, 需要, 大量, 顶尖, 高手, 开发]
Key: /2436: Value: [inoknok, 敲门, 网络, 怎样, 一个, 网站]
Key: /2437: Value: [北京, 上海, 生活, 成本, 为何, 超过, 巴黎, 纽约]
......
＝＝＝＝＝＝>
Key: /2435: Value: {181:1.0,512:1.0,618:1.0,1738:1.0,2142:1.0,2221:1.0,2222:1.0,3072:1.0,3517:1.0,3776:1.0,4518:1.0,4545:1.0,4631:1.0}
Key: /2436: Value: {210:1.0,329:1.0,2296:1.0,2666:1.0,3776:1.0,3777:1.0}
Key: /2437: Value: {441:1.0,619:1.0,1208:1.0,2066:1.0,2390:1.0,3375:1.0,3714:1.0,4173:1.0}
.......
3. Calculates the document frequencies of all terms from the input set of vectors