lucene点滴
feild和term的关系是什么?
[在feild中切分出term]
===============
为了减小索引文件的大小，Lucene对索引还使用了压缩技术。首先，对词典文件中的关键词进行了压缩，
关键词压缩为<前缀长度，后缀>，例如：当前词为“阿拉伯语”，上一个词为“阿拉伯”，那么“阿拉伯语”压缩为<3，语>。
其次大量用到的是对数字的压缩，数字只保存与上一个值的差值（这样可以减小数字的长度，进而减少保存该数字需要的字节数）。
例如当前文章号是16389（不压缩要用3个字节保存），上一文章号是16382，压缩后保存7（只用一个字节）。
Field Data文件进行了压缩，使用的是zlib。
zlib、zip不如ppmd?
++++++++++++++++++
Lucene indexes may be composed of multiple sub-indexes, or segments.
Each segment has a separate Fieldable Info file.
tocken是term的一次出现，它包含trem文本和相应的起止偏移，以及一个类型字符串。
filter的作用就是限制只查询索引的某个子集，它的作用有点像SQL语句里的where，但又有区别，它不是正规查询的一部分，
只是对数据源进行预处理，然后交给查询语句。注意它执行的是预处理，而不是对查询结果进行过滤，
所以使用filter的代价是很大的，它可能会使一次查询耗时提高一百倍。
+++++++++++++++++
建索引的时候，只有一个线程在处理
When this file Lock File is present, a writer is currently modifying the index (adding or removing documents).
++++++++++++++++++
while (left < right && array[left].fieldInfo.name.compareTo(partition.fieldInfo.name) <= 0) // 这里的"<="应该为"<"
++left;
++++++++++ Merge +++++++++++
mergeFeild一直是二路归并，没有用多路
merge的时候，io有IndexReader完成
?何时开始merge
MergeFactor
涉及的类和文件类型
FieldInfos .fnm
单个文件无法全部载入内存时，怎么处理的?
+++++++++++++++++++++++
加快indexing的方法:
Open a single writer and re-use it for the duration of your indexing session.
Turn off compound file format.(but may run out of file descriptors)
Re-use Document and Field instances
Always add fields in the same order to your Document, when using stored fields or term vectors
>Use autoCommit=false when you open your IndexWriter (3.0.1无此设置了)
>Turn off any features you are not in fact using
>Use a faster analyzer.
>Speed up document construction.
>Don't optimize unless you really need to (for faster searching)
>Index into separate indices then merge.
>setMaxBufferedDocs(int maxBufferedDocs)
控制写入一个新的segment前内存中保存的document的数目，设置较大的数目可以加快建索引速度，默认为-1（不起用）。
* Determines the minimal number of documents required
* before the buffered in-memory documents are flushed as
* a new Segment. Large values generally gives faster
* indexing.
Disabled by default (writer flushes by RAM usage)
>我们可以先把索引写入RAMDirectory，达到一定数量时再批量写进FSDirectory，减少磁盘IO次数。
>setMergeFactor (10w个doc时，100-200是比较好的值)
>setRAMBufferSizeMB 默认16m
Generally for faster
* indexing performance it's best to flush by RAM usage
* instead of document count and use as large a RAM buffer
* as you can.
>优化时间范围限制(当需要搜索指定时间范围内的结果时)
>Quick tips:
Keep the size of the index small. Eliminate norms, Term vectors when not needed. Set Store flag for a field only if it a must.
Obvious, but oft-repeated mistake. Create only one instance of Searcher and reuse.
Keep in the index on fast disks. RAM, if you are paranoid.
>Use multiple threads with one IndexWriter
+++++++++++++++++++++
dspeak@lm-vm01:~/luowl/soft/lucene-3.0.1/src/demo$ java -cp .:/home/dspeak/luowl/search-node/lib/* org.apache.lucene.demo.IndexFiles ~/luowl/soft/data1.xml
+++++++++++++++++++
每次optimize都会重新做索引
+++++++++++++++++++++
LEVEL_LOG_SPAN, level的意义是?
Whenever extra segments (beyond the merge factor upper bound) are encountered,
all segments within the level are merged.
按level分组
++++++++++++++
We keep a separate Posting hash and other state for each
* thread and then merge postings hashes from all threads
* when writing the segment.
++++++++++++++++++++++++
/*
This is the current indexing chain:
DocConsumer / DocConsumerPerThread
--> code: DocFieldProcessor / DocFieldProcessorPerThread
--> DocFieldConsumer / DocFieldConsumerPerThread / DocFieldConsumerPerField
--> code: DocFieldConsumers / DocFieldConsumersPerThread / DocFieldConsumersPerField
--> code: DocInverter / DocInverterPerThread / DocInverterPerField
--> InvertedDocConsumer / InvertedDocConsumerPerThread / InvertedDocConsumerPerField
--> code: TermsHash / TermsHashPerThread / TermsHashPerField
--> TermsHashConsumer / TermsHashConsumerPerThread / TermsHashConsumerPerField
--> code: FreqProxTermsWriter / FreqProxTermsWriterPerThread / FreqProxTermsWriterPerField
--> code: TermVectorsTermsWriter / TermVectorsTermsWriterPerThread / TermVectorsTermsWriterPerField
--> InvertedDocEndConsumer / InvertedDocConsumerPerThread / InvertedDocConsumerPerField
--> code: NormsWriter / NormsWriterPerThread / NormsWriterPerField
--> code: StoredFieldsWriter / StoredFieldsWriterPerThread / StoredFieldsWriterPerField
*/
++++++++++++++++++++++++++++++
TermVector用来做什么的?
[统计词频，求词的出现位置等等]
++++++++++++++++++++
根据代码分析和测试，不Optimize, close的时候也会flush，FormatPostingsDocsWriter.addDoc会被调用