一.环境启动代码
SolrResourceLoader.SetSolrHome("/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr"); SolrResourceLoader.SetSchemaHome("/home/taobao/solr/schema"); ShardPartion.base = "/group/tbdp-etao-adhoc/p4padhoc/tabletest"; //必须的,如果不配置会有异常,SolrCore中会使用ShardPartion.getHdfsRealtimePath ShardPartion.taskIndex = 1; ShardPartion.index = IndexUtils.getHdfsForder(1); String hdfsconf = "/duitang/dist/sys/hadoop-1.2.1/conf/"; //必须的,如果不配置会有异常,ReadOnlyDirectory.getConf() HadoopUtil.setHdfsConfDir(hdfsconf); JettySolrRunner jetty = new JettySolrRunner("/solr", 1210); jetty.start(); while (true) { Thread.sleep(1000); }
二.启动之后通过URL测试
http://localhost:1210/solr/select?q=*:*&indexpartion=20111014
三.查询入口SolrDispatchFilter
入口在SolrDispatchFilter中,通过拦截请求中的select来加载对应的requestHandler.代码如下:
init(): CoreContainer.Initializer init = new CoreContainer.Initializer(); CoreContainer cores = init.initialize(); core = cores.getCore(""); doFilter(): final SolrConfig config = core.getSolrConfig(); SolrRequestParsers parser = new SolrRequestParsers(config); SolrQueryRequest solrReq = parser.parse( core, "select", httpServletRequest) SolrRequestHandler handler = core.getRequestHandler(null) SolrQueryResponse solrRsp = new SolrQueryResponse(); solrReq.getContext().put( "webapp", req.getContextPath() ); solrReq.getCore().execute( handler, solrReq, solrRsp );
四.核心方法SolrCore.execute()
入口SolrCore.execute(),前面都是web请求忽略掉。
SolrCore.execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryResponse rsp)执行过程:
SolrCore.execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryResponse rsp) -->handler.handleRequest(req, rsp); -->setResponseHeaderValues(handler, req, rsp);
SolrCore实际调用的是SearchHandler.handleRequest(),SearchHandler继承于RequestHandlerBase,覆盖了handleRequestBody(),调用关系如下:
-->RequestHandlerBase.handleRequest(SolrQueryRequest req, SolrQueryResponse rsp) -->SearchHandler.handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)
SearchHandler.handleRequestBody的实现比较复杂,如果shards不为null,会尝试请求各shards然后对结果合并。这里先看shards为null的情况:
-->QueryComponent.process(ResponseBuilder rb) -->SolrIndexSearcher searcher = rb.req.getSearcher(); -->searcher.search(result,cmd); -->FacetComponent.process(ResponseBuilder rb)
req是SolrQueryRequest类型,实际是SolrQueryRequestBase实例,看看其getSearcher()实现方法。
1.SolrQueryRequestBase.getSearcher() -->String partion=this.getParams().get(CommonParams.PARTION,"default"); -->SolrCore.getSearcher(partion,false,false,higohb); 2.SolrCore.getSearcher(String partion, boolean mdirs, boolean isclearCache, boolean ishb -->String partionKey=this.corename+"@"+partion+"@"+SolrResourceLoader.getCacheFlushKey(p); -->SolrCore.getSearcherByPath(p,partionKey, f.getAbsolutePath(), mdirs, isclearCache, ishb) 3.SolrCore.getSearcherByPath(PartionKey p, String partionKey, String path): if partion.name == "default": //初始化IndexReader RAMDirectory rd=new RAMDirectory(); rd.setCore(this,p); IndexWriter writer=new IndexWriter(rd, null,new KeepOnlyLastCommitDeletionPolicy(), MaxFieldLength.UNLIMITED); writer.setMergeFactor(10); writer.setUseCompoundFile(false); writer.close(); reader=IndexReader.open(rd); //初始化SolrIndexSearcher SolrIndexSearcher newSearcher = new SolrIndexSearcher(this, schema, "partion_" + partionKey, reader, true); newSearcher.setPartionCacheKey(p); newSearcher.setFieldcacheDir(fieldcacheDir); newSearcher.setPartionKey(partionKey); rtn = newHolderPartion(newSearcher); searchCache.put(partionKey, rtn); else: //初始化IndexReader MdrillDirectory rtdir = new ReadOnlyDirectory(f, HadoopUtil.hadoopConfDir, ShardPartion.getHdfsRealtimePath(p.tablename, p.partion)); rtdir.getForSearch() fs.mkdirs(new Path(hdfsPath).getParent()); //hdfs_path--> /group/.../20110111 path.mkdirs(); //local_path--> /duitang/.../20110111 File links=new File(path,"indexLinks"); while f=links.readline(): FSDirectory d=LinkFSDirectory.open(f); DirectoryInfo info=new DirectoryInfo(); info.d=d; info.tp=DirectoryInfo.DirTpe.file; diskDirector.put(f, info); IndexReader[] r = new IndexReader[diskDirector.size()]; for directory in diskDirector.values: r[i] = IndexReader.open(directory) if r.length==1: reader = r[0] else: reader = new MultiReader(r, true); //初始化SolrIndexSearcher(同上)
可以看到如果indexpartion没有传递默认是default,开辟空的内存,这时没有索引被读取。当有indexpartion时会读取indexpartion目录下的indexLinks。
比如/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr/data/20110111/indexLinks。indexLinks会记录索引存放的路径,然后分别加载,最后通过MultiReader实现。
索引测试路径:
local_path:/duitang/dist/app/branches/mdrill/trunk/adhoc-core/solr/data/20110111
hdfs_path:/group/tbdp-etao-adhoc/p4padhoc/tabletest/index/20110111/part-00001
最后补充solr容器的关闭,下次分析
"Shutdown" prio=10 tid=0x0000000040e97000 nid=0x7b1d at breakpoint[0x00007f7af4772000] java.lang.Thread.State: RUNNABLE at org.apache.solr.core.SolrCore.close(SolrCore.java:473) at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:202) - locked <0x00000000e4288b70> (a java.util.LinkedHashMap) at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:104) at org.mortbay.jetty.servlet.FilterHolder.destroyInstance(FilterHolder.java:127) at org.mortbay.jetty.servlet.FilterHolder.doStop(FilterHolder.java:107) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000e4280868> (a java.lang.Object) at org.mortbay.jetty.servlet.ServletHandler.doStop(ServletHandler.java:176) - locked <0x00000000e4280768> (a org.mortbay.jetty.servlet.ServletHandler) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000e42807d8> (a java.lang.Object) at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142) at org.mortbay.jetty.servlet.SessionHandler.doStop(SessionHandler.java:125) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000e4280758> (a java.lang.Object) at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142) at org.mortbay.jetty.handler.ContextHandler.doStop(ContextHandler.java:592) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000e4280708> (a java.lang.Object) at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142) at org.mortbay.jetty.Server.doStop(Server.java:283) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000e4280658> (a java.lang.Object) at org.mortbay.jetty.Server$ShutdownHookThread.run(Server.java:561)
补充SolrCore初始化的stacktrace,下次分析
"2010005445@qtp-385415131-0" daemon prio=10 tid=0x0000000041df3800 nid=0x440c at breakpoint[0x00007f2686060000] java.lang.Thread.State: RUNNABLE at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:278) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:85) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:808) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) at org.apache.lucene.index.IndexReader.open(IndexReader.java:509) at org.apache.lucene.index.IndexReader.open(IndexReader.java:369) at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:110) at org.apache.lucene.search.spell.SpellChecker.createSearcher(SpellChecker.java:752) at org.apache.lucene.search.spell.SpellChecker.swapSearcher(SpellChecker.java:729) at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:174) - locked <0x00000000e25f3818> (a java.lang.Object) at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149) at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127) at org.apache.solr.spelling.IndexBasedSpellChecker.init(IndexBasedSpellChecker.java:55) at org.apache.solr.handler.component.SpellCheckComponent.inform(SpellCheckComponent.java:598) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:523) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:385) at org.apache.solr.core.CoreContainer.createCore(CoreContainer.java:180) at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:151) - locked <0x00000000e2e23bf0> (a java.util.LinkedHashMap) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:148)