问题一:为什么要导入数据到solr
因为solr所谓的索引可不是基于数据库的索引,而将数据库的数据导入到solr中,也就是core/data
文件下,并根据配置信息生成索引等。有点类似于redis。
问题二:怎么导入数据到solr
首先点击core/的DataImport选项:
这里提示要在solrconfig中去做一个DataImportHandler的定义。
1.首先在数据库中准备一张类似表做测试:
注意updateTime的类型为
timestamp
,默认值为:CURRENT_TIMESTAMP
:
这里是为了后面增量更新检测服务
然后准备好以下的jar包放到
core/lib
下,lib需要自己创建
第一个是数据库的驱动,后面两个在根目录的下的dist:
进入core/config【
solr-7.4.0\server\solr\helloSolr\conf
】创建一个数据源配置data-config.xml文件:
说明:
1.我是最新的mysql版本比较新因此需要写成:com.mysql.cj.jdbc.Driver
,Mysql 5.x直接写成com.mysql.jdbc.Driver
,并且Url
也不用加?serverTimezone=GMT%2B8&useSSL=false"
,貌似新版的mysql才有时区问题。
2.entity下的pk指向数据库的表id名称,比如我的id设计名称为pid,则必须要在pk中指定一下。
3.entity下query为导入到solr中的属性值,这个与下面field中的column是一一对应的关系,类似于mybtis自定义sql文件时的ORM映射关系。
4.deltaQuery为增量数据的检测sql语句,写法就是: select 你的表id from 你的表 where updateTime >'${dih.last_index_time}'
,其中updateTime为前面设计默认值为CURRENT_TIMESTAMP
的字段,该字段下,若对数据行任意列内容进行了修改,则对应的updateTime自动修改为当前系统时间,都是固定的写法,只能返回一个id,注意${dih.last_index_time}为固定写法,会对应到data-import.propertis下的一个属性值,注意不要忘记表达式外面的单引号。
5.deletedPkQuery为删除内容的增量sql,表字段中isdelete为1代表已经删除,其实就是一种伪删除手段。同样的pid为你具体表的一个id,都是固定的写法,只能返回一个id。
6.增量修改数据的查询sql语句:
select 返回的字段1,字段2,字段n from 查询的表 where 你的表id名字='${dih.delta.你的表id名字}'
注意这里的表id名字要与你pk指向的id一致。
7.
表示映射关系,这里的映射是与manged-schma.xml中的映射关系
进入core/conf【solr-7.4.0\server\solr\helloSolr\conf
】,打开manged-schma.xml,在结尾处添加一下内容:
Tips:这里的name与data-config.xml
下的field中的name是一致的对应关系。copyField为复制域,也就是共用的意思,当搜索某个商品时候,会在product_name
和product_description
域中都去寻找,也就是你淘宝买东西,不知道名字,你就搜用途描述也能找到该商品。type为text_ik的代表要用之前配置的中文分词器去分词处理,index代表索引,store代表保存。type的各种写法可以参照改文件中的具体定义,大约在207行左右的位置:
这个type和数据库中对应字段的类型保持一个对应关系,如varchar对应string,日期的对应pdate等等。
最后进入core/conf【solr-7.4.0\server\solr\helloSolr\conf
】,打开solrconfig.xml
,添加如下内容:
data-config.xml
这里也就看出为什么之前需要导入三个jar包到lib下。data-config.xml一定保证在同级目录下:
最后重新启动solr,导入数据:
点击execute即可。
然后点击query,测试搜索数据:
可以看到数据都查询了出来。
Tips:如果第一次导入后查询不到数据,可以将数据库某个记录修改一下,再重新导入
问题三:如何实现增量更新数据?
对于这个问题首先只需要理解和掌握两个东西
- 之前配置过的data-config.xml文件
前面已经提到过query、deltaQuery、deletedPkQuery、deltaImportQuery
这四个的作用要分清楚。
首先是增量更新索引的原理:首先是在数据库表中设计的一个默认值为当前系统时间的updateTime
字段,这个字段在每次修改记录时候,会自动把值设置为当前系统时间(修改那行记录,对于那行的update就自动修改),因此对于增量操作来说:只需要扫描出solr中缓存的时间${dih.last_index_time}
<updateTime
则为需要增量更新的记录。
而solr中是这样的顺序:1.扫描到updateTime>dih.last_index_time的id(deltaQuery
)2.通过这个id去增量索引(deltaImportQuery
)
对于删除的原理:其实就是扫描出isdelete=1的记录后,在solr中对应的数据记录给移除掉。
全量更新原理:就是query
这个sql执行一次的结果
- solr通过http请求去控制执行data-config.xml中的方法
打开solr的web管理主页的dataimport菜单:
在cmd中full-import则是代表全量更新,对应执行一次Query
,打开server/logs/solr.log:
同时在solr主页点击Execute,然后再观察solr.log的内容(如果没有变化,重新打开)
02:11:22.843 INFO (qtp817348612-23) [ x:helloSolr] o.a.s.h.d.DataImporter Loading DIH Configuration: data-config.xml
2018-12-04 02:11:22.850 INFO (qtp817348612-23) [ x:helloSolr] o.a.s.h.d.DataImporter Data Configuration loaded successfully
2018-12-04 02:11:22.861 INFO (qtp817348612-23) [ x:helloSolr] o.a.s.c.S.Request [helloSolr] webapp=/solr path=/dataimport params={core=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1543888704549&verbose=false} status=0 QTime=19
2018-12-04 02:11:22.873 INFO (Thread-20) [ ] o.a.s.h.d.DataImporter Starting Full Import
2018-12-04 02:11:22.877 INFO (qtp817348612-44) [ x:helloSolr] o.a.s.c.S.Request [helloSolr] webapp=/solr path=/dataimport params={indent=on&wt=json&command=status&_=1543888704549} status=0 QTime=0
2018-12-04 02:11:22.889 INFO (Thread-20) [ ] o.a.s.h.d.SimplePropertiesWriter Read dataimport.properties
2018-12-04 02:11:23.114 INFO (Thread-20) [ ] o.a.s.h.d.JdbcDataSource Creating a connection for entity product with URL: jdbc:mysql://127.0.0.1:3306/solr_base?serverTimezone=GMT%2B8&useSSL=false
2018-12-04 02:11:23.263 INFO (Thread-20) [ ] o.a.s.h.d.JdbcDataSource Time taken for getConnection(): 148
2018-12-04 02:11:23.502 INFO (Thread-20) [ ] o.a.s.h.d.DocBuilder Import completed successfully
连接了一次mysql并且打印出了发送http请求的参数。
这次打开浏览器的开发者模式,再执行一次Execute:
确实是执行了一次post请求:
http://127.0.0.1:8983/solr/helloSolr/dataimport?core=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1543888704549&verbose=falsecore=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1543888704549&verbose=false
打开Postman,将提取的url,放进去,并在sql中新插入一条记录:
send后:
{
"responseHeader":{
"status":0,
"QTime":25},
"initArgs":[
"defaults",[
"config","data-config.xml"]],
"command":"full-import",
"status":"idle",
"importResponse":"",
"statusMessages":{
"Total Requests made to DataSource":"1",
"Total Rows Fetched":"11",
"Total Documents Processed":"11",
"Total Documents Skipped":"0",
"Full Dump Started":"2018-12-04 02:17:19",
"":"Indexing completed. Added/Updated: 11 documents. Deleted 0 documents.",
"Committed":"2018-12-04 02:17:20",
"Time taken":"0:0:0.450"}}
最后在solr控制主页去query一下:
新增加的记录发现已经添加成功了,注意默认query显示10条记录,超过10条要设置start和rows的值。
-
增量更新
打开data-import菜单:
将Command切换成delta-import,然后修改数据中任意记录:
然后,执行Execute:
查看solr.log:
查看network:
同样的也是一次Post请求:
http://127.0.0.1:8983/solr/helloSolr/dataimport?core=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=delta-import&_=1543888704549&verbose=false
这个提取到Postman中send后是一样的结果
最终修改的记录都得到了更新。
总结
增量和全量对于我们开发者来说相当于像solr服务器发送两次Post请求。
不过再solr.log中会发现一个问题:
018-12-04 02:11:23.502 INFO (Thread-20) [ ] o.a.s.h.d.DocBuilder Import completed successfully
2018-12-04 02:11:23.502 INFO (Thread-20) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1618885459052593152,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2018-12-04 02:11:23.503 INFO (Thread-20) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@651aa215 commitCommandVersion:1618885459052593152
2018-12-04 02:11:23.823 INFO (qtp817348612-23) [ x:helloSolr] o.a.s.c.S.Request [helloSolr] webapp=/solr path=/dataimport params={indent=on&wt=json&command=status&_=1543888704549} status=0 QTime=1
2018-12-04 02:11:23.854 INFO (Thread-20) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6ec70aa[helloSolr] main]
2018-12-04 02:11:23.855 INFO (Thread-20) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
2018-12-04 02:11:23.855 INFO (searcherExecutor-10-thread-1) [ ] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@6ec70aa[helloSolr] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.4.0):C11/10:delGen=1) Uninverting(_4(7.4.0):C11)))}
2018-12-04 02:11:23.855 INFO (searcherExecutor-10-thread-1) [ ] o.a.s.c.QuerySenderListener QuerySenderListener done.
2018-12-04 02:11:23.861 INFO (searcherExecutor-10-thread-1) [ ] o.a.s.c.SolrCore [helloSolr] Registered new searcher Searcher@6ec70aa[helloSolr] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.4.0):C11/10:delGen=1) Uninverting(_4(7.4.0):C11)))}
2018-12-04 02:11:23.881 INFO (Thread-20
这里的日期跟实际的日期不一样,相差了八个时区。虽然再增量操作时候并没有遇到什么问题,但还是有点别扭。
修改方法:
打开solr/bin/solr.cmd,搜索UTC:
在后面+8
然后重启服务,再查看日志里的日期:
和我电脑日期一致了。
同时
core/conf/dataimport.properties
也自动变成了东八区:
定时增量/全量更新索引
之前实现了增量/全量更新索引,其实只是发送了下面的Post请求:
增量:
http://127.0.0.1:8983/solr/helloSolr/dataimport?core=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=delta-import&_=1543888704549&verbose=false
全量:
http://127.0.0.1:8983/solr/helloSolr/dataimport?core=helloSolr&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1543888704549&verbose=false
一种方式是自己写定时task,一种方式是使用solr提供的scheduler,solr只是提供了一个监听器,具体的策略代码需要自己写。gitHup也有很多solr-dataimportScheduler。使用solr提供的工具去写的好出是:
1.可以直接集成到solr的webapp中,不用单独起一个java项目
2.solr封装了直接读取dataimport.properties的对象SolrResourceLoader
,可以让操作更加简单。
这里提供一个solr-dataimportScheduler的下载地址:
个人修改的地址:https://github.com/Siwash/rpf-solr-study/tree/master/solr-dataimportscheduler-master
原作者地址:https://github.com/magese/solr-data-import-scheduler
我修改的地方:
注释了这一行的日志。因为SolrResourceLoader在solr7的时候这个方法是getInstancePath,而之前是getResourcePath。如果你开发过程中遇到了如下的报错:
2018-12-04 11:11:49.902 WARN (main) [ ] o.e.j.w.WebAppContext Failed startup of context o.e.j.w.WebAppContext@6aaceffd{/solr,file:///D:/solr_server_7.4/solr-7.4.0/server/solr-webapp/webapp/,UNAVAILABLE}{file:///D:/solr_server_7.4/solr-7.4.0/server//solr-webapp/webapp}
java.lang.NoSuchMethodError: org.apache.solr.core.SolrResourceLoader.(Ljava/lang/String;)V
at org.apache.solr.handler.dataimport.scheduler.SolrDataImportProperties.loadProperties(SolrDataImportProperties.java:36) ~[?:?]
at org.apache.solr.handler.dataimport.scheduler.BaseTimerTask.reloadParams(BaseTimerTask.java:57) ~[?:?]
at org.apache.solr.handler.dataimport.scheduler.BaseTimerTask.(BaseTimerTask.java:39) ~[?:?]
at org.apache.solr.handler.dataimport.scheduler.DeltaImportHTTPPostScheduler.(DeltaImportHTTPPostScheduler.java:20) ~[?:?]
at org.apache.solr.handler.dataimport.scheduler.ApplicationListener.contextInitialized(ApplicationListener.java:47) ~[?:?]
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558) ~[jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370) ~[jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) ~[jetty-webapp-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) ~[jetty-webapp-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287) ~[jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) [jetty-webapp-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:505) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:151) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:579) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) [jetty-deploy-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.Server.start(Server.java:419) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.Server.doStart(Server.java:386) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1569) [jetty-xml-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1509) [jetty-xml-9.4.10.v20180503.jar:9.4.10.v20180503]
at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_111]
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1508) [jetty-xml-9.4.10.v20180503.jar:9.4.10.v20180503]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) [start.jar:9.4.10.v20180503]
at org.eclipse.jetty.start.Main.start(Main.java:486) [start.jar:9.4.10.v20180503]
at org.eclipse.jetty.start.Main.main(Main.java:77) [start.jar:9.4.10.v20180503]
2018-12-04 11:11:49.921 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@738dc9b{HTTP/1.1,[http/1.1]}{0.0.0.0:8088}
2018-12-04 11:11:49.921 INFO (main) [ ] o.e.j.s.Server Started @2805ms
NoSuchMethodError其实就是因为找不到getResourcePath这个方法,将这一行打印日志注释后,就能保证同时兼容solr4-solr7了。
jar包地址:https://github.com/Siwash/rpf-solr-study/blob/master/solr-dataimportscheduler-4.x-7.x.jar
具体的应用方法:
进入solr-7.4.0\server\solr-webapp\webapp\WEB-INF
打开web.xml:
添加监听器:
org.apache.solr.handler.dataimport.scheduler.ApplicationListener
进入solr-7.4.0\server\solr\conf,注意:conf文件夹需要手动创建的,新建一个dataimport.properties:
#Tue Dec 04 10:47:18 GMT+08:00 2018
last_index_time=2018-12-04 10\:47\:17
product.last_index_time=2018-12-04 10\:47\:17
#################################################
# #
# dataimport scheduler properties #
# #
#################################################
# to sync or not to sync
# 1 - active; anything else - inactive
syncEnabled=1
# which cores to schedule
# in a multi-core environment you can decide which cores you want syncronized
# leave empty or comment it out if using single-core deployment
syncCores=你的core名字1,你的core名字2
# solr server name or IP address
# [defaults to localhost if empty]
server=你启动服务的ip地址
# solr server port
# [defaults to 80 if empty]
port=启动的端口号
# application name/context
# [defaults to current ServletContextListener's context (app) name]
webapp=默认web名字就是solr
# URL params [mandatory]
# remainder of URL
#刚才增量的参数
params=/dataimport?command=delta-import&commit=ture&wt=json&indent=true&entity=product&clean=false
# schedule interval
# number of minutes between two runs
# [defaults to 30 if empty]
#一分钟执行一次
interval=1
修改好之后,重启solr,在solr.log中查看日志记录:
2018-12-04 11:32:03.408 INFO (searcherExecutor-10-thread-1-processing-x:helloSolr) [ x:helloSolr] o.a.s.c.SolrCore [helloSolr] Registered new searcher Searcher@16013ab9[helloSolr] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.4.0):C11/10:delGen=1) Uninverting(_6(7.4.0):C12/2:delGen=2) Uninverting(_9(7.4.0):C2)))}
2018-12-04 11:33:01.442 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Process started at .............. 04.12.2018 11:33:01 441
2018-12-04 11:33:01.493 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Request method POST
2018-12-04 11:33:01.493 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Succesfully connected to server localhost
2018-12-04 11:33:01.494 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Using port 8983
2018-12-04 11:33:01.494 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Application name solr
2018-12-04 11:33:01.494 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] URL params /dataimport?command=delta-import&commit=ture&wt=json&indent=true&entity=product&clean=false
2018-12-04 11:33:01.494 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Full URL http://localhost:8983/solr/helloSolr/dataimport?command=delta-import&commit=ture&wt=json&indent=true&entity=product&clean=false
2018-12-04 11:33:01.502 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DataImporter Loading DIH Configuration: data-config.xml
2018-12-04 11:33:01.507 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DataImporter Data Configuration loaded successfully
2018-12-04 11:33:01.517 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DataImporter Starting Delta Import
2018-12-04 11:33:01.525 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.SimplePropertiesWriter Read dataimport.properties
2018-12-04 11:33:01.538 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Starting delta collection.
2018-12-04 11:33:01.562 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Running ModifiedRowKey() for Entity: product
2018-12-04 11:33:01.563 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.JdbcDataSource Creating a connection for entity product with URL: jdbc:mysql://127.0.0.1:3306/solr_base?serverTimezone=GMT%2B8&useSSL=false
2018-12-04 11:33:01.687 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.JdbcDataSource Time taken for getConnection(): 123
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Completed ModifiedRowKey for Entity: product rows obtained : 0
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Completed DeletedRowKey for Entity: product rows obtained : 0
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Completed parentDeltaQuery for Entity: product
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Delta Import completed successfully
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.h.d.DocBuilder Time taken = 0:0:0.174
2018-12-04 11:33:01.701 INFO (qtp817348612-16) [ x:helloSolr] o.a.s.u.p.LogUpdateProcessorFactory [helloSolr] webapp=/solr path=/dataimport params={indent=true&commit=ture&clean=false&wt=json&command=delta-import&entity=product}{} 0 204
2018-12-04 11:33:01.706 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Response message OK
2018-12-04 11:33:01.706 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Response code 200
2018-12-04 11:33:01.708 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Disconnected from server localhost
2018-12-04 11:33:01.708 INFO (Timer-0) [ ] o.a.s.h.d.s.HttpPostScheduler [helloSolr] Process ended at ................ 04.12.2018 11:33:01 708
定时器已经生效
这里的关键点是1.确保data-config.xml里面的sql没写错,2.新建的dataimport.propertis没写错,路径也是正确的,3.出现NoSuchMethodError确保注释了getInstancePath/getResoucePath
列举几个遇到的异常:
1.sql异常,在solr.log中出现sql异常,有的是连接异常[检查驱动和url等参数],有的是查询异常【检查sql语句写错没有,是否漏掉pk属性赋值】
2.连接异常,这个是定时器中出现的异常,比如connect refuse等【检查dataimport.propertis中web名字,连接地址,端口号和core名字等参数是否正确,可以在日志中提取url到postman进行测试】
3.NoSuchMethodError,这个建议直接注释getInstancePath/getResoucePath这一行的打印日志