Neo4j 频繁导入数据导致DeadLock的问题

2018-01-04：补充说明

如下的解决方案还是会偶尔蹦出几个DeadLock的问题，在运维查看了neo4j的内存占用时发现一个问题，neo4j十分吃swap，导致执行效率十分低下，再次仔细查阅官网发现缺少了一个很重要的内存配置参数，

dbms.memory.pagecache.size=1g

The amount of memory to use for mapping the store files, in bytes (or
kilobytes with the 'k' suffix, megabytes with 'm' and gigabytes with 'g').
If Neo4j is running on a dedicated server, then it is generally recommended
to leave about 2-4 gigabytes for the operating system, give the JVM enough
heap to hold all your transaction state and query context, and then leave the
rest for the page cache.
The default page cache memory assumes the machine is dedicated to running
Neo4j, and is heuristically set to 50% of RAM minus the max Java heap size.

---

 {"results":[],"errors":[{"code":"Neo.TransientError.Transaction.DeadlockDetected","message":"LockClient[6092] can't wait on resource RWLock[NODE(133321815), hash=1215661247] since => LockClient[6092] <-[:HELD_BY]- RWLock[NODE(38473669), hash=1470152465] <-[:WAITING_FOR]- LockClient[6091] <-[:HELD_BY]- RWLock[NODE(133321815), hash=1215661247]","stackTrace":"org.neo4j.kernel.DeadlockDetectedException: LockClient[6092] can't wait on resource RWLock[NODE(133321815), hash=1215661247] since => LockClient[6092] <-[:HELD_BY]- RWLock[NODE(38473669), hash=1470152465] <-[:WAITING_FOR]- LockClient[6091] <-[:HELD_BY]- RWLock[NODE(133321815), hash=1215661247]\n\tat org.neo4j.kernel.impl.locking.community.RagManager.checkWaitOnRecursive(RagManager.java:198)\n\tat org.neo4j.kernel.impl.locking.community.RagManager.checkWaitOnRecursive(RagManager.java:225)\n\tat org.neo4j.kernel.impl.locking.community.RagManager.checkWaitOn(RagManager.java:165)\n\tat org.neo4j.kernel.impl.locking.community.RWLock.acquireWriteLock(RWLock.java:396)\n\tat org.neo4j.kernel.impl.locking.community.LockManagerImpl.getWriteLock(LockManagerImpl.java:67)\n\tat org.neo4j.kernel.impl.locking.community.CommunityLockClient.acquireExclusive(CommunityLockClient.java:147)\n\tat org.neo4j.kernel.impl.api.LockingStatementOperations.exclusiveOptimisticLock(LockingStatementOperations.java:539)\n\tat org.neo4j.kernel.impl.api.LockingStatementOperations.acquireExclusiveNodeLock(LockingStatementOperations.java:500)\n\tat org.neo4j.kernel.impl.api.LockingStatementOperations.lockRelationshipNodes(LockingStatementOperations.java:307)\n\tat org.neo4j.kernel.impl.api.LockingStatementOperations.relationshipCreate(LockingStatementOperations.java:286)\n\tat org.neo4j.kernel.impl.api.OperationsFacade.relationshipCreate(OperationsFacade.java:854)\n\tat org.neo4j.cypher.internal.spi.v3_3.TransactionBoundQueryContext.createRelationship(TransactionBoundQueryContext.scala:109)\n\tat org.neo4j.cypher.internal.spi.v3_3.TransactionBoundQueryContext.createRelationship(TransactionBoundQueryContext.scala:70)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.ExceptionTranslatingQueryContext$$anonfun$createRelationship$2.apply(ExceptionTranslatingQueryContext.scala:200)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.ExceptionTranslatingQueryContext$$anonfun$createRelationship$2.apply(ExceptionTranslatingQueryContext.scala:200)\n\tat org.neo4j.cypher.internal.spi.v3_3.ExceptionTranslationSupport$class.translateException(ExceptionTranslationSupport.scala:32)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.ExceptionTranslatingQueryContext.translateException(ExceptionTranslatingQueryContext.scala:39)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.ExceptionTranslatingQueryContext.createRelationship(ExceptionTranslatingQueryContext.scala:200)\n\tat org.neo4j.cypher.internal.spi.v3_3.UpdateCountingQueryContext.createRelationship(UpdateCountingQueryContext.scala:92)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.BaseRelationshipPipe.org$neo4j$cypher$internal$compatibility$v3_3$runtime$pipes$BaseRelationshipPipe$$createRelationship(CreateRelationshipPipe.scala:47)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.BaseRelationshipPipe$$anonfun$internalCreateResults$1.apply(CreateRelationshipPipe.scala:41)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.BaseRelationshipPipe$$anonfun$internalCreateResults$1.apply(CreateRelationshipPipe.scala:41)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.EmptyResultPipe.internalCreateResults(EmptyResultPipe.scala:29)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.PipeWithSource.createResults(Pipe.scala:62)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.pipes.PipeWithSource.createResults(Pipe.scala:59)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.executionplan.DefaultExecutionResultBuilderFactory$ExecutionWorkflowBuilder.createResults(DefaultExecutionResultBuilderFactory.scala:102)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.executionplan.DefaultExecutionResultBuilderFactory$ExecutionWorkflowBuilder.build(DefaultExecutionResultBuilderFactory.scala:74)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.BuildInterpretedExecutionPlan$$anonfun$getExecutionPlanFunction$1.apply(BuildInterpretedExecutionPlan.scala:111)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.BuildInterpretedExecutionPlan$$anonfun$getExecutionPlanFunction$1.apply(BuildInterpretedExecutionPlan.scala:94)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.runtime.BuildInterpretedExecutionPlan$$anon$1.run(BuildInterpretedExecutionPlan.scala:62)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.Compatibility$ExecutionPlanWrapper$$anonfun$run$1.apply(Compatibility.scala:182)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.Compatibility$ExecutionPlanWrapper$$anonfun$run$1.apply(Compatibility.scala:178)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.exceptionHandler$runSafely$.apply(exceptionHandler.scala:90)\n\tat org.neo4j.cypher.internal.compatibility.v3_3.Compatibility$ExecutionPlanWrapper.run(Compatibility.scala:178)\n\tat org.neo4j.cypher.internal.PreparedPlanExecution.execute(PreparedPlanExecution.scala:29)\n\tat org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:116)\n\tat org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:111)\n\tat org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:76)\n\tat org.neo4j.server.rest.transactional.TransactionHandle.safelyExecute(TransactionHandle.java:371)\n\tat org.neo4j.server.rest.transactional.TransactionHandle.executeStatements(TransactionHandle.java:323)\n\tat org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:157)\n\tat org.neo4j.server.rest.web.TransactionalService.lambda$executeStatementsAndCommit$1(TransactionalService.java:218)\n\tat com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)\n\tat com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)\n\tat com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n\tat com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)\n\tat org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:748)\n"}]}

并发导入数据到neo4j时，时常给出这个LockDead的错误，官方没有明确的给出解决方案，但是有相关介绍【官方说deadlock】 ，上线前，想使用错误重试方案来补救并减少这中LockDead的出现几率，没有时间测试和实施（跟上头说，上头也是含糊其辞……不爽），只好强制将规则矩阵由并发改为串行，大量的体力活（因为商户数量众多，大概十家，这样的改动导致上线步骤繁琐，各种体力活）。
今天抽出空，大体跑了一下使用错误恢复补救方案，效果还可以，编码思路很简单，前期测试阶段给出三次重新错误重试次数，间隔步长为7、8、11秒，但是仍然会有极小的几率出现，是不是要改成无限重试的？还得继续观察。