Comparing Neo4j driver, py2neo and neo4jrestclient with some basic commands using the Panama Papers Data
1. Before we begin
In our last thrilling post, we installed Neo4j and downloaded the Panama Papers Data. Today, before diving into the dirty world of tax evasion, we want to benchmark the performance of 3 Python based modules. Namely Neo4j Python driver, py2neo and neo4jrestclient. If you haven’t done it already, install all of the modules by the following commands.
pip3 install neo4j-driver
pip3 install py2neo
pip3 install neo4jrestclient
Or whatever way you are accustomed to.
2. Loading the database to python
The first step, before doing anything, is to start Neo4j with the Panama papers data. If you forgot how to do this, please refer to our last post or check the “Benchmark.ipynb” in the following repository. It has all the necessary codes to replicate the experiment.
panama - Panama Papergithub.com
The next step is to load the data so that it is queryable from Python. In py2neo this is done with the following command.
from py2neo import Graph, Node, Relationship
gdb = Graph(user=”neo4j”, password=”YOURPASS")
Similarly in neo4jrestclient.
from neo4jrestclient.client import GraphDatabase
from neo4jrestclient import client
gdb2 = GraphDatabase(“http://localhost:7474", username=”neo4j”, password=”YOURPASS")
Finally in Neo4j Python driver.
from neo4j.v1 import GraphDatabase, basic_auth
driver = GraphDatabase.driver(“bolt://localhost:7687”, auth=basic_auth(“neo4j”, “YOURPASS”))
sess = driver.session()
3. Getting node labels and label-attribute pairs
The first thing we would like to do, when we encounter any new graph database, is to see what node label and relation types are there in the database. So the first thing we would do in our experiment is to get all the distinct node labels and all the associated attributes for each node labels.
In py2neo this is performed with the following code which takes about 100 ms. I am grad to see that py2neo has an built-in object which stores the node label and its attributes.
INPUT CODE py2neo:
# Get Distinct Node Labels
NodeLabel = list(gdb.node_labels)
print(NodeLabel)
# For each node type print attributes
Node = []
Attr = []
for nl in NodeLabel:
for i in gdb.schema.get_indexes(nl):
Node.append(nl)
Attr.append(format(i))
NodeLabelAttribute = pd.DataFrame(
{‘NodeLabel’: Node,’Attribute’: Attr})
NodeLabelAttribute.head(5)
However things get a little bit more nasty with neo4jrestclient and Neo4j Python driver. For neo4jrestclient it does have a way to access the node label but not the attributes. This means that we have to query it from our graph database. Not surprisingly this querying step takes quite a lot of time resulting in about 12sec for neo4jrestclient.
INPUT CODE neo4jrestclient:
# Get Distinct Node Labels
def extract(text):
import re
matches=re.findall(r'\'(.+?)\'',text)
return(",".join(matches))
NodeLabel = [extract(str(x)) for x in list(gdb2.labels)]
print(NodeLabel)
# For each node label print attributes
Node, Attr = ([] for i in range(2))
for nl in NodeLabel:
q = "MATCH (n:" + str(nl) + ")\n" + "RETURN distinct keys(n)"
temp = list(gdb2.query(q))
temp = list(set(sum(sum(temp,[]),[])))
for i in range(len(temp)):
Node.append(nl)
Attr.extend(temp)
NodeLabelAttribute = pd.DataFrame(
{'NodeLabel': Node,'Attribute': Attr})
NodeLabelAttribute.head(5)
For the Neo4j Python driver you have to query the node labels as well resulting in 20 sec.
INPUT CODE Neo4j Python Driver:
q = “””
MATCH (n)
RETURN distinct labels(n)
“””
res = sess.run(q)
NodeLabel = []
for r in res:
temp = r[“labels(n)”]
if temp != “”:
NodeLabel.extend(temp)
NodeLabel = list(filter(None, NodeLabel))
# For each node label print attributes
Node, Attr = ([] for i in range(2))
for nl in NodeLabel:
q = “MATCH (n:” + str(nl) + “)\n” + “RETURN distinct keys(n)”
res = sess.run(q)
temp = []
for r in res:
temp.extend(r[“keys(n)”])
temp2 = list(set(temp))
Attr.extend(temp2)
for i in range(len(temp2)):
Node.append(nl)
NodeLabelAttribute = pd.DataFrame(
{‘NodeLabel’: Node,’Attribute’: Attr})
NodeLabelAttribute.head(5)
4. Relation types and length of each edge list
The next thing we would like to do is make a list of all the relation types in the database and see which relation type has the longest edge list.
In py2neo this could be performed with the following code. This takes about 4min.
# Get Distinct Relation Types
RelaType = sorted(list(gdb.relationship_types))
print("There are " + str(len(RelaType)) + " relations in total")
# Calculate lengh of edge list for each types
res = []
for i in range(len(RelaType)):
#for i in range(10):
q = "MATCH (n)-[:`" + RelaType[i] + "`]-(m)\n" + "RETURN count(n)"
res.append(gdb.data(q)[0]["count(n)"])
RelaType = pd.DataFrame({'RelaType': RelaType[:len(res)],'count(n)': res})
RelaType.head(5)
In neo4jrestclient, the same thing could be implemented by the following command. Note that again, since we do not have a built-in method to get distinct relation types in neo4jrestclient, we have to query it from our graph database first. In total this takes about 4min 21s so it’s slightly slower than py2neo.
INPUT CODE neo4jrestclient:
# Get Distinct Relations
q = “””
START r =rel(*)
RETURN distinct(type(r))
“””
RelaType = sorted(sum(list(gdb2.query(q)),[]))
print(“There are “ + str(len(RelaType)) + “ relations in total”)
res = []
for i in range(len(RelaType)):
q = “MATCH (n)-[:`” + RelaType[i] + “`]-(m)\n” + “RETURN count(n)”
res.append(gdb2.query(q)[0][0])
RelaType = pd.DataFrame({‘RelaType’: RelaType,’count(n)’: res})
RelaType
Things get even more tedious in Neo4j Python driver where we have to query the Relation Types as well. However according to the The following code it takes about 4 min 10 sec so the additional query of getting the list of relation types didn’t seem to hurt much.
INPUT CODE Neo4j Python Driver:
# Get Distinct Relations
q = “””
START r =rel(*)
RETURN distinct(type(r))
“””
RelaType = []
res = sess.run(q)
for r in res:
RelaType.append(r[“(type(r))”])
RelaType = sorted(RelaType)
print(“There are “ + str(len(RelaType)) + “ relations in total”)
res2 = []
for i in range(len(RelaType)):
#for i in range(10):
q = “MATCH (n)-[:`” + RelaType[i] + “`]-(m)\n” + “RETURN count(n)”
res = sess.run(q)
for r in res:
res2.append(r[“count(n)”])
RelaType = pd.DataFrame({‘RelaType’: RelaType[:len(res2)],’count(n)’: res2})
RelaType.head(5)
5. Calculate degree distribution of all nodes
So far so good. My first impression, before ever touching the three modules, was that py2neo is the more updated cool stuff. So it was good to see that py2neo was more user-friendly as well as well-performing. But as the following example shows, there seems to be situation where neo4jrestclient and Neo4j Python driver are much faster than py2neo.
In this experiment we would gather information concerning the degree distribution of all nodes in our graph database. In py2neo this could be performed with the following code. This take about 1min 14s.
INPUT CODE py2neo:
q = """
MATCH (n)-[r]-(m)
RETURN n.node_id,n.name, count(r)
ORDER BY count(r) desc
"""
res = gdb.data(q)
NodeDegree = pd.DataFrame(res)
NodeDegree.head(5)
OUTPUT
count(r) n.name n.node_id
0 37338 None 236724
1 36374 Portcullis TrustNet (BVI) Limited 54662
2 14902 MOSSACK FONSECA & CO. (BAHAMAS) LIMITED 23000136
3 9719 UBS TRUSTEES (BAHAMAS) LTD. 23000147
4 8302 CREDIT SUISSE TRUST LIMITED 23000330
In neo4jrestclient the same thing could be performed with the following code. Now this takes about 18 sec which is about 4 times faster than py2neo!
INPUT CODE neo4jrestclient:
q = """
MATCH (n)-[r]-(m)
RETURN n.node_id, n.name, count(r)
ORDER BY count(r) desc
"""
res = list(gdb2.query(q))
NodeDegree = pd.DataFrame(res)
NodeDegree.columns = ["n.node_id","n.name","count(r)"]
NodeDegree.head(5)
Same results holds for Neo4j Python driver which take about 25 sec.
INPUT CODE Neo4j Python Driver:
Match = “MATCH (n)-[r]-(m)\n”
Ret = [“n.node_id”,”n.name”,”count(r)”]
Opt = “ORDER BY count(r) desc”
q = Match + “RETURN “ + ‘, ‘.join(Ret) + “\n” + Opt
res = sess.run(q)
res2 = []
for r in res:
#for r in islice(res,5):
res2.append([r[x] for x in range(len(Ret))])
NodeDegree = pd.DataFrame(res2)
NodeDegree.columns = Ret
NodeDegree.head(5)
6. Conclusion
At the moment I am not sure where the difference comes from. Besides some cases where there is a built-in object which preserves some basic information, we are using exactly the same query and I think there shouldn’t be much difference in it.
For the positive side, as this post shows there aren’t much difference in the coding style among the three modules. After all we are using the same query language (i.e. Cypher) to send orders to Neo4j and it is not a pain in the ass to switch from one module to another.
My recommendation? Definitely py2no is not an option. Although it is user-friendly in many respects, it is too slow for counting queries. Neo4jrestclient is not bad, but sometimes it returns nested list structure which we have to deal with using some trick (e.g. “sum(temp,[])” which I want to avoid. So I think I would go with the Neo4j Python driver. After all it is the only official release supported by Neo4j. What is your recommendation?