GraphFrames分析豆瓣用户及小组

上篇已经介绍了通过Spark Graphx构建图，在构图过程中要准备一个Long类型的字段作为节点的VertexID，而对于上篇中用到的数据集，显然增加了工作量(将oid[String类型]转换为Long)。
GraphFrames能够避免这样的操作。

GraphFrames

GraphFrames是以DataFrame为基础的构图组件。
(GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs. )
它的目标是保留GraphX功能的同时充分利用Spark DataFrames的优势来扩展GraphX的功能。其扩展功能包括motif 预测，DataFrame序列化以及高性能的图查询。
( It aims to provide both the functionality of GraphX and extended functionality taking advantage of Spark DataFrames. This extended functionality includes motif finding, DataFrame-based serialization, and highly expressive graph queries.)

利用GraphFrames进行构图

主要流程和上篇没太大差异。主要是利用GraphFrames构图时，节点DataFrames需要指明某列为id列(可以使String类型)，边DataFrames需要指明src、dst列(可以使String类型)。代码如下：

    //将RDD转为DataFrame
    val personsdf = sqlContext.createDataFrame(personsRDD).toDF("id", "name", "no", "groupno", "vertextype")
    val groupsdf = sqlContext.createDataFrame(groupsRDD).toDF("id", "name", "tags", "groupno", "vertextype").dropDuplicates("groupno")

    val groupsds = groupsdf.select("id", "groupno", "tags")
    val relation = personsdf.join(groupsds, personsdf("groupno") === groupsds("groupno")).toDF("src","personname","personno","groupno","personvertextype","dst","groupno","tags").select("src","dst","tags")

    val personsds = personsdf.dropDuplicates("id", "no").toDF("id", "name", "tags", "groupno", "vertextype")
    val unionds = personsds.union(groupsdf)

    val graph = GraphFrame(unionds, relation)

linkurious.js

linkurious.js是纯Javascript开发的js库。它利用Sigma.js进行结构数据的解析及可视化，其提供Canvas、WebGL、SVG三种渲染模式来渲染图的节点和边。
(Linkurious.js is developed in pure Javascript. It uses Sigma.js for its graph data structure and visualization engine, which provides both Canvas, WebGL and SVG renderers for nodes and edges. )

利用linkurious.js进行图的可视化

首先，需要将DataFrames输出成gexf格式的数据结构文件

    def graphFrametoGexf(g: GraphFrame) : String = {
      "\n" +
        "\n" +
        "  \n" +
        "    \n" +
        g.vertices.rdd.map(v => if(v.get(4)=="p")
        {"      \n        \n      \n"} else
        {"      \n        \n      \n"
        }).collect.mkString +
        "    \n" +
        "    \n" +
        g.edges.rdd.map(e =>
        "    \n").collect.mkString +
        "    \n" +
        "  \n" +
        ""

    val strgexf = graphFrametoGexf(graph)
    //println(strgexf)
    val writer = new PrintWriter(new File("D:\\Tomcat7.0\\webapps\\linkurious.js\\examples\\data\\4444.gexf"))
    writer.print(strgexf)
    writer.close()

其次，利用linkurious.js进行可视化

sigma.parsers.gexf('data/4444.gexf', {
  // container: 'graph-container',
  renderer: {
    container: document.getElementById('graph-container'),
    type: 'canvas'
  },
  settings: {
    edgeColor: 'default',
    defaultEdgeColor: '#ccc',
    animationsTime: 5000,
    drawLabels: true,
    labelThreshold: 5,
    scalingMode: 'inside',
    batchEdgesDrawing: true,
    hideEdgesOnMove: true,
    sideMargin: 1
  }
}, function(s) {
  s.graph.nodes().forEach(function (n) {
    if (!s.graph.degree(n.id)) {
      s.graph.dropNode(n.id);
    }
    else {
      n.x = Math.random();
      n.y = Math.random();
    }
  });
  s.refresh();

  // Configure the ForceLink algorithm:
  var fa = sigma.layouts.configForceLink(s, {
    worker: true,
    autoStop: true,
    background: true,
    scaleRatio: 30,
    gravity: 3,
    easing: 'cubicInOut'
  });

  // Bind the events:
  fa.bind('start stop', function(e) {
    console.log(e.type);
    document.getElementById('layout-notification').style.visibility = '';
    if (e.type == 'start') {
      document.getElementById('layout-notification').style.visibility = 'visible';
    }
  });

  // Start the ForceLink algorithm:
  sigma.layouts.startForceLink();
});

测试

利用200个用户及400个组进行构图测试，显示效果流畅。(注意：节点名字不能包括Unicode字符，否则linkurious.js无法解析)

关系图.png