GNN图神经网络 DropEdge：towards deep GCN on node classfication

论文笔记：DropEdge: Towards Deep Graph Covulutional Networks on Node Classification

GNN可弥补深度学习的可推理和可解释的缺陷，然而，目前大部分图卷积网络，尤其是面向节点分类的网络，都是浅层网络。这些模型分类效果往往随着深度加深而变差（即使使用残差连接），这与用于图片分类的卷积网络动辄超过几十层的现象很不一致。图卷积神经网络变深之后难以训练的现象严重制约了其表达能力。所以，如何有效的训练超深图卷积网络是图学习研究的一个重大挑战。这项工作由腾讯 AI Lab 与清华大学合作完成。
腾讯 AI Lab 和清华大学的这项研究表明，图神经网络无法做深由两个原因造成：过拟合 (Overfitting) 和过平滑（Oversmoothing）。为了解决这两个问题，文章提出了一种简洁但非常有效的方法：随机删边技术，DropEdge，极大提升了超深图卷积网络的训练效率和节点分类效果。文章正式被接收后，图灵奖获得者 Yoshua Bengio 的团队成员 Petar Veličković（注意力图网络 GAT 的发明者）在 openreview 平台上也给与了关注与好评。
论文地址：https://openreview.net/forum?id=Hkx1qkrKPr
代码地址：https://github.com/DropEdge/DropEdge

摘要

过拟合和过平滑问题是深层GCN在node classfication上的应用的障碍。过拟合使泛化能力变弱，过平滑是指GNN消息传递过程中，所有节点的输入特征收敛到一个与输入无关的子空间的过程，导致输入GCN的特征失效，并造成梯度消失。（过平滑是GCN特有的问题）
Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well.

DropEdge

本质上，减小过平滑的影响，就是要增加-smoothing layer的层数，以及减少收敛到子空间的信息损失。基于此，本文设计了dropedge方法：在每轮训练时，随机去掉输入的图上的边，即将邻接矩阵中的非零元素置0，p是删边概率，得到随机删边后的邻接矩阵，正则化后代替原来的邻接矩阵。
这种随机删边技术，可以看作dropout在图结构上的推广，实现GNN的ensemble，也有防止overfitting的作用。

结果

及