深度学习并行训练——AD-PSGD算法

结合AD-PSGD算法，浅谈深度学习并行训练中的基本概念

引用

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Geoffrey Hinton ： https://www.coursera.org/learn/neuralnetworks/home/week/6

Wikipedia ： https://en.wikipedia.org/wiki/Stochastic_gradient_descent

Geoffrey Hinton ： https://www.coursera.org/learn/neuralnetworks/home/week/6

http://engineering.skymind.io/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks

Qirong Ho and Eric P. Xing： Big ML Software for Modern ML Algorithms

Large Scale Distributed Deep Networks

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

http://engineering.skymind.io/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks

Li Mu : http://www.cs.cmu.edu/~muli/file/mu_defense_animation.pdf

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent https://arxiv.org/abs/1705.09056

Asynchronous Decentralized Parallel Stochastic Gradient Descent https://arxiv.org/abs/1710.06952

Decentralized Distributed Deep Learning https://github.com/LearningSys/sosp17/blob/master/papers/14%5CCameraReadySubmission%5Cmain_camera_ready.pdf