Awesome Big Model Training

Contents

  • Training Parallelism
    • Pipeline Parallelism
      • [NeurIPS 2019] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
  • References

Training Parallelism

Pipeline Parallelism

[NeurIPS 2019] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

  • Huang, Yanping, et al. “Gpipe: Efficient training of giant neural networks using pipeline parallelism.” Advances in neural information processing systems 32 (2019).
  • blog: [NeurIPS 2019] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

References

  • Liu, Qinghua, and Yuxiang Jiang. “Dive into Big Model Training.” arXiv preprint arXiv:2207.11912 (2022).

你可能感兴趣的:(模型部署,大模型训练)