【机器学习】机器学习系统SysML 阅读表
SysML reading list
Review
A Berkeley View of Systems Challenges for AI
https://arxiv.org/pdf/1712.05855.pdf
Strategies and Principles of Distributed Machine Learning on Big Data
https://arxiv.org/abs/1512.09295
Background
Deep learning
Nature volume 521, 2015
https://www.nature.com/articles/nature14539
Deep learning reading list
http://deeplearning.net/reading-list
Measurement
Multi-tenant GPU Clusters for Deep LearningWorkloads: Analysis and Implications
https://www.microsoft.com/en-us/research/uploads/prod/2018/05/gpu_sched_tr.pdf
Frameworks
TensorFlow: A System for Large-Scale Machine Learning
OSDI 2016
https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
Ray: A Distributed Framework for Emerging AI Applications
OSDI 2018
https://www.usenix.org/system/files/osdi18-moritz.pdf
Tuning
HyperDrive: Exploring Hyperparameters with POP Scheduling
MiddleWare 2017
https://dl.acm.org/citation.cfm?id=3135994
Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads
VLDB 2018
http://www.vldb.org/pvldb/vol11/p607-li.pdf
Automating Model Search for Large Scale Machine Learning
SoCC 2015
http://dl.acm.org/authorize?N91362
Google Vizier: A Service for Black-Box Optimization
KDD 2017
https://dl.acm.org/citation.cfm?id=3098043
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Journal of Machine Learning Research 18 (2018)
https://arxiv.org/pdf/1603.06560.pdf
Hyperopt: a Python library for model selection and hyperparameter optimization
Computational Science & Discovery, 8(1) 2015
http://iopscience.iop.org/article/10.1088/1749-4699/8/1/014008
Auto-Keras: Efficient Neural Architecture Search with Network Morphism
https://arxiv.org/pdf/1806.10282v2.pdf
Runtime execution
Cavs: An Efficient Runtime System for Dynamic Neural Networks
ATC 2018
https://www.usenix.org/system/files/conference/atc18/atc18-xu-shizhen.pdf
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
OSDI 2018
https://www.usenix.org/system/files/osdi18-chen.pdf
PipeDream: Fast and Efficient Pipeline Parallel DNN Training
https://arxiv.org/pdf/1806.03377.pdf
STRADS: A Distributed Framework for Scheduled Model Parallel Machine Learning
EuroSys 2016
https://dl.acm.org/citation.cfm?id=2901331
Dynamic Control Flow in Large-Scale Machine Learning
EuroSys 2018
https://dl.acm.org/citation.cfm?id=3190551
Improving the Expressiveness of Deep Learning Frameworks with Recursion
EuroSys 2018
https://dl.acm.org/citation.cfm?id=3190530
Continuum: A Platform for Cost-Aware, Low-Latency Continual Learning
SoCC 2018
https://dl.acm.org/citation.cfm?id=3267817
KeystoneML: Optimizing Pipelines for Large-ScaleAdvanced Analytics
ICDE 2017
https://amplab.cs.berkeley.edu/wp-content/uploads/2017/01/ICDE_2017_CameraReady_475.pdf
Owl: A General-Purpose Numerical Library in OCaml
https://arxiv.org/pdf/1707.09616.pdf
Distributed learning
Large Scale Distributed Deep Networks
NIPS 2012
https://ai.google/research/pubs/pub40565.pdf
Managed Communication and Consistency for Fast Data-Parallel Iterative Analytics
SoCC 2015
http://dl.acm.org/authorize?N91363
Ako: Decentralised Deep Learning with Partial Gradient Exchange
SOCC 2016
https://lsds.doc.ic.ac.uk/sites/default/files/ako-socc16.pdf
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
ATC 2017
https://www.usenix.org/system/files/conference/atc17/atc17-zhang.pdf
Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
SoCC 2018
https://dl.acm.org/citation.cfm?id=3267840
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
ML Systems Workshop at NIPS 2016
https://arxiv.org/pdf/1512.01274.pdf
Scaling Distributed Machine Learning with the Parameter Server
OSDI 2014
https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-li_mu.pdf
Project Adam: Building an Efficient and Scalable Deep Learning Training System
OSDI 2014
https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf
Orpheus: Efficient Distributed Machine Learning via System and Algorithm Co-design
SoCC 2018
https://dl.acm.org/citation.cfm?id=3267810
Petuum: A New Platform for Distributed Machine Learning on Big Data
KDD 2015
https://arxiv.org/pdf/1312.7651.pdf
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
https://arxiv.org/pdf/1811.06965.pdf
Serving systems and inference
DeepCPU: Serving RNN-based Deep Learning Models 10x Faster
ATC 2018
https://www.usenix.org/system/files/conference/atc18/atc18-zhang-minjia.pdf
Clipper: A Low-Latency Online Prediction Serving System
NSDI 2017
https://www.usenix.org/system/files/conference/nsdi17/nsdi17-crankshaw.pdf
Research for Practice: Prediction-Serving Systems
ACM Queue 16(1), 2018
https://queue.acm.org/detail.cfm?id=3210557
InferLine: ML Inference Pipeline Composition
https://arxiv.org/pdf/1812.01776.pdf
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
OSDI 2018
https://www.usenix.org/system/files/osdi18-lee.pdf
Olympian: Scheduling GPU Usage in a Deep Neural Network Model Serving System
MiddleWare 2018
https://dl.acm.org/citation.cfm?id=3274813
Low Latency RNN Inference with Cellular Batching
EuroSys 2018
https://dl.acm.org/citation.cfm?id=3190541
SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism
SC 2016
https://ieeexplore.ieee.org/document/7877104
NoScope: Optimizing Neural Network Queries over Video at Scale
VLDB 2017
https://dl.acm.org/citation.cfm?id=3137664
Scheduling
Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters
EuroSys 2018
https://dl.acm.org/citation.cfm?id=3190517
SLAQ: Quality-Driven Scheduling for Distributed Machine Learning
SoCC 2017
https://dl.acm.org/authorize?N46878
Proteus: agile ML elasticity through tiered reliability in dynamic resource markets
EuroSys 2017
https://dl.acm.org/citation.cfm?id=3064182
Gandiva: Introspective Cluster Scheduling for Deep Learning
OSDI 2018
https://www.usenix.org/system/files/osdi18-xiao.pdf
Topology-Aware GPU Scheduling for Learning Workloads in Cloud Environments
SC 2017
https://dl.acm.org/citation.cfm?id=3126933
Algorithmic aspects in scalable ML
Hemingway: Modeling Distributed Optimization Algorithms
ML Systems Workshop at NIPS 2016
https://arxiv.org/pdf/1702.05865.pdf
Asynchronous Methods for Deep Reinforcement Learning
ICML 2016
http://proceedings.mlr.press/v48/mniha16.pdf
Don’t Use Large Mini-Batches, Use Local SGD
https://arxiv.org/pdf/1808.07217.pdf
GeePS: Scalable deep learning on distributed GPUs with a GPU-specialized parameter server
EuroSys 2016
https://dl.acm.org/citation.cfm?id=2901323
ImageNet Training in Minutes
ICPP 2018
https://dl.acm.org/citation.cfm?id=3225069
Semantics-Preserving Parallelization of Stochastic Gradient Descent
IPDPS 2018
https://ieeexplore.ieee.org/abstract/document/8425176
HOGWILD!: A Lock-Free Approach to ParallelizingStochastic Gradient Descent
NIPS 2011
https://papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-gradient-descent.pdf
QSGD: Communication-Efficient SGD via Randomized Quantization
NIPS 2017
https://papers.nips.cc/paper/6768-qsgd-communication-efficient-sgd-via-gradient-quantization-and-encoding.pdf
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
NIPS 2017
https://papers.nips.cc/paper/7117-can-decentralized-algorithms-outperform-centralized-algorithms-a-case-study-for-decentralized-parallel-stochastic-gradient-descent.pdf
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
AIStats 2018
https://arxiv.org/pdf/1803.01113.pdf
Probabilistic Synchronous Parallel
https://arxiv.org/pdf/1709.07772.pdf
AI Testing and Verification
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
SOSP 2017
https://dl.acm.org/authorize?N47145
Programmatically Interpretable Reinforcement Learning
ICML 2018
https://arxiv.org/pdf/1804.02477.pdf
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation
SP 2018
https://ieeexplore.ieee.org/document/8418593
Interpretability and Explainability
“Why Should I Trust You?”Explaining the Predictions of Any Classifier
KDD 2016
https://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
ICML 2018
https://arxiv.org/pdf/1802.07814.pdf
A Unified Approach to Interpreting Model Predictions
NIPS 2017
https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
The Mythos of Model Interpretability
WHI 2016
https://arxiv.org/pdf/1606.03490.pdf
Model Management
MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis
SIGMOD 2018
https://dl.acm.org/citation.cfm?id=3196934
MODELDB: A System for Machine Learning Model Management
HILDA 2016
https://mitdbg.github.io/modeldb/papers/hilda_modeldb.pdf
Model Governance: Reducing the Anarchy of Production ML
ATC 2018
https://www.usenix.org/system/files/conference/atc18/atc18-sridhar.pdf
The Missing Piece in Complex Analytics: Low Latency,Scalable Model Management and Serving with Velox
CIDR 2015
http://www.bailis.org/papers/velox-cidr2015.pdf
Bandana: Using Non-volatile Memory for Storing Deep Learning Models
SysML 2019
https://arxiv.org/abs/1811.05922
Hardware
Deep learning with limited numerical precision
ICML 2015
http://proceedings.mlr.press/v37/gupta15.pdf
In-Datacenter Performance Analysis of a Tensor Processing Unit
ISCA 2017
https://dl.acm.org/citation.cfm?id=3080246
Serving DNNs in Real Timeat Datacenter Scale with Project Brainwave
IEEE MICRO 38(2), Mar./Apr. 2018
https://ieeexplore.ieee.org/document/8344479
Security aspects
Efficient Deep Learning on Multi-Source Private Data
https://arxiv.org/pdf/1807.06689.pdf
Chiron: Privacy-preserving Machine Learning as a Service
https://arxiv.org/pdf/1803.05961.pdf
MLCapsule: Guarded Offline Deployment of Machine Learning as a Service
https://arxiv.org/pdf/1808.00590.pdf
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
https://arxiv.org/pdf/1806.03287.pdf
Privado: Practical and Secure DNN Inference
https://arxiv.org/pdf/1810.00602.pdf
ML Platforms (Applied)
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
HPCA 2018
https://research.fb.com/publications/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/
Machine Learning at Facebook: Understanding Inference at the Edge
HPCA 2019
https://research.fb.com/publications/machine-learning-at-facebook-understanding-inference-at-the-edge/
Meet Michelangelo: Uber’s Machine Learning Platform
https://eng.uber.com/michelangelo/
Introducing FBLearner Flow: Facebook’s AI backbone
https://code.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/
TFX: A TensorFlow-Based Production-Scale Machine LearningPlatform
http://dl.acm.org/authorize?N33328
Horovod: fast and easy distributed deep learning in TensorFlow
https://arxiv.org/pdf/1802.05799v3.pdf
ML for Systems
Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms
SOSP 2017
https://dl.acm.org/authorize?N47144
Adaptive Execution of Continuous and Data-intensive Workflows with Machine Learning
MiddleWare 2018
https://dl.acm.org/citation.cfm?id=3274827
AuTO: Scaling Deep Reinforcement Learning to Enable Datacenter-Scale Automatic Traffic Optimization
SIGCOMM 2018
https://dl.acm.org/citation.cfm?id=3230551
Neural Adaptive Video Streaming with Pensieve
SIGCOMM 2017
https://dl.acm.org/citation.cfm?id=3098843
Neural Adaptive Content-aware Internet Video Delivery
OSDI 2018
https://www.usenix.org/system/files/osdi18-yeo.pdf
Workshops
Systems for ML and Open Source Software Workshop at NeurIPS 2018
http://learningsys.org/nips18/acceptedpapers.html
SysML 2018
http://www.sysml.cc/2018/index.html
Engineering Dependable and Secure Machine Learning Systems 2019
https://sites.google.com/view/edsmls2019/program
Engineering Dependable and Secure Machine Learning Systems 2018
https://sites.google.com/edu.haifa.ac.il/edsmls/program
Workshop on Distributed Machine Learning 2017
https://distributedml2017.wordpress.com/schedule/
ML Systems Workshop at NIPS 2016
https://sites.google.com/site/mlsysnips2016/accepted-papers
Upcoming 2019
ColumnML: Column Store Machine Learning with On The Fly Data Transformation
VLDB 2019
Continuous Integration of Machine Learning Models: A Rigorous Yet Practical Treatment
SysML 2019
Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices
ASPLOS 2019
RLgraph: Flexible Computation Graphs for Deep Reinforcement Learning
SysML 2019
https://arxiv.org/pdf/1810.09028.pdf