AutoML NAS 调研（CVPR2019）

1. [论文极简笔记] Efficient Neural Architecture Search via Parameter Sharing
2. [论文极简笔记]FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
3. [论文极简笔记] Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
4. [论文极简笔记] Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
5. [论文极简笔记] Searching for A Robust Neural Architecture in Four GPU Hours
6. [论文极简笔记] NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
7. [论文极简笔记] Weight Agnostic Neural Networks

---

1 背景

AutoML是个很宽泛的概念，宽泛到任何机器学习任务都可以理论上用AutoML完成，这就带来了很大的想象空间。最近做了一个关于此的调研工作，调研重点放在了NAS网络搜索上，因为这是目前研究的热点，在最新的CVPR2019上，该领域发表了14篇文章。

题目	PDF	源码
MFAS: Multimodal Fusion Architecture Search	PDF	GITHUB
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection	PDF	GITHUB
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation(Oral)	PDF	GITHUB
MnasNet: Platform-Aware Neural Architecture Search for Mobile	PDF	GITHUB
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search(Oral)	PDF	GITHUB
RENAS: Reinforced Evolutionary Neural Architecture Search	PDF	GITHUB
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells	PDF	GITHUB
Searching for a Robust Neural Architecture in Four GPU Hours	PDF	GITHUB
IRLAS: Inverse Reinforcement Learning for Architecture Search	PDF	GITHUB
Customizable Architecture Search for Semantic Segmentation	PDF	GITHUB
EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching From Scratch	PDF	GITHUB
A Neurobiological Evaluation Metric for Neural Network Model Search	PDF	GITHUB
MFAS: Multimodal Fusion Architecture Search	PDF	GITHUB
Transferable AutoML by Model Sharing Over Grouped Datasets	PDF	GITHUB

---

2 AutoML 定义\分类

3 AutoML map

4 具体问题

4.1 自动特征工程

4.2 自动模型选择

picking up some classifiers and setting their corresponding hyper-parameters

4.3 自动优化算法选择

contains the choice of optimization
algorithms and the values of their hyper-parameters

4.4 全范围（重点NAS）

机器学习传统方法就不介绍了，因为研究的人实在太少，下面主要介绍Full Scope 中NAS相关的工作。如上面是两篇祖师爷论文，他们的槽点也是很突出，首先是计算资源，TPU集群，几千个GPU可不是小厂能玩得起的，这背后其实也有网络搜索空间的问题，因此后面的早期工作（不完全）大多致力于解决下面1.2两个问题：

1. 减少计算，加速搜索, 最先探索的有《Efficient architecture search by network transformation》(AAAI18)、《Efficent neural architecture search via parameters sharing》(PMLR18)
2. 减少搜索空间, 最早的工作有《Learning transferable architectures for scalabel image recognition》(CVPR18)
3. 此外还有一些工作探讨网络搜索的其它问题：《Searching for activation functions》(ICLA18)、《Autoaugment: Learning augmentation policies from data》（arxiv 18）

4.4.1 手工搭建网络vs自动网络搜索

4.4.2 NAS MAP

4.4.3 搜索空间

4.4.3.1 Global Search Space

全局搜索容易理解，即除去输入输出层，其他层都在搜索范围内，而其搜索空间又可以细分为三种：

1. sequential search space
2. sequential with skips
3. architecture template
   可见上图最右© ，只有深蓝色节点及其连接操作才是可搜索的。

4.4.3.1.1 Sequential

这里是引用

4.4.3.1.2 branch

4.4.3.1.3 Architecture template

4.4.3.2 Cell-Based Search Space

三个例子