算法的归纳偏置

机器学习和深度学习算法在学习过程中对某种类型的假设有归纳偏好，这种归纳偏好可以看作学习算法自身针对假设空间的“价值观”。奥卡姆剃刀原则要求若多个假设与观察一致，选最简单的那个，但哪个是“最简单”的假设需要不简单的判断。归纳偏好对应了学习算法本身做出的关于“什么样的模型更好”的假设，具体问题中算法的归纳偏好是否与问题本身匹配，大多数时候决定了算法能否取得好的性能。“没有免费的午餐”定理（$NFL$）说明任何一个学习算法都能找出一种场景使它的性能很低或很高，任何学习算法的期望性能都相同，这就使得针对具体问题采用合适的学习算法很关键。

归纳和演绎是自然科学的两大方法，归纳指的是从一些例子中寻找共性，形成一个能泛化到很多问题的通用规则，偏好是对模型的偏好。归纳产生模型空间，偏好约束到某个具体的模型

下面是深度学习的归纳偏置是什么？中提到的文章《Relational inductive biases, deep learning, and graph networks》的笔记，第一个回答很优秀：

在深度学习时代，这种归纳性偏好更为明显。比如深度神经网络结构就偏好性的认为，层次化处理信息有更好效果；卷积神经网络认为信息具有空间局部性（locality），可以用滑动卷积共享权重方式降低参数空间；反馈神经网络则将时序信息考虑进来强调顺序重要性；图网络则是认为中心节点与邻居节点的相似性会更好引导信息流动。可以说深度学习时代，纷繁的网络结构创新就体现了不同的归纳性偏。更强的归纳性偏好其实可以提升样本的利用效率，或者说减少计算代价。比如alphago由于面向围棋这种方形棋盘，用CNN就很合适。但是拿这套框架去打星际争霸，就还是得加上RNN去考虑时序处理逻辑。所以人脑智力的确很神奇，一方面就一个大脑却可以适应不同的任务，貌似model-free，另一方面人类在处理诸多信息时，又有很强的预先偏好性，可以快速决断。这种偏好也是诸多错觉或者误判的来源。

• In particular, generalizing beyond one’s experiences—a hallmark of human intelligence from infancy—remains a formidable challenge for modern AI.
• Just as biology uses nature and nurture cooperatively, we reject the false choice between “hand-engineering” and “end-to-end” learning, and instead advocate for an approach which benefits from their complementary strengths.
• We explore how using relational inductive biases within deep learning architectures can facilitate learning about entities, relations, and rules for composing them.
• We discuss how graph networks can support relational reasoning and combinatorial generalization, laying the foundation for more sophisticated, interpretable, and flexible patterns of reasoning.
• That is, the world is compositional, or at least, we understand it in compositional terms. When learning, we either fit new knowledge into our existing structured representations, or adjust the structure itself to better accommodate (and make use of) the new and the old.
• A key reason why structured approaches were so vital to machine learning in previous eras was, in part, because data and computing resources were expensive, and the improved sample complexity afforded by structured approaches’ strong inductive biases was very valuable.
• In contrast with past approaches in AI, modern deep learning methods often follow an “end-to-end” design philosophy which emphasizes minimal a priori representational and computational assumptions, and seeks to avoid explicit structure and “hand-engineering”.

Introduction前半部分：传统机器学习因为样本数少和计算资源昂贵，往往需要通过特征工程和各种假设来提高样本的表达能力，比如一个多肽序列样本就可以算出成千上万个描述符，或者假设属性对预测结果影响独立；现代深度学习讲究的是端到端的学习，尽量减少先验和假设。另一方面人脑实际上是通过一种复合的方式理解这个世界，不是仅靠先验假设（structured representations），也不是仅靠通过样本来获得所有信息。