Life Long Learning (LLL)

• 本文为李宏毅 2021 ML 课程的笔记

Life Long Learning, Continuous Learning, Never Ending Learning, Incremental Learning

Life Long Learning

Life Long Learning

What people think about AI …

• 通过让模型不断地学习新任务来不断增强模型的能力 $\to$ Even though I have learned task 2, I do not forget task 1.
  

Life Long Learning in real-world applications

Catastrophic Forgetting

• 但实际情况是，在训练过程中，模型会逐渐忘记之前学过的任务…
• Example:
  The network has enough capacity to learn both tasks.
  

Multi-task training

• Multi-task training can be considered as the upper bound of LLL. But can multi-task training solve the problem?
  
  • No:
    

Evaluation

• First of all, we need a sequence of tasks. (目前 LLL 研究的不同任务还是比较简单的)
  • e.g. permutation 表示用某种规则将数字打乱
    

---

Evaluation

• $R_{i,j}$: after training task $i$, performance on task $j$. If $i>j$, After training task $i$, does task $j$ be forgot. If $i<j$, Can we transfer the skill of task $i$ to task $j$
• 一般评估方法有以下两种:
  • (1) Accuracy
    $$\frac{1}{T}\sum _{i=1}^T{R}_{T,i}$$
  • (2) Backward Transfer (一般为负数)
    $$\frac{1}{T-1}\sum _{i=1}^{T-1}{R}_{T,i}-R_{i,i}$$

Research Directions

Selective Synaptic Plasticity (Regularization-based Approach)

Selective Synaptic Plasticity: 选择性的突触可塑性 (只让 NN 中的某些神经元间的连接具有可塑性，其余的必须被固化)

Why Catastrophic Forgetting?

---

• Basic Idea: Some parameters in the model are important to the previous tasks. Only change the unimportant parameters. ($\theta$ should be close to ${\theta }^b$ in certain directions.)
  
  • If $b_i=0$, there is no constraint on ${\theta }_i$ $\Rightarrow$ Catastrophic Forgetting
  • If $b_i=\infty$, ${\theta }_i$ would always be equal to ${\theta }_i^b$ $\Rightarrow$ Intransigence
    

SGD 表示正常训练，会导致 catastrophic forgetting；L2 将 $b_i$ 均设为 1，会导致 Intransigence

• 那么我们如何知道某一个参数对一个任务是否重要呢？大致思想是我们可以在该任务上训练一个模型，然后观察当某一个参数改变时，会不会对 loss 产生很大的影响，如果影响特别大，那么就认为该参数 ${\theta }_i$ 比较重要，它对应的 $b_i$ 也可以设定为一个比较大的值。每次在一个 task 上训练完，都不断对 $b_i$ 进行累加作为最终的 $b_i$:
  

---

计算 $b_i$ 的不同方法

• Elastic Weight Consolidation (EWC)
• Synaptic Intelligence (SI)
• Memory Aware Synapses (MAS)
• RWalk
• Sliced Cramer Preservation (SCP)

---

Gradient Episodic Memory (GEM)

• paper: Gradient Episodic Memory for Continual Learning
• GEM 也是一种 Selective Synaptic Plasticity 方法，但它的思路是结合之前任务数据集上的梯度来计算新的梯度，因此需要额外存储少量的之前任务的训练数据:
  

Additional Neural Resource Allocation

Progressive Neural Networks

• paper: Progressive Neural Networks

PackNet

• paper: PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
  

每个任务都只用部分参数

Compacting, Picking, and Growing (CPG)

• paper: Compacting, Picking and Growing for Unforgetting Continual Learning
• 结合了 Progressive Neural Networks 和 PackNet
  

Memory Reply

• paper:
  • Continual Learning with Deep Generative Replay
  • FearNet: Brain-Inspired Model for Incremental Learning
  • LAMOL: LAnguage MOdeling for Lifelong Language Learning

---

Idea: Generating Data

• Generating pseudo-data using generative model for previous tasks
  

To learn more…

Adding new classes

• Learning without Forgetting (LwF)
• iCaRL: Incremental Classifier and Representation Learning

要学习的不同任务所涉及的类别数不同

---

• Three scenarios for continual learning

---

Curriculum Learning

• what is the proper learning order?