Focal Loss 论文理解及公式推导

原文：Focal Loss 论文理解及公式推导 - AIUAI

题目: Focal Loss for Dense Object Detection - ICCV2017

作者: Tsung-Yi, Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar

团队: FAIR

精度最高的目标检测器往往基于 RCNN 的 two-stage 方法，对候选目标位置再采用分类器处理. 而，one-stage 目标检测器是对所有可能的目标位置进行规则的(regular)、密集采样，更快速简单，但是精度还在追赶 two-stage 检测器. <论文所关注的问题于此.>

论文发现，密集检测器训练过程中，所遇到的极端前景背景类别不均衡(extreme foreground-background class imbalance)是核心原因.

对此，提出了 Focal Loss，通过修改标准的交叉熵损失函数，降低对能够很好分类样本的权重(down-weights the loss assigned to well-classified examples)，解决类别不均衡问题.

Focal Loss 关注于在 hard samples 的稀疏子集进行训练，并避免在训练过程中大量的简单负样本淹没检测器.

Focal Loss 是动态缩放的交叉熵损失函数，随着对正确分类的置信增加，缩放因子(scaling factor) 衰退到 0. 如图：

Focal Loss 的缩放因子能够动态的调整训练过程中简单样本的权重，并让模型快速关注于困难样本(hard samples).

基于 Focal Loss 的 RetinaNet 的目标检测器表现.

1. Focal Loss

Focal Loss 旨在解决 one-stage 目标检测器在训练过程中出现的极端前景背景类不均衡的问题(如，前景：背景 = 1:1000).

首先基于二值分类的交叉熵(cross entropy, CE) 引入 Focal Loss：
$$CE(p,y)=\{\begin{array}{ll}-log(p)& \text{if }y=1\\ -log(1-p)& \text{otherwise }\end{array}$$
其中，$y\in \{+1-1\}$ 为 groundtruth 类别；$p\in [0,1]$ 是模型对于类别 $y=1$ 所得到的预测概率.

符号简介起见，定义 $p_t$：
$$p_t=\{\begin{array}{ll}p& \text{if }y=1\\ 1-p& \text{otherwise }\end{array}$$
则，$CE(p,y)=CE(p_t)=-log(p_t)$.

CE Loss 如图 Figure 1 中的上面的蓝色曲线所示. 其一个显著特点是，对于简单易分的样本($p_t\gg 0.5$)，其 loss 也是一致对待. 当累加了大量简单样本的 loss 后，具有很小 loss 值的可能淹没稀少的类(rare class).

1.1 均衡交叉熵 Blanced CE

解决类别不均衡的一种常用方法是，对类别 +1 引入权重因子 $\alpha \in [0,1]$，对于类别 -1 引入权重 $1-\alpha$.

符号简介起见，定义 ${\alpha }_t$：
$${\alpha }_t=\{\begin{array}{ll}\alpha & \text{if }y=1\\ 1-\alpha & \text{otherwise }\end{array}$$
则，$\alpha$-balanced CE loss 为：

$CE(p_t)=-{\alpha }_{t}log(p_t)$

1.2 Focal Loss 定义

虽然 $\alpha$ 能够平衡 positive/negative 样本的重要性，但不能区分 easy/had 样本.

对此，Focal Loss 提出将损失函数降低 easy 样本的权重，并关注于对 hard negatives 样本的训练.

添加调制因子(modulating factor) $(1-p_t{)}^{\gamma }$ 到 CE loss，其中 $\gamma \ge 0$ 为可调的 focusing 参数.

Focal Loss 定义为：

$FL(p_t)=-(1-p_t{)}^{\gamma }log(p_t)$

如图 Figure 1，给出了 $\gamma \in [0,5]$ 中几个值的可视化.

Focal Loss 的两个属性：

• [1] - 当样本被误分，且 $p_t$ 值很小时，调制因子接近于 1，loss 不受影响. 随着 $p_t\to 1$，则调制因子接近于 0，则容易分类的样本的损失函数被降低权重.
• [2] - focusing 参数 $\gamma$ 平滑地调整哪些 easy 样本会被降低权重的比率(rate). 当 $\gamma =0$，FL=CE；随着 $\gamma $ 增加，调制因子的影响也会随之增加(实验中发现 $\gamma =2$ 效果最佳.)

直观上，调制因子能够减少 easy 样本对于损失函数的贡献，并延伸了loss 值比较地的样本范围.

例如，$\gamma =0.2$ 时，被分类为 $p_t=0.9$ 的样本，与 CE 相比，会减少 100x 倍；而且，被分类为 $p_t \approx 0.968 $ 的样本，与 CE 相比，会有少于 1000x 倍的 loss 值. 这就自然增加了将难分类样本的重要性(如 $\gamma =2$ 且 $p_t\le 0.5$ 时，难分类样本的 loss 值会增加 4x 倍.)

实际上，论文采用了 Focal Loss 的 $\alpha$ -balanced 变形：

$FL(p_t)=-{\alpha }_t(1-p_t{)}^{\gamma }log(p_t)$

1.3. Focal Loss 例示

Focal Loss 并不局限于具体的形式. 这里给出另一种例示.

假设 $p=\sigma (x)=\frac{1}{1+e^{-x}}$，

定义 $p_t$为(类似于前面对于 $p_t$ 的定义)：
$$p_t=\{\begin{array}{ll}p& \text{if }y=1\\ 1-p& \text{otherwise }\end{array}$$
定义：$x_t=yx$，其中，$y\in \{+1,-1\}$ 是 groundtruth 类别.

则：$p_t=\sigma (x_t)=\frac{1}{1+e^{yx}}$

当 $x_t>0$ 时，样本被正确分类，此时 $p_t>0.5$.

有：
$$\frac{d{p}_t}{dx}=\frac{-1}{(1+e^{yx}{)}^2}\ast y\ast e^{yx}=y\ast p_t\ast (1-p_t)=-y\ast p_t\ast (p_t-1)$$
对于交叉熵损失函数 $CE(p_t)=-log(p_t)$，由$\frac{dlnx}{dx}=\frac{1}{x}$，
$$\frac{dCE(p_t)}{dx}=\frac{dCE(p_t)}{d{p}_t}\ast \frac{d{p}_t}{dx}=(-\frac{1}{p_t})\ast (-y\ast p_t\ast (p_t-1))=y\ast (p_t-1)$$
对于 Focal Loss $FL(p_t)=-(1-p_t{)}^{\gamma }log(p_t)$，其中 $\gamma$ 为常数.
$$\frac{dFL(p_t)}{dx}=\frac{d(1-p_t{)}^{\gamma }}{dx}\ast (-log(p_t))+(1-p_t{)}^{\gamma }\ast \frac{dCE(p_t)}{dx}$$

$$\frac{dFL(p_t)}{dx}=(\gamma \ast (1-p_t{)}^{\gamma -1}\ast \frac{d(1-p_t)}{d{p}_t})\ast \frac{d{p}_t}{dx}\ast (-log(p_t))+(1-p_t{)}^{\gamma }\ast y\ast (p_t-1)$$

$$\frac{dFL(p_t)}{dx}=(\gamma \ast (1-p_t{)}^{\gamma -1}\ast (-1))\ast (-y\ast p_t\ast (p_t-1))\ast (-log(p_t))+y\ast (1-p_t{)}^{\gamma }\ast (p_t-1)$$

$$\frac{dFL(p_t)}{dx}=\gamma \ast (1-p_t{)}^{\gamma }\ast y\ast p_t\ast log(p_t)+y\ast (1-p_t{)}^{\gamma }\ast (p_t-1)$$

$$\frac{dFL(p_t)}{dx}=y\ast (1-p_t{)}^{\gamma }\ast (\gamma \ast p_t\ast log(p_t)+(p_t-1))$$

再者，假设 $p_t^{\ast }=\sigma (\gamma x_t+\beta )$，则 $F{L}^{\ast }(p_t^{\ast })=-log(p_t^{\ast })/\gamma$，其中 $\gamma$ 为常数.
$$\frac{dF{L}^{\ast }(p_t^{\ast })}{dx}=-\frac{1}{p_t^{\ast }}\ast \frac{1}{\gamma }\ast \frac{d{p}_t^{\ast }}{d(\gamma x_t+\beta )}\ast \frac{d(\gamma x_t+\beta )}{dx}$$

$$\frac{dF{L}^{\ast }(p_t^{\ast })}{dx}=-\frac{1}{p_t^{\ast }}\ast \frac{1}{\gamma }\ast (-y\ast p_t^{\ast }\ast (p_t^{\ast }-1)\ast \gamma )=y\ast (p_t^{\ast }-1)$$

则，$F{L}^{\ast }$ 包含两个参数 $\gamma$ 和 $\beta$，控制着 loss 曲线的陡度(steepness) 和移动(shift). 如 Figure 5.

1.4. Focal Loss 求导

$CE$ 关于 $x$ 的求导：

$\frac{dCE}{dx}=y(p_t-1)$

$FL$ 关于 $x$ 的求导：

$\frac{dFL}{dx}=y(1-p_t{)}^{\gamma }(\gamma p_{t}log(p_t)+p_t-1)$

$F{L}^{\ast }$ 关于 $x$ 的求导：

$\frac{dF{L}^{\ast }}{dx}=y(p_t^{\ast }-1)$

如图 Figure 6. 三种 loss 函数，对于high-confidence 的预测结果，其导数都趋近于 -1 或 0.

但，与 $CE$ 不同的是，$FL$ 和 $F{L}^{\ast }$ 的有效设置时，只要 $x_t>0$，二者的导数都是很小的.

2. SoftmaxFocalLoss 求导

Focal Loss 损失函数：
$$FL(p_t)=-\alpha (1-p_t{)}^{\gamma }log(p_t)$$
其中：
$$p_t=\{\begin{array}{ll}p& \text{if }y=1\\ 1-p& \text{otherwise }\end{array}$$
Softmax 函数：
$$p_i=\frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}$$

其中，$K$ 为类别数，$x$ 是网络全连接层等的输出向量，$x_i$ 是向量的第 $i$ 个元素值.

则 $FL$ 关于 $x$ 求导：
$$\frac{dFL}{d{x}_i}=\frac{dFL}{d{p}_i}\ast \frac{d{p}_i}{d{x}_i}$$

而，
$$\frac{dFL}{d{p}_t}=-\alpha (\frac{d(1-p_t{)}^{\gamma }}{d{p}_t}\ast log(p_t)+(1-p_t{)}^{\gamma }\ast \frac{d(log(p_t))}{d{p}_t})$$

$$\frac{dFL}{d{p}_t}=-\alpha (-\gamma \ast (1-p_t{)}^{\gamma -1}\ast log(p_t)+(1-p_t{)}^{\gamma }\ast \frac{1}{p_t})$$

Softmax 函数关于 x 的求导为：
$$\frac{d{p}_i}{d{x}_i}=\frac{d\frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}}{d{x}_i}$$

$$\frac{d{p}_i}{d{x}_i}=\frac{\frac{d(e^{x_i})}{d{x}_i}\ast {\sum }_{k=1}^K{e}^{x_k}-e^{x_i}\ast \frac{d({\sum }_{k=1}^K{e}^{x_k})}{d{x}_i}}{({\sum }_{k=1}^K{e}^{x_k}{)}^2}$$

当 $i=j$ 时，
$$\frac{d{p}_i}{d{x}_i}=\frac{e^{x_i}\ast {\sum }_{k=1}^K{e}^{x_k}-e^{x_i}\ast e^{x_i}}{({\sum }_{k=1}^K{e}^{x_k}{)}^2}$$

$$\frac{d{p}_i}{d{x}_i}=\frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}-\frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}\ast \frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}$$

$$\frac{d{p}_i}{d{x}_i}=p_i-p_i\ast p_i=p_i(1-p_i)$$

当 $i̸=j$ 时，
$$\frac{d{p}_i}{d{x}_i}=\frac{0-e^{x_i}\ast e^{x_j}}{({\sum }_{k=1}^K{e}^{x_k}{)}^2}$$

$$\frac{d{p}_i}{d{x}_i}=-\frac{e^{x_i}}{{\sum }_{k=1}^K{e}^{x_k}}\ast \frac{e^{x_j}}{{\sum }_{k=1}^K{e}^{x_k}}$$

$$\frac{d{p}_i}{d{x}_i}=-p_i\ast p_j$$

Softmax 的函数求导即为：
$$\frac{d{p}_i}{d{x}_i}=\{\begin{array}{ll}{p}_i(1-p_i)& \text{if }i=j\\ -p_i\ast p_j& \text{if }i̸=j\end{array}$$

故：
$$
\frac{d FL}{d x_i} = \begin{cases}

• \alpha (- \gamma * (1-p_i)^{\gamma - 1} * log(p_i) + (1-p_i)^{\gamma} * \frac{1}{p_i}) * p_i(1-p_i) &\text{if } i=j \
• \alpha (- \gamma * (1-p_i)^{\gamma - 1} * log(p_i) + (1-p_i)^{\gamma} * \frac{1}{p_i}) * (-p_i*p_j) &\text{if } i \neq j
  \end{cases}
  $$

$$\frac{dFL}{d{x}_i}=\{\begin{array}{ll}\alpha (-\gamma \ast (1-p_i{)}^{\gamma -1}\ast log(p_i)p_i+(1-p_i{)}^{\gamma })\ast (p_i-1)& \text{if }i=j\\ \alpha (-\gamma \ast (1-p_i{)}^{\gamma -1}\ast log(p_i)p_i+(1-p_i{)}^{\gamma })\ast p_j& \text{if }i̸=j\end{array}$$

3. Pytorch 实现

FocalLoss-PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F

class FocalLoss(nn.Module):
    def __init__(self, alpha=0.25, gamma=2, size_average=True):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = torch.Tensor([gamma])
        self.size_average = size_average
        if isinstance(alpha, (float, int, long)):
            if self.alpha > 1:
                raise ValueError('Not supported value, alpha should be small than 1.0')
            else:
                self.alpha = torch.Tensor([alpha, 1.0 - alpha])
        if isinstance(alpha, list): self.alpha = torch.Tensor(alpha)
        self.alpha /= torch.sum(self.alpha)

    def forward(self, input, target):
        if input.dim() > 2:
            input = input.view(input.size(0), input.size(1), -1)  # [N,C,H,W]->[N,C,H*W] ([N,C,D,H,W]->[N,C,D*H*W])
        # target
        # [N,1,D,H,W] ->[N*D*H*W,1]
        if self.alpha.device != input.device:
            self.alpha = torch.tensor(self.alpha, device=input.device)
        target = target.view(-1, 1)
        logpt = torch.log(input + 1e-10)
        logpt = logpt.gather(1, target)
        logpt = logpt.view(-1, 1)
        pt = torch.exp(logpt)
        alpha = self.alpha.gather(0, target.view(-1))

        gamma = self.gamma

        if not self.gamma.device == input.device:
            gamma = torch.tensor(self.gamma, device=input.device)

        loss = -1 * alpha * torch.pow((1 - pt), gamma) * logpt
        if self.size_average:
            loss = loss.mean()
        else:
            loss = loss.sum()
        return loss

4. Keras 实现

keras-focal-loss

基于 Keras 和 TensorFlow 后端实现的 Binary Focal Loss 和 Categorical/Multiclass Focal Loss.

主要设计两个参数：alpha 和 gamma.

用法：

model.compile(optimizer='adam', loss=categorical_focal_loss(gamma=2.0, alpha=0.25), metrics=['accuracy'])

实现：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Fri Oct 19 08:20:58 2018

@OS: Ubuntu 18.04
@IDE: Spyder3
@author: Aldi Faizal Dimara (Steam ID: phenomos)
"""

import keras.backend as K
import tensorflow as tf

def categorical_focal_loss(gamma=2.0, alpha=0.25):
    """
    Implementation of Focal Loss from the paper in multiclass classification
    Formula:
        loss = -alpha*((1-p)^gamma)*log(p)
    Parameters:
        alpha -- the same as wighting factor in balanced cross entropy
        gamma -- focusing parameter for modulating factor (1-p)
    Default value:
        gamma -- 2.0 as mentioned in the paper
        alpha -- 0.25 as mentioned in the paper
    """
    def focal_loss(y_true, y_pred):
        # Define epsilon so that the backpropagation will not result in NaN
        # for 0 divisor case
        epsilon = K.epsilon()
        # Add the epsilon to prediction value
        #y_pred = y_pred + epsilon
        # Clip the prediction value
        y_pred = K.clip(y_pred, epsilon, 1.0-epsilon)
        # Calculate cross entropy
        cross_entropy = -y_true*K.log(y_pred)
        # Calculate weight that consists of  modulating factor and weighting factor
        weight = alpha * y_true * K.pow((1-y_pred), gamma)
        # Calculate focal loss
        loss = weight * cross_entropy
        # Sum the losses in mini_batch
        loss = K.sum(loss, axis=1)
        return loss
    
    return focal_loss

def binary_focal_loss(gamma=2.0, alpha=0.25):
    """
    Implementation of Focal Loss from the paper in multiclass classification
    Formula:
        loss = -alpha_t*((1-p_t)^gamma)*log(p_t)
        
        p_t = y_pred, if y_true = 1
        p_t = 1-y_pred, otherwise
        
        alpha_t = alpha, if y_true=1
        alpha_t = 1-alpha, otherwise
        
        cross_entropy = -log(p_t)
    Parameters:
        alpha -- the same as wighting factor in balanced cross entropy
        gamma -- focusing parameter for modulating factor (1-p)
    Default value:
        gamma -- 2.0 as mentioned in the paper
        alpha -- 0.25 as mentioned in the paper
    """
    def focal_loss(y_true, y_pred):
        # Define epsilon so that the backpropagation will not result in NaN
        # for 0 divisor case
        epsilon = K.epsilon()
        # Add the epsilon to prediction value
        #y_pred = y_pred + epsilon
        # Clip the prediciton value
        y_pred = K.clip(y_pred, epsilon, 1.0-epsilon)
        # Calculate p_t
        p_t = tf.where(K.equal(y_true, 1), y_pred, 1-y_pred)
        # Calculate alpha_t
        alpha_factor = K.ones_like(y_true)*alpha
        alpha_t = tf.where(K.equal(y_true, 1), alpha_factor, 1-alpha_factor)
        # Calculate cross entropy
        cross_entropy = -K.log(p_t)
        weight = alpha_t * K.pow((1-p_t), gamma)
        # Calculate focal loss
        loss = weight * cross_entropy
        # Sum the losses in mini_batch
        loss = K.sum(loss, axis=1)
        return loss
    
    return focal_loss    

Related

[1] - Focal Loss 的前向与后向公式推导