交叉熵 (Cross Entropy)

交叉熵 (Cross Entropy)

Machine Learning 很大一部分二分类任务的 loss function 使用 Cross Entrophy.，正常前面会有$sigmoid$激活函数，具体前向和反先可以参考前面我的博客：https://blog.csdn.net/jmu201521121021/article/details/86658163

公式

$$P(y|x)=-\frac{1}{m}\sum _{i=1}^m{y}_i\ast log(y_i)+(1-y_i)log(1-y_i)(1)$$

• $y_i$ 表示$i^{th}样本$标注好的标签
• $y_i$ 表示$i^{th}$样本输出预测值

推导

• $y_i\in [0,1]$，-1代表负类标签，1代表正类标签
• when $y=1$,$P(y|x)=y$ 。when $y=0,P(y|x)=1-y$,
• 所以 可以写成 :
  $$P(y|x)=y^y\ast (1-y{)}^{1-y}(2)$$
• 由最大似然估计可得 :
  $$P(y1,y2,...|x_1,x_2,...)=\prod _{i=1}^{m}P(y_i|x_i)=\prod _{i=1}^m{y}_i^{y_i}\ast (1-y_i{)}^{1-y_i}(3)$$
• 由于乘法对于后面反向传播求导不方便求解，所以加个log，相乘变为相加，公式(3)变为 :
  $$P(y|x)=\sum _{i=1}^m{y}_i\ast log(y_i)+(1-y_i)log(1-y_i)(4)$$
• 为了让loss表示小点，前面加个 $\frac{1}{m},有取平均效果$，由于目标是为了取到loss最小，前面再加个$-1$
  ，即公式(4)变为:$$P(y|x)=-\frac{1}{m}\sum _{i=1}^m{y}_i\ast log(y_i)+(1-y_i)log(1-y_i)(5)$$

代码实现

# GRADED FUNCTION: compute_cost

def compute_cost(AL, Y):
    """
    Implement the cost function defined by equation (7).

    Arguments:
    AL -- probability vector corresponding to your label predictions, shape (1, number of examples)
    Y -- true "label" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)

    Returns:
    cost -- cross-entropy cost
    """
    
    m = Y.shape[1]

    # Compute loss from aL and y.
    ### START CODE HERE ### (≈ 1 lines of code)
    cost = -1.0 / m * np.sum( Y * np.log(AL) + (1 - Y) * np.log(1 - AL)) 
    ### END CODE HERE ###
    
    cost = np.squeeze(cost)      # To make sure your cost's shape is what we expect (e.g. this turns [[17]] into 17).
    assert(cost.shape == ())
    
    return cost