L ( t ) = ∑ i = 1 n l ( y i , y ^ i ( t − 1 ) + f t ( x i ) ) + Ω ( f t ) L^{(t)} = \sum_{i=1}^n l(y_i, \hat y_i^{(t-1)} + f_t (x_i) ) + \Omega(f_t) L(t)=i=1∑nl(yi,y^i(t−1)+ft(xi))+Ω(ft)
损失函数二阶泰勒展开
L ( t ) = ∑ i = 1 n [ l ( y i , y ^ ( t − 1 ) ) + g i f t ( x i ) + 1 2 h i f t 2 ( x i ) ] + Ω ( f t ) L^{(t)} = \sum_{i=1}^n [ l(y_i, \hat y^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) ] + \Omega(f_t) L(t)=i=1∑n[l(yi,y^(t−1))+gift(xi)+21hift2(xi)]+Ω(ft),其中 g i = ∂ y ^ ( t − 1 ) l ( y i , y ^ ( t − 1 ) ) g_i = \partial_{\hat y^{(t-1)}}l(y_i, \hat y^{(t-1)}) gi=∂y^(t−1)l(yi,y^(t−1)), h i = ∂ y ^ ( t − 1 ) 2 l ( y i , y ^ ( t − 1 ) ) h_i = \partial^2_{\hat y^{(t-1)}}l(y_i, \hat y^{(t-1)}) hi=∂y^(t−1)2l(yi,y^(t−1))
去掉当中的常数项
L ( t ) = ∑ i = 1 n [ g i f t ( x i ) + 1 2 h i f t 2 ( x i ) ] + Ω ( f t ) L^{(t)} = \sum_{i=1}^n [ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) ] + \Omega(f_t) L(t)=i=1∑n[gift(xi)+21hift2(xi)]+Ω(ft)
将正则化项展开
L ( t ) = ∑ i = 1 n [ g i f t ( x i ) + 1 2 h i f t 2 ( x i ) ] + γ T + 1 2 λ ∑ j = 1 T w j 2 = ∑ j = 1 T [ ( ∑ i ∈ I j g i ) w j + 1 2 ( ∑ i ∈ I j h i + λ ) w j 2 ] + γ T L^{(t)} = \sum_{i=1}^n [ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) ] + \gamma T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2 \\ = \sum_{j=1}^T[ (\sum_{i \in I_j} g_i)w_j + \frac{1}{2} (\sum_{i \in I_j} h_i + \lambda)w_j^2 ] + \gamma T L(t)=i=1∑n[gift(xi)+21hift2(xi)]+γT+21λj=1∑Twj2=j=1∑T[(i∈Ij∑gi)wj+21(i∈Ij∑hi+λ)wj2]+γT
其中 I j = { i ∣ q ( x i ) = j } I_j = \{i |q(x_i) = j \} Ij={i∣q(xi)=j} 表示一个叶子节点 j; q ( x i ) = j q(x_i)=j q(xi)=j 表示一个样本 x i x_i xi经过树结构 q ( x ) q(x) q(x)到达了叶子节点 j j j
那么对于任何一个固定的树结构 q ( x ) q(x) q(x),我们都能够得到叶子节点的最优的权重 w j ∗ w_j^* wj∗,以及对应的最小损失
w j ∗ = − ∑ i ∈ I j g i ∑ i ∈ I j h i + λ w_j^* = - \frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda} wj∗=−∑i∈Ijhi+λ∑i∈Ijgi
L ( t ) ( q ) = − 1 2 ∑ j = 1 T ( ∑ i ∈ I j g i ) 2 ∑ i ∈ I j h i + λ + γ T L^{(t)}(q) = - \frac{1}{2} \sum_{j=1}^T \frac{(\sum_{i \in I_j}g_i)^2}{ \sum_{i \in I_j} h_i + \lambda } + \gamma T L(t)(q)=−21j=1∑T∑i∈Ijhi+λ(∑i∈Ijgi)2+γT
这个损失函数有多种用途,例如用来做节点的切分
L s p l i t = 1 2 [ ( ∑ i ∈ I L g i ) 2 ∑ i ∈ I L h i + λ + ( ∑ i ∈ I R g i ) 2 ∑ i ∈ I R h i + λ − ( ∑ i ∈ I g i ) 2 ∑ i ∈ I h i + λ ] + γ T L_{split} = \frac{1}{2} [ \frac{(\sum_{i \in I_L}g_i)^2}{ \sum_{i \in I_L} h_i + \lambda } + \frac{(\sum_{i \in I_R}g_i)^2}{ \sum_{i \in I_R} h_i + \lambda } - \frac{(\sum_{i \in I}g_i)^2}{ \sum_{i \in I} h_i + \lambda } ]+ \gamma T Lsplit=21[∑i∈ILhi+λ(∑i∈ILgi)2+∑i∈IRhi+λ(∑i∈IRgi)2−∑i∈Ihi+λ(∑i∈Igi)2]+γT