LASSO近端梯度下降法Proximal Gradient Descent公式推导及代码

LASSO by Proximal Gradient Descent

Prepare:
准备：

from itertools import cycle
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import lasso_path, enet_path
from sklearn import datasets
from copy import deepcopy

X = np.random.randn(100,10)
y = np.dot(X,[1,2,3,4,5,6,7,8,9,10])

Proximal Gradient Descent Framework近端梯度下降算法框架

1. randomly set ${\beta }^{(0)}$ for iteration 0
2. For $k$th iteration:
   ----Compute gradient $\nabla f({\beta }^{(k-1)})$
   ----Set $z={\beta }^{(k-1)}-\frac{1}{L}\nabla f({\beta }^{(k-1)})$
   ----Update ${\beta }^{(k)}=\text{sgn}(z)\cdot \text{max}[|z|-\frac{\lambda }{L},0]$
   ----Check convergence: if yes, end algorithm; else continue update
   Endfor

Here $f(\beta )=\frac{1}{2N}(Y-X\beta {)}^T(Y-X\beta )$, and $\nabla f(\beta )=-\frac{1}{N}{X}^T(Y-X\beta )$,
where the size of $X$, $Y$, $\beta$ is $N\times p$, $N\times 1$, $p\times 1$, which means $N$ samples样本 and $p$ features特征. Parameter $L\ge 1$ can be chosen, and $\frac{1}{L}$ can be considered as step size步长.

Proximal Gradient Descent Details近端梯度下降细节推导

Consider optimization problem:
现考虑优化问题：
$${\text{min}}_{x}f(x)+\lambda \cdot g(x),$$
where $x\in {\mathbb{R}}^{p\times 1}$, $f(x)\in \mathbb{R}$. And $f(x)$ is differentiable convex function, and $g(x)$ is convex but may not differentiable.
$f(x)$是可微凸函数，$g(x)$是凸函数但不一定可微。

For $f(x)$, according to Lipschitz continuity, for $\forall x,y$, there always exists a constant $L$ s.t.
对于$f(x)$，根据利普希茨连续性，对于任意$x,y$，一定存在常数$L$使得满足
$$|f^{\prime }(y)-f^{\prime }(x)|\le L|y-x|.$$
Then this problem can be solved using Proximal Gradient Descent.
可以用近似梯度下降来解决这种问题。

Denote $x^{(k)}$ as the $k$th updated result for $x$, then for $x\to x^{(k)}$, the proximation of $f(x)$ can be a function of $x$ and $x^{(k)}$:
用 $x^{(k)}$来表示 $x$的第$k$次更新后的结果，则对于 $x$逼近$x^{(k)}$时，$f(x)$的近似可以用$x$ 和 $x^{(k)}$的函数来表示：
$$\begin{gathered} \hat{f}(x,x^{(k)})=f(x^{(k)})+\nabla f^T(x^{(k)})(x-x^{(k)})+\frac{L}{2}{\Vert x-x^{(k)}\Vert }^2 \\ =\frac{L}{2}[x-(x^{(k)}-\frac{1}{L}\nabla f(x^{(k)})){]}^2+\text{CONST} \end{gathered}$$
where $\text{CONST}$ is not related with $x$, so can be ignored.
其中$\text{CONST}$是与$x$无关的项，可被忽略。

The original solution for $x$ at iteration $k+1$ is
$x$的第$k+1$次迭代的原始解为
$$x^{(k+1)}={\text{argmin}}_x\{f(x)+\lambda \cdot g(x)\},$$
In proximal gradient method, we use below instead:
在近端梯度下降法中，我们可用下式替代：
$$\begin{gathered} x^{(k+1)}={\text{argmin}}_x\{\hat{f}(x,x^{(k)})+\lambda \cdot g(x)\} \\ ={\text{argmin}}_x\{\frac{L}{2}[x-(x^{(k)}-\frac{1}{L}\nabla f(x^{(k)})){]}^2+\text{CONST}+\lambda \cdot g(x)\}. \end{gathered}$$

Given $g(x)={\Vert x\Vert }_1$, and let $z=x^{(k)}-\frac{1}{L}\nabla f(x_k)$, we have
$$x^{(k+1)}={\text{argmin}}_x\{\frac{L}{2}{\Vert x-z\Vert }^2+\lambda {\Vert x\Vert }_1\}.$$

To solve Equation above, let $F(x)=\frac{L}{2}{\sum }_{j=1}^p(x_j-z_j{)}^2+\lambda {\sum }_{j=1}^p|x_j|$. For $j$th element in $x$, it’s easy to know that the optimal $x_j^{\ast }$ satisfies
为了求解上面的等式，令 $F(x)=\frac{L}{2}{\sum }_{j=1}^p(x_j-z_j{)}^2+\lambda {\sum }_{j=1}^p|x_j|$。对于$x$的第$j$个分量/元素，易知最优解$x_j^{\ast }$满足
$$\frac{\partial F(x)}{\partial x_j}=L(x_j-z_j)+\lambda \cdot \text{sgn}(x_j)=0,$$
and we can get
$$z_j=x_j+\frac{\lambda }{L}\text{sgn}(x_j)$$.

The goal is to express $x_j^{\ast }$ as a function of $z_j$.This can be done by swapping the $x$-$z$ axes of the plot of $z_j=x_j+\frac{\lambda }{L}\text{sgn}(x_j)$ :
我们的目标是将$x_j^{\ast }$ 表达为 $z_j$的函数。这可以通过交换$z_j=x_j+\frac{\lambda }{L}\text{sgn}(x_j)$图像中的$x$-$z$轴来做到：

Then $x_j$ can be expressed as
然后 $x_j$可以被表示为
$$\begin{gathered} x_j=\text{sgn}(z_j)(|z_j|-\frac{\lambda }{L}{)}_{+} \\ =\text{sgn}(z_j)\cdot \text{max}\{|z_j|-\frac{\lambda }{L},0\} \end{gathered}$$

Simplified Code简化版代码

The code is Python version of proximal gradient descent from Stanford MATLAB LASSO demo.
这里的代码实现是Stanford MATLAB LASSO demo中近似梯度下降法的Python版本。

In the original code, to make sure the differentiable part of objective function $f(\beta )=\frac{1}{2N}{\Vert Y-X\beta \Vert }^2$ will decrease after each time weight update, there is a check to see whether the first-order approximation of $f(\beta )$ is smaller than the value from previous $\beta$:
在原始代码中，为了确保目标函数的可微部分$f(\beta )=\frac{1}{2N}{\Vert Y-X\beta \Vert }^2$在每次权重更新后会减小，需要检查$f(\beta )$的一阶近似是否小于前一次$\beta$带入后的结果：
$$\text{Check whether }\frac{1}{2N}{\Vert Y-X{\beta }^{(k)}\Vert }^2\le \frac{1}{2N}{\Vert Y-X{\beta }^{(k-1)}\Vert }^2+\nabla f^T({\beta }^{(k-1)})({\beta }^{(k)}-{\beta }^{(k-1)})+\frac{L}{2}{\Vert {\beta }^{(k)}-{\beta }^{(k-1)}\Vert }^2$$

def f(X, y, w):
    n_samples, _ = X.shape
    tmp = y - np.dot(X,w)
    return 2*np.dot(tmp, tmp) / n_samples

def objective(X,y,w,lmbd):
    n_samples, _ = X.shape
    tmp = y - np.dot(X,w)
    obj_v = 2 * np.dot(tmp,tmp) / n_samples + lmbd * np.sum(np.abs(w))
    return obj_v
    

def LASSO_proximal_gradient(X, y, lmbd, L=1, max_iter=1000, tol=1e-4):
    beta = 0.5 # used to update L for finding proper step size
    n_samples, n_features = X.shape
    w = np.empty(n_features, dtype=float)
    w_prev = np.empty(n_features, dtype=float) # store the old weights
    xty_N = np.dot(X.T, y) / n_samples
    xtx_N = np.dot(X.T, X) / n_samples
    prox_thres = lmbd / L
    h_prox_optval = np.empty(max_iter, dtype=float)
    for k in range(max_iter):
        while True:
            grad_w = np.dot(xtx_N, w)- xty_N
            z = w - grad_w/L
            w_tmp = np.sign(z) * np.maximum(np.abs(z)-prox_thres, 0)
            w_diff = w_tmp - w
            if f(X, y, w_tmp) <= f(X, y, w) + np.dot(grad_w, w_diff) + L/2 * np.sum(w_diff**2):
                break
            L = L / beta
        w_prev = copy(w)
        w = copy(w_tmp)
        
        h_prox_optval[k] = objective(X,y,w,lmbd)
        if k > 0 and abs(h_prox_optval[k] - h_prox_optval[k-1]) < tol:
            break
    return w, h_prox_optval[:k]

def pgd_lasso_path(X, y, lmbds):
    n_samples, n_features = X.shape
    pgd_coefs = np.empty((n_features, len(lmbds)), dtype=float)
    for i, lmbd in enumerate(lmbds):
        w, _ = LASSO_proximal_gradient(X, y, lmbd)
        pgd_coefs[:,i] = w
    return lmbds, pgd_coefs

experiment:
实验：

lmbds, pgd_coefs = pgd_lasso_path(X, y, my_lambdas)
# Display results
plt.figure(1)
colors = cycle(['b', 'r', 'g', 'c', 'k', 'y','m'])
neg_log_lambdas_lasso = -np.log10(my_lambdas)
for coef_l, c in zip(coefs_lasso, colors):
    l1 = plt.plot(neg_log_lambdas_lasso, coef_l, c=c)

plt.xlabel('-Log($\lambda$)')
plt.ylabel('coefficients')
plt.title('PGD Lasso Paths')
plt.axis('tight')

the result looks similar with that using Coordinate Descent Method.
结果与之前用CDM的结果相似。

Speed Comparison计算速度比较

print("Coordinate descent method from scikit-learn:")
%timeit lambdas_lasso, coefs_lasso, _ = lasso_path(X, y, eps, fit_intercept=False)

print("PGD method:")
%timeit lmbds, pgd_coefs = pgd_lasso_path(X, y, my_lambdas)

Results:
结果：

Coordinate descent method from scikit-learn:
4.88 ms ± 97 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

PGD method:
778 ms ± 21.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Pure python PGD is faster than that of coordinate descent method in my previous blog, but slower than Cython coordinate descent method.
纯Python版本的PGD要比本人之前博客中的CDM实现快，但是比Cython的CDM慢。

完整代码

放到GitHub上了，包含了前面coordinate descent method的实验