深度学习-从零开始(2) - LinearRegression

本章背景

本章是来源于coursera课程 python-machine-learning中的作业2内容。

本章内容

多项式线性回归
决定系数 R2 (coefficient of determination) 的计算
ridge线性回归
lasso线性回归

参考

评价回归模型
r2_score为负数的问题探讨

0. Polynomial LinearRegression(多项式线性回归)

随机创建如下数据：

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split


np.random.seed(0)
n = 15
x = np.linspace(0,10,n) + np.random.randn(n)/5
y = np.sin(x)+x/6 + np.random.randn(n)/10


X_train, X_test, y_train, y_test = train_test_split(x, y, random_state=0)

创建特定维度的多项式线性回归：

使用degree = 1,3,6 and 9 来训练X_train，并随机生成一些预测数据验证拟合结果：

def test_1():
    from sklearn.linear_model import LinearRegression
    from sklearn.preprocessing import PolynomialFeatures

    xx_pred = np.linspace(0, 10, 100)

    degrees = [1, 3, 6, 9]

    results = []
    for index in range(0, 4):
        degree = degrees[index]
        regression = LinearRegression()
        featurizer = PolynomialFeatures(degree=degree)
        X_train_features = featurizer.fit_transform(X_train.reshape(-1, 1))
        regression.fit(X_train_features, y_train.reshape(-1, 1))
        xx_pred_features = featurizer.transform(xx_pred.reshape(-1, 1))
        yy_pred = regression.predict(xx_pred_features)
        # append the results
        results.append(yy_pred.reshape(1, -1))
        
    return np.concatenate(results)

# 将几个维度的多项式拟合曲线绘制出来
def plot_results(degree_predictions):
    import matplotlib.pyplot as plt
    plt.figure(figsize=(10,5))
    plt.plot(X_train, y_train, 'o', label='training data', markersize=10)
    plt.plot(X_test, y_test, 'o', label='test data', markersize=10)
    for i,degree in enumerate([1,3,6,9]):
        plt.plot(np.linspace(0,10,100), degree_predictions[i], alpha=0.8, lw=2, label='degree={}'.format(degree))
    plt.ylim(-1,2.5)
    plt.legend(loc=4)
    plt.show()

plot_results(test_1())

1. coefficient of determination (决定系数R2)

决定系数(coefficient of determination)的计算方法：

R2-score计算公式

R^{2期待值越接近1说明拟合越好，越接近0说明拟合结果差。同时根据公式可以发现，在实际计算中R}2是可能为负数的，通过简单计算，当R^2为负数时,说明:

R2-score为负数

说明拟合结果差到不如使用均值。
还是使用上面的数据，分别计算X_train 和 X_test的R^2:

def test_2():
    from sklearn.linear_model import LinearRegression
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.metrics.regression import r2_score

    # Your code here
    results = []
    model_cnt = 10
    result_train = []
    result_test = []

    for degree in range(0, model_cnt):
        linearRegression = LinearRegression()
        featurizer = PolynomialFeatures(degree=degree)
        X_train_features = featurizer.fit_transform(X_train.reshape(-1, 1))
        y_train_features = y_train.reshape(-1, 1)
        linearRegression.fit(X_train_features, y_train_features)
        y_train_pred = linearRegression.predict(X_train_features)

        print('X_train r2_score ==> : ', r2_score(y_train_features, y_train_pred))
        result_train.append(r2_score(y_train_features, y_train_pred))

        # calculate the values for Test set
        X_test_features = featurizer.transform(X_test.reshape(-1, 1))
        y_test_pred = linearRegression.predict(X_test_features)
        y_test_features = y_test.reshape(-1, 1)
        print('X_test r2-score ==> : ', r2_score(y_test_features, y_test_pred))
        result_test.append(r2_score(y_test_features, y_test_pred))

    results.append(np.array(result_train).reshape(10, ))
    results.append(np.array(result_test).reshape(10, ))
    print(results[0].shape, results[1].shape)
    print(results)
    return results# Your answer here

从代码中也可以看出来，在sklearn工具包中已经包含了r2_score的计算函数，直接传入真实值和预测值即可计算，当然可以通过如上公式自行计算。

2. Ridge线性回归

Ridge线性回归是改良后的最小二乘法, 是有偏估计的回归方法, 即给损失函数加上一个正则化项, 也叫惩罚项(L2范数),防止过拟合。

Ridge回归的特点是以损失部分信息、降低精度为代价获得回归系数更为符合实际、更可靠的回归方法，对病态数据的拟合要强于最小二乘法。

Ridge的公式如下

Ridge回归损失函数

其中, m是样本量, n是特征数, λ是惩罚项参数(其取值大于0), 加惩罚项主要为了让模型参数的取值不能过大. 当λ趋于无穷大时, 对应βj趋向于0, 而βj表示的是因变量随着某一自变量改变一个单位而变化的数值(假设其他自变量均保持不变), 这时, 自变量之间的共线性对因变量的影响几乎不存在, 故其能有效解决自变量之间的多重共线性问题, 同时也能防止过拟合.

代码示例：

    import numpy as np
    import pandas as pd

    from sklearn.model_selection import train_test_split
    from matplotlib.pyplot import MultipleLocator
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.linear_model import Lasso, LinearRegression, Ridge
    from sklearn.metrics.regression import r2_score
    
    np.random.seed(0)
    n = 15
    x = np.linspace(0, 10, n) + np.random.randn(n) / 5
    y = np.sin(x) + x / 6 + np.random.randn(n) / 10

    X_train, X_test, y_train, y_test = train_test_split(x, y, random_state=0)



    # Your code here
    normalLinearRegression = LinearRegression()
    ridgeLinearRegression = Ridge(alpha=0.01, max_iter=10000, solver='svd')

    featurizer = PolynomialFeatures(degree=12)
    X_train_features = featurizer.fit_transform(X_train.reshape(-1, 1))
    X_test_features = featurizer.transform(X_test.reshape(-1, 1))
    # train the non-regularized linearRegression model
    normalLinearRegression.fit(X_train_features, y_train.reshape(-1, 1))
    y_test_pred_normal = normalLinearRegression.predict(X_test_features)
    # cal the R2 score for non-regularized model
    r2_score_normal = r2_score(y_test.reshape(-1, 1), y_test_pred_normal)
    print('non-regularized linearRegression r2-score ==> ', r2_score_normal)

    # train the ridge linearRegression model
    ridgeLinearRegression.fit(X_train_features, y_train.reshape(-1, 1))
    y_test_pred_ridge = ridgeLinearRegression.predict(X_test_features)
    # cal the R2-score for Ridge-regression
    r2_score_ridge = r2_score(y_test.reshape(-1, 1), y_test_pred_ridge)
    print('ridge-regularized linearRegression r2-score ==> ', r2_score_ridge)

Ridge回归并不能解决减少系数/维度数量的问题，其系数均不为0。

3. LASSO线性回归

模型参数越多复杂度越高，即使用线性回归依然有很多的参数需要训练，并且这也会造成一定程度上的过拟合。并且最终得到的模型的可解释性也不高。这个时候可以考虑引入lasso回归。

Lasso回归的特点是可以在拟合训练数据的同时进行变量选择（Variable Selection）。那么它是通过什么机制选择的呢？答案就是：正则化（Regularization）！或者可以简单的把这个东西叫做惩罚项。

LASSO的公式如下

Lasso回归损失函数