box-cox转换及变换参数lambda估算方法
我们进行数据转换的原因是：除了小样本可以考虑非参数，大部分的统计原理和参数检验都是基于正态分布推得。
关于box-cox转换的基础内容请看：BoxCox-变换方法及其实现运用.pptx
了解极大似然估计：极大似然估计思想的最简单解释
通过上面的内容可以知道，
- boxcox1p变换中y+c的+c是为了确保(y+c)>0,因为在boxcox变换中要求y>0
- python代码：
- y_boxcox = special.boxcox1p(y, lam_best) 利用llf获得优化后的lambda或boxcox_normmax(x) 得到优化后的lambda
boxcox_normmax(x)说明，详情见https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox_normmax.html
scipy.stats.boxcox_normmax(x, brack=(-2.0, 2.0), method='pearsonr')[source]
Compute optimal Box-Cox transform parameter for input data.
Parameters:
x : array_like Input array.
brack : 2-tuple, optional
The starting interval for a downhill bracket search with optimize.brent. Note that this is in most cases not critical; the final result is allowed to be outside this bracket.
method : str, optional
The method to determine the optimal transform parameter (boxcox lmbda parameter). Options are:
‘pearsonr’ (default)
Maximizes the Pearson correlation coefficient between y = boxcox(x) and the expected values for y if x would be normally-distributed.
‘mle’
Minimizes the log-likelihood boxcox_llf. This is the method used in boxcox. ()
‘all’
Use all optimization methods available, and return all results. Useful to compare different methods.
Returns:
maxlog : float or ndarray
The optimal transform parameter found. An array instead of a scalar for method='all'.
接下来，用kaggle中House Prices: Advanced Regression Techniques比赛的数据集做个练习。
scipy.stats.boxcox_llf使用详见https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox_llf.html
import pandas as pd
import numpy as np
from scipy import stats,special
import matplotlib.pyplot as plt
train = pd.read_csv('./data/train.csv')
y = train['SalePrice']
print(y.shape)
lam_range = np.linspace(-2,5,100) # default nums=50
llf = np.zeros(lam_range.shape, dtype=float)
# lambda estimate:
for i,lam in enumerate(lam_range):
llf[i] = stats.boxcox_llf(lam, y) # y 必须>0
# find the max lgo-likelihood(llf) index and decide the lambda
lam_best = lam_range[llf.argmax()]
print('Suitable lam is: ',round(lam_best,2))
print('Max llf is: ', round(llf.max(),2))
plt.figure()
plt.axvline(round(lam_best,2),ls="--",color="r")
plt.plot(lam_range,llf)
plt.show()
plt.savefig('boxcox.jpg')
# boxcox convert:
print('before convert: ','\n', y.head())
#y_boxcox = stats.boxcox(y, lam_best)
y_boxcox = special.boxcox1p(y, lam_best)
print('after convert: ','\n', pd.DataFrame(y_boxcox).head())
# inverse boxcox convert:
y_invboxcox = special.inv_boxcox1p(y_boxcox, lam_best)
print('after inverse: ', '\n', pd.DataFrame(y_invboxcox).head())
结果如下，
比外，也可以通过scipy.stats.boxcox_normplot确定lambda，详见http://scipy.github.io/devdocs/generated/scipy.stats.boxcox_normplot.html