bias & variance

原文出处：http://scott.fortmann-roe.com/docs/BiasVariance.html，感谢作者。

Conceptual Definition

• Error due to Bias: The error due to bias is taken as the difference between the expected (or average) prediction of our model and the correct value which we are trying to predict. Of course you only have one model so talking about expected or average prediction values might seem a little strange. However, imagine you could repeat the whole model building process more than once: each time you gather new data and run a new analysis creating a new model. Due to randomness in the underlying data sets, the resulting models will have a range of predictions. Bias measures how far off in general these models' predictions are from the correct value.

• Error due to Variance: The error due to variance is taken as the variability of a model prediction for a given data point. Again, imagine you can repeat the entire model building process multiple times. The variance is how much the predictions for a given point vary between different realizations of the model.

Mathematical Definition

after Hastie, et al. 2009 ¹

If we denote the variable we are trying to predict as Y and our covariates as X, we may assume that there is a relationship relating one to the other such as Y=f(X)+ where the error term  is normally distributed with a mean of zero like so ～N(0,σ).

We may estimate a model f^(X) of f(X) using linear regressions or another modeling technique. In this case, the expected squared prediction error at a point x is:

Err(x)=E[(Yf^(x))2]

This error may then be decomposed into bias and variance components:

Err(x)=(E[f^(x)]f(x))2+E[f^(x)E[f^(x)]]2+σ2e

Err(x)=Bias2+Variance+Irreducible Error

That third term, irreducible error, is the noise term in the true relationship that cannot fundamentally be reduced by any model. Given the true model and infinite data to calibrate it, we should be able to reduce both the bias and variance terms to 0. However, in a world with imperfect models and finite data, there is a tradeoff between minimizing the bias and minimizing the variance.

总结：

bias：多次训练模型的训练误差，度量了某种学习算法的平均估计结果所能逼近学习目标的程度

variance：多次训练模型，判断结果P和判断结果的平均值P hat的差值，在面对同样规模的不同训练集时，学习算法的估计结果发生变动的程度（测试集，