scikit-learn：3.2. Grid Search: Searching for estimator parameters
参考：http://scikit-learn.org/stable/modules/grid_search.html
GridSearchCV通过（蛮力）搜索参数空间（参数的所有可能组合），寻找最好的 Cross-validation: evaluating estimator performance score对应的超参数（翻译文章参考：http://blog.csdn.net/mmc2015/article/details/47099275）。例如Support Vector Classifier的 C, kernel and gamma ，Lasso的alpha，etc。
A search consists of:
- an estimator (regressor or classifier such as sklearn.svm.SVC());
- a parameter space;
- a method for searching or sampling candidates;
- a cross-validation scheme
- a score function.
本节我们介绍 GridSearchCV、RandomizedSearchCV 、以及parameter search的小Tips，最后介绍蛮力搜索的alternatives。
1、Exhaustive Grid Search
GridSearchCV的参数param_grid定义搜索网格。
两个例子说明一切：
- See Parameter estimation using grid search with cross-validation for an example of Grid Search computation on the digits dataset.
- See Sample pipeline for text feature extraction and evaluation for an example of Grid Search coupling parameters from a text documents feature extractor (n-gram count vectorizer and TF-IDF transformer) with a classifier (here a linear SVM trained with SGD with either elastic net or L2 penalty) using a pipeline.Pipeline instance.
2、Randomized Parameter Optimization
RandomizedSearchCV 通过在参数可能的取值的某个分布中sample一组参数，好处是：可以设定独立于参数（及所有取值）具体数量的一个搜索次数；添加无效的参数也不会降低效率。
搜索的次数通过 n_iter 设定，对于每一个参数，如果是连续的取值，则通过一定的分布sample，如果是离散的取值，则通过uniform分布sample，例如：
scipy.stats module提供了很多用来sample参数的distributions，如expon, gamma, uniform or randint.
对于连续的参数，如 C ，一定要选择连续的分布来sample，并且适当增大 n_iter 一般会搜索到更好的参数组合。
给个例子：
- Comparing randomized search and grid search for hyperparameter estimation compares the usage and efficiency of randomized search and grid search.
3、Tips for parameter search（这几个建议非常靠谱。。。）
1）具体化目标函数
参数搜索默认使用score function（ 即，分类用sklearn.metrics.accuracy_score 回归用sklearn.metrics.r2_score ）来衡量参数的好坏对于有些应用（比如分类unbalance，score不是很好的标准），通过具体化GridSearchCV和RandomizedSearchCV 的scoring parameter。See The scoring parameter: defining model evaluation rules for more details.
2）综合estimators和parameter sapces（同时考虑预测器和参数空间）
Pipeline: chaining estimators describes building composite estimators whose parameter space can be searched with these tools.
3）模型选择：先训练、再评估
用训练集选择模型，用测试集验证模型（using the cross_validation.train_test_split utility function.）（it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.）
4）并行搜索
n_jobs=-1. 自动使用所有核。
5）robustness to failure（增强搜索错误的鲁棒性）
有些参数组合对于某些folds of the data会failure，进而导致整个search failure，尽管其他的参数组合没有问题。设定 error_score=0 (or =np.NaN) 可以使search过程忽略这样的failure，仅仅抛出一个warning，并将这样的search结果设为0 (or =np.NaN) ，能够提高搜索遇到错误时的鲁棒性！
4、Alternatives to brute force parameter search（没太看懂，还是不翻译了）
3.2.4.1. Model specific cross-validation
Some models can fit data for a range of value of some parameter almost as efficiently as fitting the estimator for a single value of the parameter. This feature can be leveraged to perform a more efficient cross-validation used for model selection of this parameter.
The most common parameter amenable to this strategy is the parameter encoding the strength of the regularizer. In this case we say that we compute theregularization path of the estimator.
Here is the list of such models:
| linear_model.ElasticNetCV([l1_ratio, eps, ...]) | Elastic Net model with iterative fitting along a regularization path |
| linear_model.LarsCV([fit_intercept, ...]) | Cross-validated Least Angle Regression model |
| linear_model.LassoCV([eps, n_alphas, ...]) | Lasso linear model with iterative fitting along a regularization path |
| linear_model.LassoLarsCV([fit_intercept, ...]) | Cross-validated Lasso, using the LARS algorithm |
| linear_model.LogisticRegressionCV([Cs, ...]) | Logistic Regression CV (aka logit, MaxEnt) classifier. |
| linear_model.MultiTaskElasticNetCV([...]) | Multi-task L1/L2 ElasticNet with built-in cross-validation. |
| linear_model.MultiTaskLassoCV([eps, ...]) | Multi-task L1/L2 Lasso with built-in cross-validation. |
| linear_model.OrthogonalMatchingPursuitCV([...]) | Cross-validated Orthogonal Matching Pursuit model (OMP) |
| linear_model.RidgeCV([alphas, ...]) | Ridge regression with built-in cross-validation. |
| linear_model.RidgeClassifierCV([alphas, ...]) | Ridge classifier with built-in cross-validation. |
3.2.4.2. Information Criterion
Some models can offer an information-theoretic closed-form formula of the optimal estimate of the regularization parameter by computing a single regularization path (instead of several when using cross-validation).
Here is the list of models benefitting from the Aikike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) for automated model selection:
| linear_model.LassoLarsIC([criterion, ...]) | Lasso model fit with Lars using BIC or AIC for model selection |
3.2.4.3. Out of Bag Estimates
When using ensemble methods base upon bagging, i.e. generating new training sets using sampling with replacement, part of the training set remains unused. For each classifier in the ensemble, a different part of the training set is left out.
This left out portion can be used to estimate the generalization error without having to rely on a separate validation set. This estimate comes “for free” as no additional data is needed and can be used for model selection.
This is currently implemented in the following classes:
| ensemble.RandomForestClassifier([...]) | A random forest classifier. |
| ensemble.RandomForestRegressor([...]) | A random forest regressor. |
| ensemble.ExtraTreesClassifier([...]) | An extra-trees classifier. |
| ensemble.ExtraTreesRegressor([n_estimators, ...]) | An extra-trees regressor. |
| ensemble.GradientBoostingClassifier([loss, ...]) | Gradient Boosting for classification. |
| ensemble.GradientBoostingRegressor([loss, ...]) | Gradient Boosting for regression. |