贝叶斯优化xgboost_超参数调整xgboost和神经网络的hyperopt贝叶斯优化

贝叶斯优化xgboost

Hyperparameters: These are certain values/weights that determine the learning process of an algorithm.

超参数：这些是确定算法学习过程的某些值/权重。

Certain parameters for an Machine Learning model: learning-rate, alpha, max-depth, col-samples , weights, gamma and so on.

机器学习模型的某些参数：学习率，alpha，最大深度，col-samples，权重，gamma等。

Certain parameters for an Deep Learning model: units(no of units), layer(no of layers), dropout ratio, kernel regularizers, activation function and so on.

深度学习模型的某些参数 ：单位(无单位)，层(无层)，辍学率，内核正则化函数，激活函数等。

Hyperparameter optimization is the selection of optimum or best parameter for a machine learning / deep learning algorithm. Often, we end up tuning or training the model manually with various possible range of parameters until a best fit model is obtained. Hyperparameter tuning helps in determining the optimal tuned parameters and return the best fit model, which is the best practice to follow while building an ML/DL model.

超参数优化是针对机器学习/深度学习算法的最佳或最佳参数的选择。 通常，我们最终会使用各种可能的参数范围手动调整或训练模型，直到获得最佳拟合模型为止。 超参数调整有助于确定最佳的调整参数并返回最佳拟合模型，这是构建ML / DL模型时遵循的最佳实践。

In this section let's discuss on one of the most accurate and successful hyperparameter method, which is HYPEROPT and algorithm to apply

在本节中，我们讨论一种最准确，最成功的超参数方法，即HYPEROPT和要应用的算法

Optimization is nothing but finding a minimum of cost function , that determines an overall better performance of a model on both train-set and test-set.

优化只不过是找到最小的成本函数，它决定了模型在训练集和测试集上的总体更好的性能。

HYPEROPT: It is a powerful python library that search through an hyperparameter space of values . It implements three functions for minimizing the cost function,

HYPEROPT：这是一个功能强大的python库，可在值的超参数空间中进行搜索。 它实现了三个功能以最小化成本功能，

1. Random Search
   随机搜寻
2. TPE (Tree Parzen Estimators)
   TPE(树Parzen估计器)
3. Adaptive TPE
   自适应TPE

Importing required packages:

导入所需的软件包：

import hyperopt
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

Hyperopt functions for optimization:

Hyperopt优化功能：

• hp.choice(label, options) — Returns one of the options, which should be a list or tuple.

  hp.choice(label, options) —返回选项之一，应该是列表或元组。

• hp.randint(label, upper) — Returns a random integer between the range [0, upper).

  hp.randint(label, upper) —返回介于[0，upper)范围之间的随机整数。

• hp.uniform(label, low, high) — Returns a value uniformly between low and high.

  hp.uniform(label, low, high) -统一在low和high之间返回一个值。

• hp.quniform(label, low, high, q) — Returns a value round(uniform(low, high) / q) * q, i.e it rounds the decimal values and returns an integer

  hp.quniform(label, low, high, q) —返回值round(uniform(low，high)/ q)* q，即四舍五入十进制值并返回整数

• hp.normal(label, mean, std) — Returns a real value that’s normally-distributed with mean and standard deviation sigma.

  hp.normal(label, mean, std) -返回一个均值和标准偏差sigma正态分布的实数值。

hyperopt涉及机器学习算法XGBOOST的步骤： (Steps involved in hyperopt for a Machine learning algorithm-XGBOOST:)

Step 1: Initialize space or a required range of values:

步骤1：初始化空间或所需的值范围：

space={'max_depth': hp.quniform("max_depth", 3, 18, 1),
        'gamma': hp.uniform ('gamma', 1,9),
        'reg_alpha' : hp.quniform('reg_alpha', 40,180,1),
        'reg_lambda' : hp.uniform('reg_lambda', 0,1),
        'colsample_bytree' : hp.uniform('colsample_bytree', 0.5,1),
        'min_child_weight' : hp.quniform('min_child_weight', 0, 10, 1),
        'n_estimators': 180
    }

Step 2: Define objective function:

步骤2：定义目标函数：

# Regression: 
def hyperparameter_tuning(space):
    model=xgb.XGBRegressor(n_estimators =space['n_estimators'], max_depth = int(space['max_depth']), gamma = space['gamma'],
                         reg_alpha = int(space['reg_alpha'],min_child_weight=space['min_child_weight'],
                         colsample_bytree=space['colsample_bytree']))
    
    evaluation = [( x_train, y_train), ( x_test, y_test)]
    
    model.fit(x_train, y_train,
            eval_set=evaluation, eval_metric="rmse",
            early_stopping_rounds=10,verbose=False)


    pred = model.predict(x_test)
    mse= mean_squared_error(y_test, pred)
    print ("SCORE:", mse)
    #change the metric if you like
    return {'loss':mse, 'status': STATUS_OK, 'model': model}
    
# Classifier:
def hyperparameter_tuning(space):
    model = xgb.XGBClassifier(n_estimators =space['n_estimators'], max_depth = int(space['max_depth']), gamma = space['gamma'],
                         reg_alpha = int(space['reg_alpha'],min_child_weight=space['min_child_weight'],
                         colsample_bytree=space['colsample_bytree']))
    evaluation = [( x_train, y_train), ( x_test, y_test)]
    
    model.fit(x_train, y_train,
            eval_set=evaluation, eval_metric="rmse",
            early_stopping_rounds=10,verbose=False)


    pred = model.predict(x_test)
    accuracy = accuracy_score(y_test, pred>0.5)
    print ("SCORE:", accuracy)
    #change the metric if you like
    return {'loss': -accuracy, 'status': STATUS_OK, 'model': model}

Step 3: Run Hyperopt function:

步骤3：运行Hyperopt功能：

trials = Trials()
best = fmin(fn=hyperparameter_tuning,
            space=space,
            algo=tpe.suggest,
            max_evals=100,
            trials=trials)


print (best)

Here, ‘best’ gives you the optimal parameters of the best fitted model and its loss function value.

在这里，“最佳”为您提供最佳拟合模型的最佳参数及其损失函数值。

Trials: It is an object that contains or stores all the statistical and diagnostic information such as hyperparameter, loss-functions for each set of parameters that the model has been trained.

试用：I t是一个对象，其中包含或存储所有统计和诊断信息，例如针对模型已训练的每组参数的超参数，损失函数。

fmin: It is an optimization function that minimizes the loss and takes in 4 inputs. Algorithm used is ‘tpe.suggest’ , other algorithm that can be used is ‘tpe.rand.suggest’.

fmin：这是一项优化功能，可最大程度地减少损耗并接受4个输入。 使用的算法是“ tpe.suggest ”，可以使用的其他算法是“ tpe.rand.suggest ”。

Step 4: Load/obtain the best model:

步骤4：加载/获取最佳模型：

loss = trials.best_trial['result']['loss']
print ('loss :', loss)
model = trials.best_trial['result']['model']
print (model)

Here, the object ‘model’ is the best tuned model for the given data. Use this model to evaluate against the test-set or use it for the prediction.

在这里，对象“模型”是给定数据的最佳调整模型。 使用此模型可以根据测试集进行评估或将其用于预测。

深度学习算法/神经网络的hyperopt涉及的步骤： (Steps involved in hyperopt for a Deep learning algorithm/neural networks:)

Step 1: Initialize space or a required range of values:

步骤1：初始化空间或所需的值范围：

space = { 'choice': hp.choice('num_layers',[{'layers':'two', }]),
            'units1': hp.uniform('units1', 50,1024),
            'units2': hp.uniform('units2', 50,1024),
            'dropout1': hp.uniform('dropout1', .25,.75),
            'dropout2': hp.uniform('dropout2',  .25,.75),
            'epochs' :  1000,
            'optimizer': hp.choice('optimizer',['adam','rmsprop'])
        }

Step 2: Define objective function:

步骤2：定义目标函数：

def hyperparameters(space):   
    from keras.models import Sequential
    from keras.layers.core import Dense, Dropout, Activation
    from keras.optimizers import Adadelta, Adam, rmsprop


    print ('Params testing: ', space)
    model=keras.Sequential()
    model.add(keras.layers.LSTM(units=int(params['units1']), return_sequences=False, 
                                input_shape=(None,X_train.shape[2],),
                                kernel_regularizer=keras.regularizers.l2(0.01)))
    model.add(keras.layers.Dropout(params['dropout1']))
    model.add(keras.layers.Activation('relu'))
    model.add(keras.layers.Dense(units=int(params['units2']),y_train.shape[2]))
    model.add(keras.layers.Dropout(params['dropout2']))
    model.add(keras.layers.Activation('relu'))
    
    model.compile(loss='mse',metrics=['accuracy'],optimizer=params['optimizer'])
    model.summary()  


    model.fit(X_train,y_train,epochs=epochs,validation_data=(X_test,y_test), batch_size = 45,verbose=0)


    pred_auc =model.predict_proba(X_test, batch_size = 45, verbose = 0)
    acc = roc_auc_score(y_test, pred_auc)
    print('AUC:', acc)
    sys.stdout.flush() 
    return {'loss': acc, 'status': STATUS_OK, 'model': model}

Note: Use appropriate regularization function and batch size that is suitable for your concerned data.

注意：使用适合您所关注数据的适当的正则化功能和批处理大小。

Step 3: Run Hyperopt function:

步骤3：运行Hyperopt功能：

from hyperopt import rand
trials = Trials()
best = fmin(fn=hyperparameters,
            space=space,
            algo=tpe.suggest,
            max_evals=100,
            trials=trials)

Step 4: Load/obtain the best model:

步骤4：加载/获取最佳模型：

loss = trials.best_trial['result']['loss']
print ('loss :', loss)
model = trials.best_trial['result']['model']
print (model)

Here, the object ‘model’ is the best tuned model for the given data. Use this model to evaluate against the test-set or use it for the prediction.

在这里，对象“模型”是给定数据的最佳调整模型。 使用此模型可以根据测试集进行评估或将其用于预测。

Another popular parameter tuner is Optuna. Look into the below article link for the comparison of Optuna v/s Hyperopt.

另一个流行的参数调谐器是Optuna。 查看下面的文章链接，以比较Optuna v / s Hyperopt。

结论： (Conclusion:)

We have discussed on how to use sklearn python library ‘hyperopt’ which is widely preferred in the field of Data Science. Hyperparameter tuning is an important step in building a learning algorithm model and it needs to be well scrutinized. Thanks for the time on reading this article, do appreciate!

我们已经讨论了如何使用sklearn python库“ hyperopt”，该库在数据科学领域广受青睐。 超参数调整是构建学习算法模型的重要步骤，需要仔细检查。 感谢您抽出宝贵的时间阅读本文，不胜感激！

翻译自: https://medium.com/@tinurohith18/hyperparameter-tuning-hyperopt-bayesian-optimization-for-xgboost-and-neural-network-434917d53e58

贝叶斯优化xgboost