机器学习之-weight decay

Weight decay is a regularization technique that is used to regularize the size of the weights of certain parameters in machine learning models. Weight decay is most widely used regularization technique for parametric machine learning models. Weight decay is also known as L2 regularization, because it penalizes weights according to their L2 norm. In weight decay technique, the objective function of minimizing the prediction loss on the training data is replaced with the new objective function, minimizing the sum of the prediction loss and the penalty term. It involves adding a term to the objective function that is proportional to the sum of the squares of the weights. This is how the new loss function looks like using weight decay technique: