Random Search and Grid Search for Function Optimization

Function optimization requires the selection of an algorithm to efficiently sample the search space and locate a good or best solution. There are many algorithms to choose from, although it is important to establish a baseline for what types of solutions are feasible or possible for a problem. This can be achieved using a naive optimization algorithm, such as a random search or a grid search. The results achieved by a naive optimization algorithm are computationally efficient to generate and […]

Basin Hopping Optimization in Python

Basin hopping is a global optimization algorithm. It was developed to solve problems in chemical physics, although it is an effective algorithm suited for nonlinear objective functions with multiple optima. In this tutorial, you will discover the basin hopping global optimization algorithm. After completing this tutorial, you will know: Basin hopping optimization is a global optimization that uses random perturbations to jump basins, and a local search algorithm to optimize each basin. How to use the basin hopping optimization algorithm […]

Gradient Descent With Nesterov Momentum From Scratch

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function. A limitation of gradient descent is that it can get stuck in flat areas or bounce around if the objective function returns noisy gradients. Momentum is an approach that accelerates the progress of the search to skim across flat areas and smooth out bouncy gradients. In some cases, the acceleration of momentum can cause the search […]

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function. A limitation of gradient descent is that the progress of the search can slow down if the gradient becomes flat or large curvature. Momentum can be added to gradient descent that incorporates some inertia to updates. This can be further improved by incorporating the gradient of the projected new position rather than the current position, called […]

How to Manually Optimize Machine Learning Model Hyperparameters

Machine learning algorithms have hyperparameters that allow the algorithms to be tailored to specific datasets. Although the impact of hyperparameters may be understood generally, their specific effect on a dataset and their interactions during learning may not be known. Therefore, it is important to tune the values of algorithm hyperparameters as part of a machine learning project. It is common to use naive optimization algorithms to tune hyperparameters, such as a grid search and a random search. An alternate approach […]

Two-Dimensional (2D) Test Functions for Function Optimization

Function optimization is a field of study that seeks an input to a function that results in the maximum or minimum output of the function. There are a large number of optimization algorithms and it is important to study and develop intuitions for optimization algorithms on simple and easy-to-visualize test functions. Two-dimensional functions take two input values (x and y) and output a single evaluation of the input. They are among the simplest types of test functions to use when […]

Iterated Local Search From Scratch in Python

Iterated Local Search is a stochastic global optimization algorithm. It involves the repeated application of a local search algorithm to modified versions of a good solution found previously. In this way, it is like a clever version of the stochastic hill climbing with random restarts algorithm. The intuition behind the algorithm is that random restarts can help to locate many local optima in a problem and that better local optima are often close to other local optima. Therefore modest perturbations […]