Local Optimization Versus Global Optimization

Optimization refers to finding the set of inputs to an objective function that results in the maximum or minimum output from the objective function. It is common to describe optimization problems in terms of local vs. global optimization. Similarly, it is also common to describe optimization algorithms or search algorithms in terms of local vs. global search. In this tutorial, you will discover the practical differences between local and global optimization. After completing this tutorial, you will know: Local optimization […]

Read more

How to Use Nelder-Mead Optimization in Python

The Nelder-Mead optimization algorithm is a widely used approach for non-differentiable objective functions. As such, it is generally referred to as a pattern search algorithm and is used as a local or global search procedure, challenging nonlinear and potentially noisy and multimodal function optimization problems. In this tutorial, you will discover the Nelder-Mead optimization algorithm. After completing this tutorial, you will know: The Nelder-Mead optimization algorithm is a type of pattern search that does not use function gradients. How to […]

Read more

Visualization for Function Optimization in Python

Function optimization involves finding the input that results in the optimal value from an objective function. Optimization algorithms navigate the search space of input variables in order to locate the optima, and both the shape of the objective function and behavior of the algorithm in the search space are opaque on real-world problems. As such, it is common to study optimization algorithms using simple low-dimensional functions that can be easily visualized directly. Additionally, the samples in the input space of […]

Read more

3 Books on Optimization for Machine Learning

Optimization is a field of mathematics concerned with finding a good or best solution among many candidates. It is an important foundational topic required in machine learning as most machine learning algorithms are fit on historical data using an optimization algorithm. Additionally, broader problems, such as model selection and hyperparameter tuning, can also be framed as an optimization problem. Although having some background in optimization is critical for machine learning practitioners, it can be a daunting topic given that it […]

Read more

Code Adam Gradient Descent Optimization From Scratch

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function. A limitation of gradient descent is that a single step size (learning rate) is used for all input variables. Extensions to gradient descent like AdaGrad and RMSProp update the algorithm to use a separate step size for each input variable but may result in a step size that rapidly decreases to very small values. The Adaptive […]

Read more

Univariate Function Optimization in Python

How to Optimize a Function with One Variable? Univariate function optimization involves finding the input to a function that results in the optimal output from an objective function. This is a common procedure in machine learning when fitting a model with one parameter or tuning a model that has a single hyperparameter. An efficient algorithm is required to solve optimization problems of this type that will find the best solution with the minimum number of evaluations of the objective function, […]

Read more

Feature Selection with Stochastic Optimization Algorithms

Typically, a simpler and better-performing machine learning model can be developed by removing input features (columns) from the training dataset. This is called feature selection and there are many different types of algorithms that can be used. It is possible to frame the problem of feature selection as an optimization problem. In the case that there are few input features, all possible combinations of input features can be evaluated and the best subset found definitively. In the case of a […]

Read more

How to Choose an Optimization Algorithm

Optimization is the problem of finding a set of inputs to an objective function that results in a maximum or minimum function evaluation. It is the challenging problem that underlies many machine learning algorithms, from fitting logistic regression models to training artificial neural networks. There are perhaps hundreds of popular optimization algorithms, and perhaps tens of algorithms to choose from in popular scientific code libraries. This can make it challenging to know which algorithms to consider for a given optimization […]

Read more

Curve Fitting With Python

Curve fitting is a type of optimization that finds an optimal set of parameters for a defined function that best fits a given set of observations. Unlike supervised learning, curve fitting requires that you define the function that maps examples of inputs to outputs. The mapping function, also called the basis function can have any form you like, including a straight line (linear regression), a curved line (polynomial regression), and much more. This provides the flexibility and control to define […]

Read more

Stochastic Hill Climbing in Python from Scratch

Stochastic Hill climbing is an optimization algorithm. It makes use of randomness as part of the search process. This makes the algorithm appropriate for nonlinear objective functions where other local search algorithms do not operate well. It is also a local search algorithm, meaning that it modifies a single solution and searches the relatively local area of the search space until the local optima is located. This means that it is appropriate on unimodal optimization problems or for use after […]

Read more
1 2 3