Ensemble learning is a general meta approach to machine learning that seeks better predictive performance by combining the predictions from multiple models. Although there are a seemingly unlimited number of ensembles that you can develop for your predictive modeling problem, there are three methods that dominate the field of ensemble learning. So much so, that rather than algorithms per se, each is a field of study that has spawned many more specialized methods. The three main classes of ensemble learning […]

Ensemble methods involve combining the predictions from multiple models. The combination of the predictions is a central part of the ensemble method and depends heavily on the types of models that contribute to the ensemble and the type of prediction problem that is being modeled, such as a classification or regression. Nevertheless, there are common or standard techniques that can be used to combine predictions that can be easily implemented and often result in good or best predictive performance. In […]

Stacked generalization, or stacking, may be a less popular machine learning ensemble given that it describes a framework more than a specific model. Perhaps the reason it has been less popular in mainstream machine learning is that it can be tricky to train a stacking model correctly, without suffering data leakage. This has meant that the technique has mainly been used by highly skilled experts in high-stakes environments, such as machine learning competitions, and given new names like blending ensembles. […]

Dynamic ensemble selection is an ensemble learning technique that automatically selects a subset of ensemble members just-in-time when making a prediction. The technique involves fitting multiple machine learning models on the training dataset, then selecting the models that are expected to perform best when making a prediction for a specific new example, based on the details of the example to be predicted. This can be achieved using a k-nearest neighbor model to locate examples in the training dataset that are […]

Ensemble member selection refers to algorithms that optimize the composition of an ensemble. This may involve growing an ensemble from available models or pruning members from a fully defined ensemble. The goal is often to reduce the model or computational complexity of an ensemble with little or no effect on the performance of an ensemble, and in some cases find a combination of ensemble members that results in better performance than blindly using all contributing models directly. In this tutorial, […]

Mixture of experts is an ensemble learning technique developed in the field of neural networks. It involves decomposing predictive modeling tasks into sub-tasks, training an expert model on each, developing a gating model that learns which expert to trust based on the input to be predicted, and combines the predictions. Although the technique was initially described using neural network experts and gating models, it can be generalized to use models of any type. As such, it shows a strong similarity […]

It is common to describe ensemble learning techniques in terms of weak and strong learners. For example, we may desire to construct a strong learner from the predictions of many weak learners. In fact, this is the explicit goal of the boosting class of ensemble learning algorithms. Although we may describe models as weak or strong generally, the terms have a specific formal definition and are used as the basis for an important finding from the field of computational learning […]