Home > Ensembles

Bagging (Breiman, 1996), a name derived from “bootstrap aggregation”, was the first effective method of ensemble learning and is one of the simplest methods of arching [1]. The meta-algorithm, which is a special case of the model averaging, was originally designed for classification and is usually applied to decision tree models, but it can be used with any type of model for classification or regression. The method uses multiple versions of a training set by using the bootstrap, i.e. sampling with replacement. Each of these data sets is used to train a different model. The outputs of the models are combined by averaging (in case of regression) or voting (in case of classification) to create a single output. Bagging is only effective when using unstable (i.e. a small change in the training set can cause a significant change in the model) nonlinear models.

[1] Arching (adaptive reweighting and combining) is a generic term that refers to reusing or selecting data in order to improve classification.

Seminal Paper

  • BREIMAN, Leo, 1996. Bagging predictors, Machine Learning 24(2):123140. [Cited by 2460] (239.89/year)
  • Links