Agnostic Bayesian Learning of Ensembles

Alexandre Lacoste, Mario Marchand, François Laviolette, Hugo Larochelle
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(1):611-619, 2014.

Abstract

We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, we can thus obtain a posterior that represents our uncertainty about that choice and construct a weighted ensemble of predictors accordingly. This approach has the advantage of not requiring that the predictors be probabilistic themselves, can deal with arbitrary measures of performance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAC-learning, we refer to our method as agnostic Bayesian learning. We also propose a method to address the case where the performance estimate is obtained from k-fold cross validation. While being efficient and easily adjustable to any loss function, our experiments confirm that the agnostic Bayes approach is state of the art compared to common baselines such as model selection based on k-fold cross-validation or a linear combination of predictor outputs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-lacoste14, title = {Agnostic Bayesian Learning of Ensembles}, author = {Lacoste, Alexandre and Marchand, Mario and Laviolette, François and Larochelle, Hugo}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {611--619}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {1}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/lacoste14.pdf}, url = {https://proceedings.mlr.press/v32/lacoste14.html}, abstract = {We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, we can thus obtain a posterior that represents our uncertainty about that choice and construct a weighted ensemble of predictors accordingly. This approach has the advantage of not requiring that the predictors be probabilistic themselves, can deal with arbitrary measures of performance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAC-learning, we refer to our method as agnostic Bayesian learning. We also propose a method to address the case where the performance estimate is obtained from k-fold cross validation. While being efficient and easily adjustable to any loss function, our experiments confirm that the agnostic Bayes approach is state of the art compared to common baselines such as model selection based on k-fold cross-validation or a linear combination of predictor outputs.} }
Endnote
%0 Conference Paper %T Agnostic Bayesian Learning of Ensembles %A Alexandre Lacoste %A Mario Marchand %A François Laviolette %A Hugo Larochelle %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-lacoste14 %I PMLR %P 611--619 %U https://proceedings.mlr.press/v32/lacoste14.html %V 32 %N 1 %X We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, we can thus obtain a posterior that represents our uncertainty about that choice and construct a weighted ensemble of predictors accordingly. This approach has the advantage of not requiring that the predictors be probabilistic themselves, can deal with arbitrary measures of performance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAC-learning, we refer to our method as agnostic Bayesian learning. We also propose a method to address the case where the performance estimate is obtained from k-fold cross validation. While being efficient and easily adjustable to any loss function, our experiments confirm that the agnostic Bayes approach is state of the art compared to common baselines such as model selection based on k-fold cross-validation or a linear combination of predictor outputs.
RIS
TY - CPAPER TI - Agnostic Bayesian Learning of Ensembles AU - Alexandre Lacoste AU - Mario Marchand AU - François Laviolette AU - Hugo Larochelle BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/01/27 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-lacoste14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 1 SP - 611 EP - 619 L1 - http://proceedings.mlr.press/v32/lacoste14.pdf UR - https://proceedings.mlr.press/v32/lacoste14.html AB - We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, we can thus obtain a posterior that represents our uncertainty about that choice and construct a weighted ensemble of predictors accordingly. This approach has the advantage of not requiring that the predictors be probabilistic themselves, can deal with arbitrary measures of performance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAC-learning, we refer to our method as agnostic Bayesian learning. We also propose a method to address the case where the performance estimate is obtained from k-fold cross validation. While being efficient and easily adjustable to any loss function, our experiments confirm that the agnostic Bayes approach is state of the art compared to common baselines such as model selection based on k-fold cross-validation or a linear combination of predictor outputs. ER -
APA
Lacoste, A., Marchand, M., Laviolette, F. & Larochelle, H.. (2014). Agnostic Bayesian Learning of Ensembles. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(1):611-619 Available from https://proceedings.mlr.press/v32/lacoste14.html.

Related Material