Estimating Accuracy from Unlabeled Data: A Bayesian Approach

Emmanouil Antonios Platanios, Avinava Dubey, Tom Mitchell
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:1416-1425, 2016.

Abstract

We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers, and the related question of how outputs from several classifiers performing the same task can be combined based on their estimated accuracies. To answer these questions, we first present a simple graphical model that performs well in practice. We then provide two nonparametric extensions to it that improve its performance. Experiments on two real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-platanios16, title = {Estimating Accuracy from Unlabeled Data: A Bayesian Approach}, author = {Platanios, Emmanouil Antonios and Dubey, Avinava and Mitchell, Tom}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {1416--1425}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/platanios16.pdf}, url = {https://proceedings.mlr.press/v48/platanios16.html}, abstract = {We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers, and the related question of how outputs from several classifiers performing the same task can be combined based on their estimated accuracies. To answer these questions, we first present a simple graphical model that performs well in practice. We then provide two nonparametric extensions to it that improve its performance. Experiments on two real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs.} }
Endnote
%0 Conference Paper %T Estimating Accuracy from Unlabeled Data: A Bayesian Approach %A Emmanouil Antonios Platanios %A Avinava Dubey %A Tom Mitchell %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-platanios16 %I PMLR %P 1416--1425 %U https://proceedings.mlr.press/v48/platanios16.html %V 48 %X We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers, and the related question of how outputs from several classifiers performing the same task can be combined based on their estimated accuracies. To answer these questions, we first present a simple graphical model that performs well in practice. We then provide two nonparametric extensions to it that improve its performance. Experiments on two real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs.
RIS
TY - CPAPER TI - Estimating Accuracy from Unlabeled Data: A Bayesian Approach AU - Emmanouil Antonios Platanios AU - Avinava Dubey AU - Tom Mitchell BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-platanios16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 1416 EP - 1425 L1 - http://proceedings.mlr.press/v48/platanios16.pdf UR - https://proceedings.mlr.press/v48/platanios16.html AB - We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers, and the related question of how outputs from several classifiers performing the same task can be combined based on their estimated accuracies. To answer these questions, we first present a simple graphical model that performs well in practice. We then provide two nonparametric extensions to it that improve its performance. Experiments on two real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs. ER -
APA
Platanios, E.A., Dubey, A. & Mitchell, T.. (2016). Estimating Accuracy from Unlabeled Data: A Bayesian Approach. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:1416-1425 Available from https://proceedings.mlr.press/v48/platanios16.html.

Related Material