Consistent Multiclass Algorithms for Complex Performance Measures

Harikrishna Narasimhan, Harish Ramaswamy, Aadirupa Saha, Shivani Agarwal
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:2398-2407, 2015.

Abstract

This paper presents new consistent algorithms for multiclass learning with complex performance measures, defined by arbitrary functions of the confusion matrix. This setting includes as a special case all loss-based performance measures, which are simply linear functions of the confusion matrix, but also includes more complex performance measures such as the multiclass G-mean and micro F_1 measures. We give a general framework for designing consistent algorithms for such performance measures by viewing the learning problem as an optimization problem over the set of feasible confusion matrices, and give two specific instantiations based on the Frank-Wolfe method for concave performance measures and on the bisection method for ratio-of-linear performance measures. The resulting algorithms are provably consistent and outperform a multiclass version of the state-of-the-art SVMperf method in experiments; for large multiclass problems, the algorithms are also orders of magnitude faster than SVMperf.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-narasimhanb15, title = {Consistent Multiclass Algorithms for Complex Performance Measures}, author = {Narasimhan, Harikrishna and Ramaswamy, Harish and Saha, Aadirupa and Agarwal, Shivani}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {2398--2407}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/narasimhanb15.pdf}, url = {https://proceedings.mlr.press/v37/narasimhanb15.html}, abstract = {This paper presents new consistent algorithms for multiclass learning with complex performance measures, defined by arbitrary functions of the confusion matrix. This setting includes as a special case all loss-based performance measures, which are simply linear functions of the confusion matrix, but also includes more complex performance measures such as the multiclass G-mean and micro F_1 measures. We give a general framework for designing consistent algorithms for such performance measures by viewing the learning problem as an optimization problem over the set of feasible confusion matrices, and give two specific instantiations based on the Frank-Wolfe method for concave performance measures and on the bisection method for ratio-of-linear performance measures. The resulting algorithms are provably consistent and outperform a multiclass version of the state-of-the-art SVMperf method in experiments; for large multiclass problems, the algorithms are also orders of magnitude faster than SVMperf.} }
Endnote
%0 Conference Paper %T Consistent Multiclass Algorithms for Complex Performance Measures %A Harikrishna Narasimhan %A Harish Ramaswamy %A Aadirupa Saha %A Shivani Agarwal %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-narasimhanb15 %I PMLR %P 2398--2407 %U https://proceedings.mlr.press/v37/narasimhanb15.html %V 37 %X This paper presents new consistent algorithms for multiclass learning with complex performance measures, defined by arbitrary functions of the confusion matrix. This setting includes as a special case all loss-based performance measures, which are simply linear functions of the confusion matrix, but also includes more complex performance measures such as the multiclass G-mean and micro F_1 measures. We give a general framework for designing consistent algorithms for such performance measures by viewing the learning problem as an optimization problem over the set of feasible confusion matrices, and give two specific instantiations based on the Frank-Wolfe method for concave performance measures and on the bisection method for ratio-of-linear performance measures. The resulting algorithms are provably consistent and outperform a multiclass version of the state-of-the-art SVMperf method in experiments; for large multiclass problems, the algorithms are also orders of magnitude faster than SVMperf.
RIS
TY - CPAPER TI - Consistent Multiclass Algorithms for Complex Performance Measures AU - Harikrishna Narasimhan AU - Harish Ramaswamy AU - Aadirupa Saha AU - Shivani Agarwal BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-narasimhanb15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 2398 EP - 2407 L1 - http://proceedings.mlr.press/v37/narasimhanb15.pdf UR - https://proceedings.mlr.press/v37/narasimhanb15.html AB - This paper presents new consistent algorithms for multiclass learning with complex performance measures, defined by arbitrary functions of the confusion matrix. This setting includes as a special case all loss-based performance measures, which are simply linear functions of the confusion matrix, but also includes more complex performance measures such as the multiclass G-mean and micro F_1 measures. We give a general framework for designing consistent algorithms for such performance measures by viewing the learning problem as an optimization problem over the set of feasible confusion matrices, and give two specific instantiations based on the Frank-Wolfe method for concave performance measures and on the bisection method for ratio-of-linear performance measures. The resulting algorithms are provably consistent and outperform a multiclass version of the state-of-the-art SVMperf method in experiments; for large multiclass problems, the algorithms are also orders of magnitude faster than SVMperf. ER -
APA
Narasimhan, H., Ramaswamy, H., Saha, A. & Agarwal, S.. (2015). Consistent Multiclass Algorithms for Complex Performance Measures. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:2398-2407 Available from https://proceedings.mlr.press/v37/narasimhanb15.html.

Related Material