Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence

Yung-Kyun Noh, Masashi Sugiyama, Song Liu, Marthinus C. Plessis, Frank Chongwoo Park, Daniel D. Lee
Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, PMLR 33:669-677, 2014.

Abstract

Asymptotically unbiased nearest-neighbor estimators for K-L divergence have recently been proposed and demonstrated in a number of applications. With small sample sizes, however, these nonparametric methods typically suffer from high estimation bias due to the non-local statistics of empirical nearest-neighbor information. In this paper, we show that this non-local bias can be mitigated by changing the distance metric, and we propose a method for learning an optimal Mahalanobis-type metric based on global information provided by approximate parametric models of the underlying densities. In both simulations and experiments, we demonstrate that this interplay between parametric models and nonparametric estimation methods significantly improves the accuracy of the nearest-neighbor K-L divergence estimator.

Cite this Paper

BibTeX
@InProceedings{pmlr-v33-noh14, title = {{Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence}}, author = {Noh, Yung-Kyun and Sugiyama, Masashi and Liu, Song and Plessis, Marthinus C. and Park, Frank Chongwoo and Lee, Daniel D.}, booktitle = {Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics}, pages = {669--677}, year = {2014}, editor = {Kaski, Samuel and Corander, Jukka}, volume = {33}, series = {Proceedings of Machine Learning Research}, address = {Reykjavik, Iceland}, month = {22--25 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v33/noh14.pdf}, url = {https://proceedings.mlr.press/v33/noh14.html}, abstract = {Asymptotically unbiased nearest-neighbor estimators for K-L divergence have recently been proposed and demonstrated in a number of applications. With small sample sizes, however, these nonparametric methods typically suffer from high estimation bias due to the non-local statistics of empirical nearest-neighbor information. In this paper, we show that this non-local bias can be mitigated by changing the distance metric, and we propose a method for learning an optimal Mahalanobis-type metric based on global information provided by approximate parametric models of the underlying densities. In both simulations and experiments, we demonstrate that this interplay between parametric models and nonparametric estimation methods significantly improves the accuracy of the nearest-neighbor K-L divergence estimator.} }
Endnote
%0 Conference Paper %T Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence %A Yung-Kyun Noh %A Masashi Sugiyama %A Song Liu %A Marthinus C. Plessis %A Frank Chongwoo Park %A Daniel D. Lee %B Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2014 %E Samuel Kaski %E Jukka Corander %F pmlr-v33-noh14 %I PMLR %P 669--677 %U https://proceedings.mlr.press/v33/noh14.html %V 33 %X Asymptotically unbiased nearest-neighbor estimators for K-L divergence have recently been proposed and demonstrated in a number of applications. With small sample sizes, however, these nonparametric methods typically suffer from high estimation bias due to the non-local statistics of empirical nearest-neighbor information. In this paper, we show that this non-local bias can be mitigated by changing the distance metric, and we propose a method for learning an optimal Mahalanobis-type metric based on global information provided by approximate parametric models of the underlying densities. In both simulations and experiments, we demonstrate that this interplay between parametric models and nonparametric estimation methods significantly improves the accuracy of the nearest-neighbor K-L divergence estimator.
RIS
TY - CPAPER TI - Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence AU - Yung-Kyun Noh AU - Masashi Sugiyama AU - Song Liu AU - Marthinus C. Plessis AU - Frank Chongwoo Park AU - Daniel D. Lee BT - Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics DA - 2014/04/02 ED - Samuel Kaski ED - Jukka Corander ID - pmlr-v33-noh14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 33 SP - 669 EP - 677 L1 - http://proceedings.mlr.press/v33/noh14.pdf UR - https://proceedings.mlr.press/v33/noh14.html AB - Asymptotically unbiased nearest-neighbor estimators for K-L divergence have recently been proposed and demonstrated in a number of applications. With small sample sizes, however, these nonparametric methods typically suffer from high estimation bias due to the non-local statistics of empirical nearest-neighbor information. In this paper, we show that this non-local bias can be mitigated by changing the distance metric, and we propose a method for learning an optimal Mahalanobis-type metric based on global information provided by approximate parametric models of the underlying densities. In both simulations and experiments, we demonstrate that this interplay between parametric models and nonparametric estimation methods significantly improves the accuracy of the nearest-neighbor K-L divergence estimator. ER -
APA
Noh, Y., Sugiyama, M., Liu, S., Plessis, M.C., Park, F.C. & Lee, D.D.. (2014). Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 33:669-677 Available from https://proceedings.mlr.press/v33/noh14.html.

Related Material