Generalization Error Bounds for Bayesian Mixture Algorithms
Ron Meir, Tong Zhang; 4(Oct):839-860, 2003.
Abstract
Bayesian approaches to learning and estimation have played a
significant role in the Statistics literature over many years.
While they are often provably optimal in a frequentist setting,
and lead to excellent performance in practical applications, there
have not been many precise characterizations of their performance
for finite sample sizes under general conditions. In this paper we
consider the class of Bayesian mixture algorithms, where an
estimator is formed by constructing a data-dependent mixture over
some hypothesis space. Similarly to what is observed in practice,
our results demonstrate that mixture approaches are particularly
robust, and allow for the construction of highly complex
estimators, while avoiding undesirable overfitting effects. Our
results, while being data-dependent in nature, are insensitive to
the underlying model assumptions, and apply whether or not these
hold. At a technical level, the approach applies to unbounded
functions, constrained only by certain moment conditions. Finally,
the bounds derived can be directly applied to non-Bayesian mixture
approaches such as Boosting and Bagging.
[abs][pdf][ps.gz][ps]
