Approximation of unbiased convex classification error rate estimator

Abstract

Convex classification error rate estimator is described as weighted combination of the low-biased estimator and the high-biased estimator. If the underlying data model is known, the coefficients (weights) can be optimized so that the bias and root-mean-square error of the estimator is minimized. However, in most situations, data model is unknown. In this paper we propose a new error estimation method, based on approximation of unbiased convex error rate estimator. Experiments with real world and synthetic data sets show that common error estimation methods, such as resubstitution, repeated 10-foldcross-validation, leave-one-out and random subsampling are outperformed (in terms of root-mean-square error) by the proposed method.