We analyze the problem of selecting the model that best describes a given dataset.
We focus on the case where the best model is the one with the smallest error, respect
to the reference data. To select the best model, we consider two components: (a) an
error measure to compare individual data points, and (b) a function that combines
the individual errors for all the points. We show that working with the most general
definition of consistency, it is impossible to extend individual error measures in
a way that provides a unanimous consensus about which is the best model. We also prove
that, in the best case, modifying the notion of consistency leads to expressions that
are too ill-behaved to be of any practical utility. These results show that selecting
the model that best describes a dataset depends heavily on the way one measures the
individual errors, even if these measures are consistent.
