Theoretical solutions for the evaluation of discrimination capacity of species distribution models

Abstract

The discriminating capacity (i.e., ability to correctly classify presences and absences) of species distribution models (SDMs) is commonly evaluated with metrics such as the Area Under the Receiving Operating Characteristic Curve, the Kappa statistic and the True Skill Statistic (TSS). AUC and Kappa have been repeatedly criticised, but the TSS has fared relatively well since its introduction, mainly because it has been considered as independent of prevalence. In addition, discrimination metrics have been contested because they should be calculated on presence-absence data, but are often used on presence-only or presence-background data. Here, we investigate the TSS and an alternative set of metrics -similarity indices, also known as F-measures. We first show that even in ideal conditions (i.e., perfectly random presence-absence sampling), TSS can be misleading because of its dependence on prevalence, whereas similarity/F-measures provide adequate estimations of model discrimination capacity. Second, we show that in real-world situations where sample prevalence is different from true species prevalence (i.e., biased sampling or presence-pseudoabsence), no discrimination capacity metric provide adequate estimations of model discrimination capacity, including metrics specifically designed for presence-pseudoabsence. Our conclusions are twofold. First, they unequivocally appeal SDM users to understand the potential shortcomings of discrimination metrics when quality presence-absence data are lacking, and we provide recommendations to obtain such data. Second, in the specific case of virtual species, which are increasingly used to develop and test SDM methodologies, we strongly recommend the use of similarity/F-measures, which were not biased by prevalence, contrary to TSS.