The eXplainable Artificial Intelligence (XAI) Triad: Models, Importances, and Significance at Scale
In this study, we present a comprehensive evaluation framework for comparing various combinations of artificial intelligence (AI) methods in the context of explainable AI (XAI) for variable selection in experimental biological and biomedical data. Our goal was to assess the efficiency, computational cost, and accuracy of different method combinations across six simulated scenarios, each replicated ten times. These scenarios encompass various classification and regression complexities, including variance differences, bimodal distributions, eXclusive-OR (XOR) interactions, concentric circles, and nonlinear relationships such as parabolic and sinusoidal functions. We tested several machine learning algorithms, including Decision Trees (DT), Random Forests (RF), Support Vector Machines (SVM), and Multi-Layer Perceptrons (MLP). We combined these models with diverse feature-importance methods such as Gini importance, accuracy decrease, SHAP values (Shapley Additive exPlanations), and Oldens method. We further applied significance-thresholding approaches, namely PIMP (Permutation IMPortance), mProbes, and the novel simThresh developed for this study. Additionally, we explored different dataset sizes to evaluate the scalability of these methods. Our analysis revealed substantial differences in computational demands, ranging from very rapid evaluations (e.g., DT combined with Gini importance and simThresh averaging 0.15 seconds) to extensive computations (e.g., MLP combined with SHAP and PIMP exceeding 7 hours). Among the tested combinations, RF/Accuracy/PIMP achieved the best overall performance, successfully identifying 59 out of 60 replicates in our benchmark study. However, this approach raises concerns regarding its scalability when applied to large-scale omics datasets in real-world settings due to its computational demands. In contrast, Decision Tree or Random Forest models using Gini and simThresh criteria ranked second, with 50 out of 60 detections. While less accurate, these methods require fewer computational resources, making them more promising candidates for scalable applications in omics data analysis. The proposed evaluation framework thus serves as a valuable tool for method selection, particularly relevant when dealing with large-scale omics datasets where computational resources and accuracy are both critical considerations.