Do transformers and CNNs learn different concepts of brain age?

Abstract

"Predicted brain age" refers to a biomarker of structural brain health derived from machine learning analysis of T1-weighted brain magnetic resonance (MR) images. A range of machine learning methods have been used to predict brain age, with convolutional neural networks (CNNs) currently yielding state-of-the-art accuracies. Recent advances in deep learning have introduced transformers, which are conceptually distinct from CNNs, and appear to set new benchmarks in various domains of computer vision. However, transformers have not yet been applied to brain age prediction. Thus, we address two research questions: First, are transformers superior to CNNs in predicting brain age? Second, do conceptually different deep learning model architectures learn similar or different "concepts of brain age"? We adapted a Simple Vision Transformer (sViT) and a Shifted Window Transformer (SwinT) to predict brain age, and compared both models with a ResNet50 on 46,381 T1-weighted structural MR images from the UK Biobank. We found that SwinT and ResNet performed on par, while additional training samples will most likely give SwinT the edge in prediction accuracy. We identified that different model architectures may characterize different (sub-)sets of brain aging effects, representing diverging concepts of brain age. Thus, we systematically tested whether sViT, SwinT and ResNet focus on different concepts of brain age by examining variations in their predictions and clinical utility for indicating deviations in neurological and psychiatric disorders. Reassuringly, we did not find substantial differences in the structure of brain age predictions between model architectures. Based on our results, the choice of deep learning model architecture does not appear to have a confounding effect on brain age studies.