Stochastic variational variable selection for high-dimensional microbiome data

Abstract

BackgroundThe rapid and accurate identification of a minimal-size core set of representative microbial species plays an important role in the clustering of microbial community data and interpretation of clustering results. However, the huge dimensionality of microbial metagenomics datasets is a major challenge for the existing methods such as Dirichlet multinomial mixture (DMM) models. In the framework of the existing methods, the computational burden of identifying a small number of representative species from a large number of observed species remains a challenge. ResultsWe proposed a novel framework to improve the performance of the widely used DMM approach by combining three ideas: (i) we extended the finite DMM model to an infinite case by considering Dirichlet process mixtures and estimating the number of clusters as a random variables; (ii) we proposed an indicator variable to identify representative operational taxonomic units that substantially contribute to the differentiation among clusters; and (iii) to address the computational burden of high-dimensional microbiome data, we proposed a stochastic variational inference, which approximates the posterior distribution using a controllable distribution called variational distribution, and stochastic optimization algorithms for fast computation. Using the proposed method, stochastic variational variable selection (SVVS), we analyzed the root microbiome data collected in our soybean field experiment, the human gut microbiome data from three published datasets of large-scale case-control studies and the healthy human microbiome data from the Human Microbiome Project. ConclusionsSVVS demonstrated a better performance and significantly faster computation than those of the existing methods in all cases of testing datasets. In particular, SVVS is the only method that can analyze massive high-dimensional microbial data with more than 50,000 microbial species and 1,000 samples. Furthermore, recent microbiome studies have suggested that selection of the microbial species used as a core set is important.