bioRxiv · 10.64898/2025.12.10.693603
Fast Loading and Transformation of Large-Volume Bioimaging Data Consistently Approaching the I/O Peak Speed
Abstract
High-resolution large-volume biological imaging techniques are now widely used in biological research, but inefficiencies in data processing and visualization persist due to bottlenecks in loading/saving pipelines and limitations of conventional pyramid formats. To address these issues, we have decoupled data loading process into reading from disk and in-memory decompression, while data saving has been separated into in-memory compression and writing to disk. This approach allows us to dedicate more CPU cores to decompression and compression tasks, optimizing resource allocation and maximizing disk I/O throughput. As a result, both loading and saving processes operate near the disks peak data transfer capacity. Further, we integrated a video stream-inspired format into the pyramid structure, allowing direct access to regions of interest (ROIs) without loading extraneous data into RAM, reducing memory overhead. Compared to the latest methods, our approach improves pyramid data transformation efficiency by at least 6x and dramatically accelerates downstream tasks like visualization and deep learning inference. TeaserNovel bioimaging data processing pipeline achieves near-peak I/O throughput via decoupled architecture and video-formatting.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Cai, L., Qu, X., Zhou, H., Li, N., Gou, X., Li, S., Huang, J., Guo, S., Liu, X., Lv, X., Quan, T., Zeng, S.. 2025-12-12. Fast Loading and Transformation of Large-Volume Bioimaging Data Consistently Approaching the I/O Peak Speed. https://doi.org/10.64898/2025.12.10.693603
Cite the original work for its findings. Save a collection to share your selection of sources.