Biology subjects

Weissman, T.

Publications and source records attributed to Weissman, T..

11 recordsLinked to original sources

bioRxiv2026

Compression Detects Changes in Spiking Neural Data from Cortical Lesions

1ObjectiveThe complexity of neural data changes as the brain processes information during events. Universal lossless compression algorithms, which are broadly applicable and grounded in information theory, identify and exploit redundancies in data in order to compress it to essentially-optimal sizes regardless of underlying statistics. These algorithms may be used to conveniently and efficiently estimate a given signals Shannon entropy rate, a biologically relevant measure of the complexity of a signal. It is therefore natural to explore their effectiveness in the analysis of spiking neural data. ApproachThis work focuses on using compression to analyze recordings (96-channel Utah arrays) taken from motor cortex of animals performing reaching tasks for three days before and three days after administering electrolytic lesions (Subject U: 4 lesions, H: 3). In particular, we use the inverse compression ratio (ICR), which compares the sizes of compressed and uncompressed data to estimate the amount of statistically unique information. We calculate ICR with temporally-independent lossless compression (gzip) and temporally-dependent lossy compression (H.264, MPEG-2). Compression-based ICR was compared to single-neuron measures used to understand spiking data, such as average firing rates and Fano factor. Compression is also compared to common dimensionality reduction techniques, principal component analysis (PCA) and factor analysis (FA). Main ResultsStatistical tests on aggregate data comparing each metric before and after lesioning reveal that ICR is able to significantly (Mann-Whitney U test, p < 0.01) detect lesions with higher accuracy than single-neuron metrics, but not dimensionality reduction (ICR methods: 85.7%, single-neuron methods: 78.6%, dimensionality reduction: 100%). Additionally, statistical results on the same data show that ICR metrics remain more stable than single-neuron methods after lesion. The bitrate parameter of lossy compression algorithms is swept to better understand the effect of information rates and "optimal" compression on lesion detection performance. Our conclusions are confirmed by the same analyses performed on several different simulated neural datasets. SignificanceThese results suggest that compression algorithms may be a useful tool to detect and better understand perturbations to the underlying structure of neural data. Information-theoretic analyses may complement techniques like dimensionality reduction and firing rate tuning as a convenient and useful tool to characterize neural data.

neuroscience↗

bioRxiv2025

A Framework for Compressive On-chip Action Potential Recording

Scaling neural recording systems to thousands of channels creates extreme bandwidth demands, posing a challenge for resource-constrained, implantable devices. This work introduces an adaptive, multi-stage compression framework for high-bandwidth neural interfaces. The system combines a Wired-OR analog-to-digital compressive readout with a digital core that adaptively requantizes, selectively samples, and encodes the neural signals. Although prior work suggests that action potential recordings can be re-quantized to approximately the signal-to-noise (SNR) number of bits without significantly degrading decoding performance, our results show that the required resolution can often be reduced even further. By matching the number of quantization levels to the electrodes maximum SNR ({lceil}log2 SNR{rciel} number of bits), we retain waveform fidelity while eliminating unnecessary precision that primarily captures noise. Recorded spike samples are selected using a mutual information-based criterion to preserve both spatial and temporal discriminative waveform features. A static entropy coder completes the pipeline with low computation overhead compression optimized for neural signal statistics. Evaluated on 512-channel macaque retina ex vivo data, the system preserves 90% of spikes while achieving a 1098x total compression over baseline.

bioengineering↗

bioRxiv2022

The Human Pangenome's sequence conservation reveals a landscape of polymorphic structural variation

The Human Pangenome is a new reference build that addresses many of the limitations of the current reference. The first release is based on 94 high-quality haploid assemblies from individuals with diverse backgrounds. To facilitate the Pangenomes adoption in the wider research community, we conducted a multiple genome comparative analysis against the current GRCh38 reference. We applied a k-mer indexing strategy to identify highly conserved sequences that are omnipresent across all Pangenome assemblies, the reference and CHM13, a reference assembly with telomere-to-telomere chromosomes. Pan-conserved tag segments provided an informative set of universally conserved sequences. Examining the intervals between pairs of these segments defined highly conserved segments of the genome versus ones that have structurally related polymorphisms. We identified a Pangenome landscape of 60,764 polymorphic intervals with unique and geo-ethnic features. Overall, this study of the Pangenome revealed the conserved versus divergent features including the landscape of polymorphic structural variants.

genomics↗

bioRxiv2021

Magnetic DNA random access memory with nanopore readouts and exponentially-scaled combinatorial addressing

The storage of data in DNA typically involves encoding and synthesizing data into short oligonucleotides, followed by reading with a sequencing instrument. Major challenges include the molecular consumption of synthesized DNA, issues with basecalling errors, and limitations with scaling up read access operations for individual data elements. Addressing these challenges, we describe a DNA storage system called MDRAM (Magnetic DNA-based Random Access Memory) that enables repetitive and efficient readouts of targeted files with nanopore-based sequencing. Through conjugation of synthesized DNA to magnetic beads, we enabled repeated readouts of data while preserving the original DNA analyte and maintaining data readout quality. MDRAM also utilizes an efficient convolutional coding scheme that leverages soft information in raw nanopore sequencing signals to achieve information reading costs comparable to Illumina sequencing despite substantially higher error rates. Finally, we demonstrate a proof-of-concept DNA-based proto-filesystem that enables an exponentially-scalable data address space using only small numbers of targeting primers for assembly and readout. ONE-SENTENCE SUMMARYWe demonstrate a novel DNA data storage system that leverages conjugation of DNA onto magnetic beads, new computational advances in data encoding, and exponentially scalable access of individual data elements.

synthetic biology↗

bioRxiv2021

NanoSpring: reference-free lossless compression of nanopore sequencing reads using an approximate assembly approach

The amount of data produced by genome sequencing experiments has been growing rapidly over the past several years, making compression important for efficient storage, transfer and analysis of the data. In recent years, nanopore sequencing technologies have seen increasing adoption since they are portable, real-time and provide long reads. However, there has been limited progress on compression of nanopore sequencing reads obtained in FASTQ files since most existing tools are either general-purpose or specialized for short read data. We present NanoSpring, a reference-free compressor for nanopore sequencing reads, relying on an approximate assembly approach. We evaluate NanoSpring on a variety of datasets including bacterial, metagenomic, plant, animal, and human whole genome data. For recently basecalled high quality nanopore datasets, NanoSpring, which focuses only on the base sequences in the FASTQ file, uses just 0.35-0.65 bits per base which is 3-6x lower than general purpose compressors like gzip. NanoSpring is competitive in compression ratio and compression resource usage with the state-of-the-art tool CoLoRd while being significantly faster at decompression. NanoSpring is available on GitHub at https://github.com/qm2/NanoSpring.

bioinformatics↗

bioRxiv2021

KmerKeys: a web resource for searching indexed genome assemblies and variants

K-mers are short DNA sequences that are used for genome sequence analysis. Applications that use k-mers include genome assembly and alignment. Despite these current applications, the wider bioinformatic use of k-mers in has challenges related to the massive scale of genomic sequence data. A single human genome assembly has billions of these short sequences. The sheer amount of computation for effective use of k-mer information is enormous, particularly when involving multiple genome assemblies. To address these issues, we developed a new k-mer indexing data structure based on a hash table tuned for the lookup of k-mer keys. This web application, referred to as KmerKeys (https://kmerkeys.dgi-stanford.org/), provides performant, rapid query speeds for cloud computation on genome assemblies. We enable fuzzy as well as exact k-mer-based searches of assemblies. To enable robust and speedy performance, the website implements cache-friendly hash tables, memory mapping and massive parallel processing. Our method employs a scalable and efficient data structure that can be used to jointly index and search a large collection of human genome assembly information. One can include variant databases and their associated metadata such as the gnomAD population variant catalog. This feature enables the incorporation of future genomic information into sequencing analysis.

bioinformatics↗

bioRxiv2020

Impact of lossy compression of nanopore raw signal data on basecall and consensus accuracy

MotivationNanopore sequencing provides a real-time and portable solution to genomic sequencing, enabling better assembly, structural variant discovery and modified base detection than second generation technologies. The sequencing process generates a huge amount of data in the form of raw signal contained in fast5 files, which must be compressed to enable efficient storage and transfer. Since the raw data is inherently noisy, lossy compression has potential to significantly reduce space requirements without adversely impacting performance of downstream applications. ResultsWe explore the use of lossy compression for nanopore raw data using two state-of-the-art lossy time-series compressors, and evaluate the tradeoff between compressed size and basecalling/consensus accuracy. We test several basecallers and consensus tools on a variety of datasets at varying depths of coverage, and conclude that lossy compression can provide 35-50% further reduction in compressed size of raw data over the state-of-the-art lossless compressor with negligible impact on basecalling accuracy ([lsim]0.2% reduction) and consensus accuracy ([lsim]0.002% reduction). In addition, we evaluate the impact of lossy compression on methylation calling accuracy and observe that this impact is minimal for similar reductions in compressed size, although further evaluation with improved benchmark datasets is required for reaching a definite conclusion. The results suggest the possibility of using lossy compression, potentially on the nanopore sequencing device itself, to achieve significant reductions in storage and transmission costs while preserving the accuracy of downstream applications. AvailabilityThe code is available at https://github.com/shubhamchandak94/lossy_compression_evaluation. Supplementary informationSupplementary data are available at Bioinformatics online. Contactschandak@stanford.edu

bioinformatics↗

bioRxiv2019

Overcoming high nanopore basecaller error rates for DNA storage via basecaller-decoder integration and convolutional codes

As magnetization and semiconductor based storage technologies approach their limits, bio-molecules, such as DNA, have been identified as promising media for future storage systems, due to their high storage density (petabytes/gram) and long-term durability (thousands of years). Furthermore, nanopore DNA sequencing enables high-throughput sequencing using devices as small as a USB thumb drive and thus is ideally suited for DNA storage applications. Due to the high insertion/deletion error rates associated with basecalled nanopore reads, current approaches rely heavily on consensus among multiple reads and thus incur very high reading costs. We propose a novel approach which overcomes the high error rates in basecalled sequences by integrating a Viterbi error correction decoder with the basecaller, enabling the decoder to exploit the soft information available in the deep learning based basecaller pipeline. Using convolutional codes for error correction, we experimentally observed 3x lower reading costs than the state-of-the-art techniques at comparable writing costs. The code, data and Supplementary Material is available at https://github.com/shubhamchandak94/nanopore_dna_storage.

bioengineering↗

bioRxiv2019

Improved read/write cost tradeoff in DNA-based data storage using LDPC codes

With the amount of data being stored increasing rapidly, there is significant interest in exploring alternative storage technologies. In this context, DNA-based storage systems can offer significantly higher storage densities (petabytes/gram) and durability (thousands of years) than current technologies. Specifically, DNA has been found to be stable over extended periods of time which has been demonstrated in the analysis of organisms long since extinct. Recent advances in DNA sequencing and synthesis pipelines have made DNA-based storage a promising candidate for the storage technology of the future.\n\nRecently, there have been multiple efforts in this direction, focusing on aspects such as error correction for synthesis/sequencing errors and erasure correction for handling missing sequences. The typical approach is to use separate codes for handling errors and erasures, but there is limited understanding of the efficiency of this framework. Furthermore, the existing techniques use short block-length codes and heavily rely on read consensus, both of which are known to be suboptimal in coding theory.\n\nIn this work, we study the tradeoff between the writing and reading costs involved in DNA-based storage and propose a practical scheme to achieve an improved tradeoff between these quantities. Our scheme breaks with the traditional separation framework and instead uses a single large block-length LDPC code for both erasure and error correction. We also introduce novel techniques to handle insertion and deletion errors introduced by the synthesis process. For a range of writing costs, the proposed scheme achieves 30-40% lower reading costs than state-of-the-art techniques on experimental data obtained using array synthesis and Illumina sequencing.\n\nThe code, data, and Supplementary Material is available at https://github.com/shubhamchandak94/LDPC_DNA_storage.

bioengineering↗

bioRxiv2019

Cross-linked RNA Secondary Structure Analysis using Network Techniques

Next generation sequencing and biochemical cross-linking methods have been combined into powerful tools to probe RNA secondary structure. One such method, known as PARIS, has been used to produce near base-pair maps of long-range and alternative RNA structures in living cells. However, the procedure for generating these maps typically relies on laborious manual analysis. We developed an automated method for producing RNA secondary structure maps using network analysis techniques. We produced an analysis pipeline, dubbed cross-linked RNA secondary structure analysis using network techniques (CRSSANT), which automates the grouping of gapped RNA sequencing reads produced using the PARIS assay, and tests the validity of secondary structures implied by the groups. We validated the clusters and secondary structures produced by CRSSANT using manually-produced grouping maps and known secondary structures. We implemented CRSSANT in Python using the network analysis package NetworkX and RNA folding software package ViennaRNA. CRSSANT is fast and efficient, and is available as Python source code at https://github.com/ihwang/CRSSANT.

bioinformatics↗

bioRxiv2019

Denoising of Aligned Genomic Data

Noise in genomic sequencing data is known to have effects on various stages of genomic data analysis pipelines. Variant identification is an important step of many of these pipelines, and is increasingly being used in clinical settings to aid medical practices. We propose a denoising method, dubbed SAMDUDE, which operates on aligned genomic data in order to improve variant calling performance. Denoising human data with SAMDUDE resulted in improved variant identification in both individual chromosome as well as whole genome sequencing (WGS) data sets. In the WGS data set, denoising led to identification of almost 2,000 additional true variants, and elimination of over 1,500 erroneously identified variants. In contrast, we found that denoising with other state-of-the-art denoisers significantly worsens variant calling performance. SAMDUDE is written in Python and is freely available at https://github.com/ihwang/SAMDUDE.

bioinformatics↗