deltacells

deltacells#

Fast, compact tiles of single-cell count matrices, built for feeding a training DataLoader.

Training a model on tens to hundreds of millions of cells means reading raw counts over and over, and the format they are stored in determines how fast that goes and how much disk it takes. deltacells stores a count matrix as a directory (or bucket prefix) of tiles of 10,000 cells (configurable). Each tile is one self-contained object that a small C library decodes straight into CSR arrays.

Why deltacells#

  • Very fast data loading. A 10,000-cell tile decodes in about 0.1 s on one core and 0.02 s on eight, with no intermediate AnnData or Arrow objects. A PyTorch DataLoader reading shuffled batches on a laptop sustains 45,000 to 95,000 cells per second, depending on the number of workers (benchmarks).

  • Small on disk. A dataset is about 40% smaller than the same data in BPCells (3.8 vs 6.7 kB per cell) and roughly a third of the size of a gzip-compressed h5ad file. The key idea is to store genes in order of decreasing total counts, which makes the values that remain to be compressed small and repetitive (how the encoding works).

  • Fast enough to keep a GPU fed. On a single thread, decoding is slower than BPCells’ decoder (about 2.6 times as long per tile), in exchange for the smaller files. With a few DataLoader workers, each decoding on its own core, deltacells is designed to keep a GPU busy, as in the data loading of cellarium-ml. Smaller files also mean less to download when the dataset lives in a bucket.

One real 10,000-cell shard (38,094 genes), counts only, read from a local file into memory on an M2 Pro laptop:

format

size on disk

decode, 1 thread

decode, 8 threads

h5ad (gzip)

12.5 kB per cell

1170 ms

single-threaded reader

BPCells (C++ decoder)

6.7 kB per cell

48 ms

30 ms

deltacells

3.8 kB per cell

125 ms

39 ms

The benchmarks page has the details, other hardware, and how to reproduce them.

Quick start#

pip install "deltacells[h5ad,obs]"
deltacells convert --h5ad-glob "shards/*.h5ad" --output my_dataset

import numpy as np
from deltacells import open_dataset

ds = open_dataset("my_dataset")
batch = ds.get_batch(np.array([5, 99_000, 12]))  # any cells, in any order
batch.to_scipy()  # or batch.to_torch_csr()

Contents