Converting h5ad files

Converting h5ad files#

deltacells convert turns any number of h5ad files into one deltacells dataset. The files can be any size, and local or in Google Cloud Storage. The converter sorts the genes, re-chunks the cells into tiles of exactly the same size, compresses them, and stores the cell metadata (obs) and the gene table (var) alongside.

It needs the h5ad and obs extras (pip install "deltacells[h5ad,obs]"; see Install).

What your files need#

  • Raw integer counts between 0 and 2,147,483,647. Normalized or float data is rejected.

  • The same genes in every file. All files must have identical var_names, in the same order.

  • CSR storage for X. Dense X is converted too, but much more slowly.

Convert#

From files on disk:

deltacells convert --h5ad-glob "shards/*.h5ad" --output my_dataset

Or from a list of files, one path per line (blank lines and lines starting with # are skipped):

deltacells convert --file-list files.txt --output my_dataset

The cells keep the order of the files and of the cells within each file. The files are sorted naturally (shard_2 comes before shard_10); add --keep-order to use the order you gave instead. The converter does not shuffle cells. A tile holds consecutive cells, so if your input is sorted by donor or tissue, shuffle the cells across files first.

When it finishes, look at the result and check it:

deltacells info my_dataset      # cells, genes, tiles, size on disk
deltacells verify my_dataset    # checks the CRC of every tile and decodes it

Common options#

option

default

meaning

--tile-size

10000

cells per tile; every tile but the last holds exactly this many

--level

19

zstd level: higher is smaller but slower to write (see below)

--workers

number of cores

processes that encode tiles in parallel (limited by memory)

--no-sort-genes

off

keep the genes in their original order (this costs compression, see Encoding)

--overwrite

off

replace an existing dataset at --output

--no-obs, --no-var

off

do not store the cell metadata or the gene table

--obs-exclude a,b

none

leave out these obs columns

Run deltacells convert --help for all of them.

Choosing the compression level#

The default --level 19 gives the smallest files and takes about 25 core-seconds to compress a 10,000-cell tile. Decoding is not slower at higher levels, so a higher level only costs time when writing. For a very large dataset, --level 12 compresses about 4 times faster for files that are 6% larger, and --level 3 about 7 times faster for files 11% larger. The benchmarks page has the measurements.

Converting from and to Google Cloud Storage#

Install the extra (pip install "deltacells[gcs]"), and give gs:// paths for the inputs, the output, or both:

gcloud storage ls 'gs://my-bucket/h5ad/*.h5ad' > files.txt
deltacells convert --file-list files.txt \
    --output gs://my-bucket/deltacells/my_dataset \
    --staging-dir /mnt/scratch --workers 12

gs:// paths are not expanded by the converter, so list them into a file as above. The list can also be piped in (--file-list -).

Credentials. deltacells uses Google Application Default Credentials, which are not the login that gcloud and gsutil use. If reading works but the first upload fails with Forbidden ... Provided scope(s) are not authorized, run this once:

gcloud auth application-default login

The account needs read access to the inputs and write access to the output. Before reading anything, the converter uploads and deletes a small test file in the output, so a missing permission is reported within seconds.

Disk space. Nothing has to fit on the local disk. Each input file is downloaded into --staging-dir just before its first tile needs it and deleted after its last tile is written, and each finished tile is uploaded and deleted at once. Plan for about 5 files per worker plus a few tiles (12 workers on 60 to 100 MB files need a few GB); the run prints its estimate. The staging directory defaults to the system temp directory, so point it at a disk with room.

Time. Compression is the bottleneck, not the network. As a guide, converting 94 million cells from 9,447 files with 12 workers on a 16-vCPU VM took on the order of 11 hours at the default level. Use physical cores, not vCPUs, when choosing --workers.

A note for scripts. The converter starts worker processes with spawn, so a Python script that calls deltacells.convert.convert_h5ad needs the usual if __name__ == "__main__": guard. The deltacells command does not.

Progress and safety#

While it runs, the converter prints one line when each phase starts and ends, and every --progress-interval seconds (60 by default) a progress line with the speed and an estimate of the time left.

  • The input files are only read, never changed or deleted.

  • --overwrite deletes the existing dataset at --output first, but only files that a deltacells dataset has. If the output holds anything else, the converter stops and deletes nothing. It also refuses to run if an input file lies inside the output. Without --overwrite, a non-empty output is an error.

  • manifest.json is written last, so a directory or prefix without a manifest is not a dataset.

  • If a conversion fails, for example on an unreadable file, the tiles written so far stay in place but there is no manifest, and the error says so. There is no resume: start again with --overwrite.

From Python#

from deltacells.convert import convert_h5ad

convert_h5ad(["shard_0.h5ad", "shard_1.h5ad"], "my_dataset", tile_size=10_000)