compression-codecs

Compression Codecs for Zarr

Configure, select, and optimize compression codecs for Zarr arrays using numcodecs. This skill covers every compressor and filter in the Zarr ecosystem, thread safety for multi-process workloads, codec pipelines in Zarr v3, and performance trade-offs.

Zarr Performance Guide: https://zarr.readthedocs.io/en/latest/user-guide/performance/ numcodecs Reference: https://numcodecs.readthedocs.io/ GitHub: https://github.com/zarr-developers/numcodecs

Quick Reference: Codec Selection

Codec	Speed (compress)	Speed (decompress)	Ratio	Best For
Blosc+LZ4	Very Fast	Very Fast	Low-Med	Real-time analysis, frequent reads
Blosc+Zstd	Medium	Fast	High	General purpose (v2 default)
Zstd standalone	Medium	Fast	High	Zarr v3 default
Blosc+LZ4HC	Slow	Very Fast	Medium	Write-once, read-many
Gzip	Slow	Medium	Med-High	Interop with non-Python tools
LZ4 standalone	Very Fast	Very Fast	Low	Maximum throughput
LZMA	Very Slow	Very Slow	Very High	Archival only

When to Use This Skill

• Selecting a codec for a new Zarr array or dataset
• Optimizing storage size vs read/write speed
• Configuring pre-compression filters (Delta, Quantize, etc.)
• Debugging compression issues (thread safety, poor ratio, codec not found)
• Migrating between v2 and v3 codec configuration

Core Concepts

1. Blosc: The Meta-Compressor (v2 Default)

Blosc wraps internal algorithms and adds byte-shuffling — the single most impactful setting for numerical data compression. Shuffle rearranges bytes to expose patterns, yielding 10–40× better ratios.

Parameter	Options	Default
cname	blosclz, lz4, lz4hc, snappy, zlib, zstd	blosclz
clevel	0 (none) – 9 (max)	5
shuffle	NOSHUFFLE (0), SHUFFLE (1), BITSHUFFLE (2)	SHUFFLE

from numcodecs import Blosc
Blosc(cname='zstd', clevel=5, shuffle=Blosc.SHUFFLE)    # balanced
Blosc(cname='lz4', clevel=1, shuffle=Blosc.SHUFFLE)     # max speed
Blosc(cname='zstd', clevel=9, shuffle=Blosc.BITSHUFFLE) # max ratio

2. Blosc Thread Safety (CRITICAL)

Blosc's internal threading is not fork-safe. Multi-process use (Dask workers, multiprocessing, joblib) can cause silent data corruption.

from numcodecs import blosc
blosc.use_threads = False  # ALWAYS set this in multi-process environments

# For Dask distributed:
client.run(lambda: setattr(__import__('numcodecs').blosc, 'use_threads', False))

3. Standalone Codecs

Codec	Import	Key Config
Zstd (v3 default)	from numcodecs import Zstd	Zstd(level=3) — levels 1–22
LZ4	from numcodecs import LZ4	LZ4(acceleration=1)
Gzip	from numcodecs import GZip	GZip(level=5) — levels 1–9
Zlib	from numcodecs import Zlib	Zlib(level=4) — levels 1–9
BZ2	from numcodecs import BZ2	BZ2(level=5) — levels 1–9
LZMA	from numcodecs import LZMA	LZMA(preset=6) — presets 0–9

4. Pre-Compression Filters

Filters transform data before compression to improve ratios. Applied in order.

Filter	Use Case	Example
Delta	Monotonic data (timestamps, indices)	Delta(dtype='int64')
Quantize	Reduce float precision	Quantize(digits=3, dtype='float64')
FixedScaleOffset	Convert floats to ints	FixedScaleOffset(offset=273.15, scale=100, dtype='float64', astype='int32')
PackBits	Boolean arrays (8× reduction)	PackBits()
Categorize	String→integer encoding	Categorize(labels=['a','b','c'], dtype='U10', astype='u1')

# v2: filters + compressor
z = zarr.open_array('data.zarr', mode='w', shape=(10000,), dtype='int64', chunks=(1000,),
                    filters=[Delta(dtype='int64')], compressor=Blosc(cname='zstd', clevel=5))

# Chain: Delta → Quantize → compressor
filters=[Delta(dtype='float64'), Quantize(digits=3, dtype='float64')]

5. Zarr v3 Codec Pipeline

v3 replaces compressor + filters with a unified pipeline: array→array → array→bytes → bytes→bytes.

import zarr

# v3 with default Zstd
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', zarr_format=3)

# v3 with explicit compressor
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=zarr.codecs.ZstdCodec(level=5))

# No compression
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=None)

6. Codec Selection Decision Tree

Primary constraint?
├── STORAGE SIZE → Zstd level 9 or LZMA (archival only)
├── READ SPEED  → Blosc+LZ4 with SHUFFLE (numerical) or LZ4 standalone
├── WRITE SPEED → LZ4(acceleration=10) or Blosc+LZ4 clevel=1
├── BALANCED    → Blosc+Zstd clevel=3 (v2) or Zstd level=3 (v3)
├── INTEROP     → Gzip (universal) or Zlib (NetCDF compat)
└── DATA TYPE
    ├── Monotonic → Delta filter + any compressor
    ├── Boolean   → PackBits + LZ4
    ├── Integer   → Blosc BITSHUFFLE
    └── Limited precision float → Quantize filter + Zstd

Best Practices

• Start with defaults (Blosc+Zstd for v2, Zstd for v3) — only change if benchmarks justify it
• Always use shuffle for numerical data (SHUFFLE for floats, BITSHUFFLE for ints)
• Never use LZMA/BZ2 for frequently-read data
• Set blosc.use_threads = False in any multi-process environment
• Benchmark on real data — synthetic data gives misleading results
• For cloud storage, prioritize decompression speed over compression ratio
• Compression level has diminishing returns above level 5 for most codecs

References

• Patterns: references/patterns.md — codec configs, filter pipelines, per-variable encoding
• Examples: references/examples.md — benchmarking, real-world codec selection
• Common Issues: references/common-issues.md — corruption, poor ratio, codec errors
• Benchmark Script: assets/codec-comparison.py — runnable codec comparison

External Links

• numcodecs: https://numcodecs.readthedocs.io/
• Blosc: https://www.blosc.org/
• Zstandard: https://facebook.github.io/zstd/
• Squash Benchmark: https://quixdb.github.io/squash-benchmark/