By anam-org
Manage feature versions, dependencies, and data lineage for multimodal ML pipelines using a Python framework with metadata stores and CLI tools, plus explore feature graphs via MCP.
Metaxy is a metadata layer for multimodal Data and ML pipelines. Metaxy tracks lineage and versioning across complex computational graphs for multimodal datasets. Metaxy can cache every single sample and scale to handle millions of them.
Metaxy manages metadata while data typically lives elsewhere:
┌─────────────────────────────────┐ ┌─────────────────────────┐
│ Metadata (Metaxy) │ │ Data (e.g., S3) │
├──────┬──────────┬──────┬────────┤ │ │
│ ID │ path │ size │version │ │ 📦 s3://my-bucket/ │
├──────┼──────────┼──────┼────────┤ │ │
│ img1 │ s3://... │ 2.1M │a3fdsf │ ────────>│ ├─ img1.jpg │
│ img2 │ s3://... │ 1.8M │b7e123 │ ────────>│ ├─ img2.jpg │
└──────┴──────────┴──────┴────────┘ └─────────────────────────┘
The feature that makes Metaxy stand out is the ability to track partial data dependencies and detect prunable updates — updates that don't trigger change propagation through certain paths in the dependency graph because they modify fields that aren't dependencies of those downstream features. For example, updating audio upstream of a face recognition step allows pruning the face recognition branch since it only depends on video frames. This problem is specific to multimodal pipelines and doesn't typically emerge in traditional data engineering.
Metaxy's goal is to provide a standard instrument for any kind of multimodal (or purely tabular) incremental pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata. Or, in short, to be a universal glue for incremental data pipelines.
Metaxy is very reliable and is fanatically tested across all supported Python versions and platforms 1.
Read the docs to learn more.
Install Metaxy from PyPI:
uv add metaxy
Metaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.
Metaxy provides integrations with popular tools such as Dagster, Ray, ClickHouse, DeltaLake, SQLModel.
The full list can be found here.
Announcement post by Anam
Dagster + Metaxy by Dagster labs
Docling + Slurm + Metaxy by ASCII
See CONTRIBUTING.md.
The CLI is not tested on Windows yet. ↩
Own this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claimOwn this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claimBased on adoption, maintenance, documentation, and repository signals. Not a security audit or endorsement.
npx claudepluginhub anam-org/metaxy --plugin metaxyTrack and manage model versions
Data engineering plugin - warehouse exploration, pipeline authoring, Airflow integration
Quick insights from dlt pipeline data. Connect to a pipeline, profile tables, plan charts, and assemble marimo dashboards.
Self-documenting, self-improving framework for analytical repositories
DataHub development and interaction toolkit with connector planning, PR review, catalog search, metadata enrichment, lineage tracing, data quality management, and connection setup skills
Connect to Knowledge Catalog to discover, manage, monitor, and govern data and AI artifacts across your data platform