hf2ms

hf2ms

Cloud-to-cloud ML repo migration via Modal. Transfer models and datasets between HuggingFace and ModelScope without downloading anything locally.

How It Works

Your machine sends the command. Modal containers do all the work.

Your Machine              Modal Container (ephemeral)              Platforms
┌──────────┐  modal run  ┌──────────────────────────┐  API calls  ┌──────────┐
│ Terminal  │ ─────────> │                          │ <─────────> │ HF Hub   │
│ or Claude │            │  Download from source    │             └──────────┘
│ Code      │            │  Upload to destination   │  API calls  ┌──────────┐
└──────────┘             │  Verify SHA256 hashes    │ <─────────> │ MS Hub   │
                         └──────────────────────────┘             └──────────┘
                              ↕ spins up, transfers, shuts down

No files touch your machine. Modal provisions a container, downloads from the source platform's API, uploads to the destination, verifies SHA256 integrity, and destroys the container. For large repos, it fans out to up to 100 parallel containers.

Features

• Zero local storage — everything transfers cloud-to-cloud on Modal containers
• Parallel chunked migration — splits large repos across up to 100 containers for TB-scale transfers
• SHA256 verification — LFS file hashes checked after upload (skips platform-generated files)
• Auto git fallback — if the Hub API fails (403, storage lock), seamlessly retries via git clone + git lfs pull
• Visibility preservation — private repos stay private on the destination
• Fire & forget — detached mode lets migrations continue after you disconnect
• Bidirectional — HuggingFace → ModelScope and ModelScope → HuggingFace

Quick Start

1. Install Modal

pip install modal
modal token new

2. Set Tokens

cp .env.example .env
# Fill in your tokens, then:

# bash/zsh
export $(cat .env | xargs)

# PowerShell
Get-Content .env | ForEach-Object { if ($_ -match '^([^#].+?)=(.*)$') { [Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

Variable	Where to Get It
HF_TOKEN	https://huggingface.co/settings/tokens (read + write)
MODAL_TOKEN_ID	modal token new or https://modal.com/settings
MODAL_TOKEN_SECRET	Same as above
MODELSCOPE_TOKEN	https://modelscope.ai/my/myaccesstoken
MODELSCOPE_DOMAIN	Optional. modelscope.ai for international (default: modelscope.cn)

3. Validate

python scripts/validate_tokens.py

4. Smoke Test

modal run scripts/modal_migrate.py::hello_world

If this prints SDK versions, you're good to go.

Usage

Single Repo

# HuggingFace → ModelScope (auto-detect type)
modal run scripts/modal_migrate.py::main --source "username/my-model" --to ms

# ModelScope → HuggingFace
modal run scripts/modal_migrate.py::main --source "damo/text-to-video" --to hf --repo-type model

# Custom destination name
modal run scripts/modal_migrate.py::main --source "username/my-model" --to ms --dest "OrgName/model-v2"

# Dataset
modal run scripts/modal_migrate.py::main --source "username/my-dataset" --to ms --repo-type dataset

Windows: Prefix commands with PYTHONIOENCODING=utf-8 to avoid Unicode errors from Modal CLI.

Parallel Mode (Large Repos)

For repos over ~10 GB, parallel mode splits the transfer across multiple containers. Each container clones the repo structure, downloads only its assigned files, and uploads them independently.

# Parallel with default 20 GB chunks
modal run scripts/modal_migrate.py::main --source "org/large-dataset" --to ms --parallel

# Custom chunk size (in GB)
modal run scripts/modal_migrate.py::main --source "org/large-dataset" --to ms --parallel --chunk-size 30

# Parallel dataset
modal run scripts/modal_migrate.py::main --source "org/my-dataset" --to ms --repo-type dataset --parallel

Chunk size auto-adjusts upward if the repo would exceed 100 containers. Parallel mode is currently HuggingFace → ModelScope only.

Batch (Multiple Repos)

Each repo gets its own container, running in parallel via starmap(). Repos that already exist on the destination are automatically skipped.

modal run scripts/modal_migrate.py::batch \
  --source "user/model1,user/model2,user/model3" \
  --to ms --repo-type model

Detached Mode (Fire & Forget)

Add --detach before the script path. The migration continues in Modal's cloud even after you close your terminal:

modal run --detach scripts/modal_migrate.py::main \
  --source "username/my-model" --to ms

Monitor detached runs: