fleet-rollback

Fleet Rollback

When a change applied to many repos at once breaks things, you need to roll back all of them in parallel. The risk is that fixes for one repo depend on the bad commit, so reverts need to be carefully sequenced.

The Pattern

1. Identify the breaking commit — the SHA, commit message, or PR
2. Find every repo that has it — git log --all --grep="<message>" or GitHub search
3. Decide the rollback strategy:
   • git revert <sha> — preserves history, safe for shared branches
   • git reset --hard <previous-sha> && git push --force-with-lease — clean history, risky for shared branches
4. Apply in parallel — delegate_task to multiple agents, or batch via Python
5. Verify each repo — re-trigger CI Build, check for green

When to Use

• A bad Prisma 5→7 migration broke 3+ repos (e.g., the lazy proxy pattern has a runtime bug)
• A Dependabot config that auto-archives repos (now archived, can't be re-archived from archived state)
• A CI workflow change that causes infinite loops
• An .env* .gitignore pattern that accidentally gitignores package.json

How to Avoid Needing This

• Test on 1 repo first, wait 24 hours, verify nothing broke
• Use feature flags where possible
• For fleet-wide changes, use dry_run: true flags in your scripts
• Have a documented rollback path BEFORE the change

Recipe

# Find the breaking commit SHA
gh search commits --owner camster91 "<message>" --json sha,repository --limit 50

# For each affected repo, revert
for repo in <list>; do
    cd /tmp/$repo
    git clone https://github.com/camster91/$repo.git . --depth=10
    git log --oneline -5  # find the bad commit
    git revert <bad_sha> --no-edit
    git push origin main --force-with-lease
done

Common Pitfalls

1. Force-push on protected branches — use git push --force-with-lease (checks if remote changed)
2. Reverting a merge commit — git revert -m 1 <merge_sha> to specify which parent
3. Reverting on a branch with newer commits that depend on the bad one — the revert will fail with conflicts; resolve manually
4. Forgetting to trigger CI — after rollback, the workflows don't auto-run; trigger with gh workflow run

Related Skills & Chains

• fleet-ci-audit — Identifies which repos are affected
• branch-protection — Understanding force-push on protected branches