release-rollback

Rollback Runbook

Required connectors: ClickUp, GitHub. See CONNECTORS.md.

This skill operationalises the Auro 2.0 Rollback Runbook. The canonical document is the Rollback Runbook page in the ClickUp Release Process Playbook; this skill is the active companion that helps the rollback executor run the process under pressure.

When to invoke

Invoke this skill when:

• The release-incident-response process has decided to roll back rather than fix forward, and the team is ready to execute.
• A staging release has broken staging badly enough that a rollback is the cleaner path (rare — staging is usually fix-forward).

Do not invoke this skill if a rollback decision has not been made — the rollback-vs-fix-forward criteria sit in release-incident-response. Do not invoke this skill for general production incidents not caused by a recent release.

What to do when invoked

1. Confirm the rollback decision has been made and a rollback executor (tech lead or EM) is in the conversation. If the user is the release runner only, immediately surface that the release runner cannot self-serve a production rollback and that the tech lead / EM must be paged.
2. Walk through the pre-flight checks and surface anything missing.
3. Help identify the previous good GitHub release (target version + commit hash).
4. Talk the executor through the rollback workflow trigger.
5. Help verify the rollback in production.
6. Hand back to release-incident-response for resolution and customer comms.

Critical access constraint

Production rollback requires the tech lead or engineering manager. The release runner cannot self-serve. If the user appears to be the release runner without a tech lead / EM in the loop, the first action is:

Page the on-shift tech lead / EM via PagerDuty before going any further. Production rollback workflow access is restricted to those roles.

Pre-flight checks

Before triggering the rollback workflow, the rollback executor confirms all of:

1. Target version identified — previous good GitHub release named, with tag and commit hash.
2. No destructive migration ran — current release's migrations reviewed; if any was destructive (column drop, type narrowing, table removal, irreversible data transformation), stop and reassess.
3. No data captured under the new version is at risk — if the new release accepted writes the previous version cannot read, rolling back will look like data loss.
4. No hotfix in flight — no-one is mid-deploy of a hotfix at the same time.
5. GitHub Actions and Azure are healthy — a green run today is a reasonable signal; if either is degraded, rollback may not work.

If any check fails, do not roll back until the situation is reassessed with the incident commander. A botched rollback is worse than a longer outage.

Pre-flight post template for the Teams thread:

Pre-flight: target = vX.Y.Z (commit abc1234), no destructive migrations, no data-at-risk,
no hotfix in flight, deploy infra healthy. Triggering rollback workflow now.

Production rollback — click-by-click

Step 1 — Identify the target version

• GitHub repo → Releases page.
• Find the release that was live before the failed deploy (recorded against the release rollup in ClickUp).
• Note the tag (e.g. v2026.04.30-1) and the commit hash. Both go in the Teams thread.

Step 2 — Trigger the rollback workflow

• GitHub Actions → production rollback workflow (specific workflow filename TBD per the runbook's open items).
• Click Run workflow.
• Inputs — target environment = production, target version / tag = the tag from Step 1.
• Click Run workflow to trigger.
• Post the workflow run URL in the Teams thread immediately.

Step 3 — Watch the workflow

• Tail the workflow logs.
• Confirm each job (checkout, build, deploy, smoke) completes successfully.
• If any step fails, stop and escalate. Do not retry blindly. See edge cases below.

Step 4 — Verify rollback in production

• Confirm running version in production matches the target tag (location of version exposure is TBD per the runbook's open items — typically a /version endpoint or response header).
• Re-run the post-deploy smoke test against production — release-critical flows plus standard regression flows.
• Confirm the original incident symptom has cleared.
• Watch Application Insights, Sentry, and PagerDuty for at least 15 minutes.

Verification post template:

Rollback verified. Production now on vX.Y.Z (commit abc1234).
Symptom cleared at HH:MM. Dashboards clean for 15 min.

Step 5 — Hand back to incident response

The rollback is complete; the incident is not. Hand back to the release-incident-response skill / process for resolution declaration and customer-comms close-out. The technical lead opens the post-rollback hotfix branch so the failed release can be re-shipped, fixed, in the next slot.

Staging rollback

Staging is normally fix-forward. Roll staging back only when:

• Staging is so broken the team cannot use it for verification of the next prod release.
• Fix-forward is more than a few hours away.
• No in-flight engineering work depends on changes in current staging.

Mechanics same as production — identify previous good staging release, trigger the rollback workflow with target environment = staging, verify with the staging smoke test, note in the Teams thread and on the staging release rollup. No formal SLA. No customer comms.

Edge cases

Destructive migration ran

• Do not roll back automatically. Escalate to tech lead / EM and DBA / data engineer.
• Choose between forward-fix on the new schema (usually correct) or coordinated reverse migration plus rollback (slow, not a runbook step).
• Treat data captured under the new release as in-scope for the postmortem.

Workflow run fails partway through

• Stop. Do not retry blindly.
• Read failed step logs and post a failure summary in the Teams thread.
• Page platform / DevOps if the failure is infrastructure (GitHub Actions, Azure).
• If the workflow partially deployed, treat as P1 escalation — production may be in a worse state than the original incident.

Multi-service release

• Don't roll back one service and leave the other on the new version unless version skew is verified safe.
• Run the rollback for each affected service.
• Verify each service independently.

Hotfix in flight

• Pause. Confirm in the Teams thread which path is being chosen — rollback or hotfix. Two simultaneous deploys must not happen.
• If the hotfix is already partway through, allow it to complete, then verify before deciding whether rollback is still needed.

Cannot identify a known-good previous version

• Walk the GitHub Releases list backwards with the technical lead.
• Cross-reference with monitoring history for stability.
• If no known-good version can be confidently identified within 10 minutes, switch to fix-forward via hotfix; rollback to an unverified version is worse than an extended outage.

SLA expectations

Phase	Target time
Rollback decision → Pre-flight checks complete	5 minutes
Pre-flight complete → Rollback workflow triggered	1 minute
Workflow complete → Verification complete	15 minutes
Total — decision to verified rollback	Under 30 minutes

If any phase materially overruns, post a status update in the Teams thread and consider escalation.

Behavioural rules

• Surface the access constraint early. If the user is the release runner only, push them to page the tech lead / EM before any other step.
• Do not invent rollback workflow specifics. Filename, input names, and version-exposure locations are TBD per the runbook's open items. If the user asks for these, point them to the runbook page and offer to help draft updates once they have the details.
• Insist on the pre-flight checks. Do not move to triggering the workflow until all five checks have been confirmed in the Teams thread — even under time pressure.
• Default to escalation on workflow failure. Never recommend a blind retry of a failed rollback workflow.
• Treat destructive migrations as a stop-everything signal. This is the highest-friction edge case in the runbook and the one most likely to cause data damage if rushed.
• Hand back, don't continue. Once the rollback is verified, immediately hand back to release-incident-response for the resolution and comms close-out. Do not extend into the resolution flow yourself.

Standard outputs to offer

When invoked, offer to draft any of:

• Pre-flight check post — using the template above with the named target version and the five checks confirmed.
• Workflow trigger announcement — short post with the workflow run URL.
• Workflow status update — what step is currently running, time elapsed, anomalies if any.
• Verification post — using the template above.
• Failure / escalation post — short, factual summary of what failed and what is being escalated.
• Hotfix kick-off note — for the technical lead, summarising the rolled-back release and the symptom that needs fixing.

References

• Rollback Runbook (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-9194
• Release Incident Response (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-9174
• Release Process Overview (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-8974
• Alerting & Incident Management Plan (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-6014