release-incident-response

Release Incident Response

Required connectors: ClickUp, GitHub. See CONNECTORS.md.

This skill operationalises the Auro 2.0 Release Incident Response process. The canonical document is the Release Incident Response page in the ClickUp Release Process Playbook; this skill is the active companion that helps the release runner work the process under pressure.

When to invoke

Invoke this skill when any of the following is happening within the one-hour active monitoring window after a release:

• The post-deploy smoke test fails or behaves unexpectedly.
• An error or latency spike appears in dashboards (Application Insights, Sentry) that was not present before the deploy.
• A PagerDuty page fires on a service that was just released to.
• A customer or internal user reports a problem that correlates with the deploy time.
• The deploy pipeline itself fails partway through (deploy succeeded but migration failed, or vice versa).

If the symptoms started before the deploy or have no plausible link to the change set, stop — this is a general production incident; defer to the Alerting & Incident Management Plan.

What to do when invoked

This skill is a tactical companion. When invoked:

1. Confirm the situation looks like a release-caused incident. Ask one or two clarifying questions only if needed.
2. Walk the user through the action sequence in the right order, surfacing only the next step rather than the whole runbook.
3. Help with the rollback-vs-fix-forward decision using the documented criteria.
4. Draft Teams updates, in-app banner copy, status-page entries, and the resolution post when asked.
5. Build the postmortem-entry timeline from the Teams thread when the incident is being closed.
6. Never invent decisions on the user's behalf — surface the criteria, ask for the user's call, then act.

Severity reference (existing scheme)

Level	Criteria	Default action
P1	Production down, data loss, security breach, all customers affected	Page on-call immediately. Roll back unless rollback unsafe.
P2	Major feature broken, significant degradation, partial outage	15-min response. Roll back unless small fix already prepared.
P3	Minor issues, single customer, non-critical errors	4-hour response. Usually fix forward in next release.
P4	Cosmetic, warnings, low-impact	Ticket. Fix forward in next release.

The action sequence

T+0 — Detect & acknowledge (within 1 minute)

• Acknowledge any PagerDuty page immediately.
• Open or join the incident Teams thread. Use the opener template:

INCIDENT — [severity] — [one-line symptom] — deploy at [time] — investigating

T+5 — Triage (within 5 minutes)

• Confirm the symptom on production directly — not via a single dashboard.
• Confirm the timing correlates with the most recent GitHub release.
• Set a working severity per the table above. Restate it in the thread.
• Pull in the technical lead (typically the release author) if not already involved.

T+15 — Decision (within 15 minutes)

Apply the rollback-vs-fix-forward criteria. Roll back unless all of the following are true:

• The fix is a small, isolated change the technical lead has already identified.
• The fix can be written, reviewed, and verified on staging in under 15 minutes.
• Rolling back would itself cause customer impact (e.g., destructive migration already ran, or it would lose data captured under the new version).

If any point is uncertain, roll back. If rolling back, hand off to the release-rollback skill — the release runner cannot self-serve a production rollback; the tech lead or EM has to trigger the workflow.

T+30 — Customer comms

For P1 / P2 customer-visible incidents:

• Raise the in-app banner. Banner copy template:

We are currently investigating a problem with [feature].
Updates will be posted here as we learn more.

• Open the public status page incident at the same impact level.
• Update the Teams thread with the customer-facing wording used.

T+45 — Verify

• Confirm the rollback or hotfix has reached production.
• Re-run the post-deploy smoke test.
• Confirm the original symptom has cleared.
• Watch dashboards and Sentry for at least 15 minutes after the fix lands.

T+60 — Resolve & handover

Resolution post template for the Teams thread:

RESOLVED — [one-line symptom] — [duration: HH:MM]
Cause: [one-line cause, can be "TBC, see postmortem"]
Action taken: [rollback / hotfix vX.Y.Z]
Customers affected: [scope]
Postmortem: [ClickUp link]

Open the postmortem entry against the release rollup. P1 / P2 get a full postmortem within 5 working days; P3 gets a short note on the rollup; P4 gets a brief note on the rollup.

Standard outputs to offer

When invoked, offer to draft any of:

• Teams thread opener — using the template above with current severity and symptom.
• Status update for the Teams thread — concise, factual, what changed since the last update.
• In-app banner copy — at investigating, mitigating, or resolved stages.
• Status page entry — investigating, identified, monitoring, or resolved.
• Customer outreach email — for direct outreach to a single affected customer.
• Resolution post — using the template above.
• Postmortem timeline draft — built from the Teams thread, ready for the standard postmortem template.

Behavioural rules

• Do not declare severity on the user's behalf. Surface the criteria, recommend, and let the user confirm.
• Default to rollback for P1 / P2. If the user wants to fix forward, walk through the three criteria and ask explicitly which they have satisfied.
• Do not promise a timeline to customers. Banner and status-page copy uses "investigating", "mitigating", or "resolved" — never specific minutes-to-fix.
• Push to documentation, not memory. When the user is uncertain about a step, point them to the relevant section of the Release Incident Response page rather than improvising a substitute process.
• Stay above the diagnosis. This skill helps run the response; it does not diagnose the bug. The technical lead does that. Do not get pulled into root-cause analysis at the expense of comms and process.

Hand-offs

• Rolling back? Hand to the release-rollback skill / Rollback Runbook.
• Postmortem? Hand to the existing Postmortem Process page in the playbook.
• General incident not caused by a release? Hand to the Alerting & Incident Management Plan.

References

• Release Incident Response (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-9174
• Rollback Runbook (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-9194
• Release Process Overview (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-8974
• Alerting & Incident Management Plan (ClickUp): https://app.clickup.com/9014870220/v/dc/8cn806c-1774/8cn806c-6014