coreweave-incident-runbook

CoreWeave Incident Runbook

Triage Steps

# 1. Check pod status
kubectl get pods -l app=inference -o wide

# 2. Check recent events
kubectl get events --sort-by=.lastTimestamp | tail -20

# 3. Check node status
kubectl get nodes -l gpu.nvidia.com/class -o wide

# 4. Check GPU health
kubectl exec -it $(kubectl get pod -l app=inference -o name | head -1) -- nvidia-smi

Common Incidents

Inference Service Down

1. Check pod status and events
2. If OOMKilled: reduce batch size or upgrade GPU
3. If ImagePullBackOff: check registry credentials
4. If Pending: check GPU quota and availability

GPU Node Failure

1. Pods will be rescheduled automatically
2. If no capacity: scale down non-critical workloads
3. Contact CoreWeave support for extended outages

Model Loading Failure

1. Check HuggingFace token secret exists
2. Verify model name spelling
3. Check PVC has sufficient storage
4. Review container logs for download errors

Rollback

kubectl rollout undo deployment/inference

Resources

• CoreWeave Support
• CoreWeave Status

Next Steps

For data handling, see coreweave-data-handling.