k8s-debug

K8s Debug

Systematically diagnose why a Kubernetes service is unhealthy in the local Tilt environment.

Arguments

• {service} — The Tilt resource name to diagnose (required)

Process

Phase 1: Evidence Collection

1.1 Resource Status

Call tilt_get_resource for the service. Record:

• runtimeStatus and updateStatus
• Pod name, status, restart count
• Build history (last 3 builds — errors, warnings)
• Conditions
• Waiting reason (if any)

1.2 Logs

Call tilt_logs with lines: 200. Search for:

• ERROR, Exception, Fatal, panic
• Connection refused, ECONNREFUSED
• OOMKilled, signal: killed
• migration, database, table
• Stack traces

1.3 Full Description

Call tilt_describe for K8s-level events and pod spec details.

1.4 Code Changes (if applicable)

# Check recent changes to the service source
git log --oneline -10 -- src/{ServiceName}/ 2>/dev/null || true

Phase 2: Triage

Check each common failure mode:

Failure Mode	Indicators	Root Cause
Build error	updateStatus: error	Compilation failure — check build error message
CrashLoopBackOff	High restart count, runtimeStatus: error	Startup crash — check logs for exception
ImagePullBackOff	Pod status contains ImagePull	Image not found — check image name/tag/registry
OOMKilled	Logs show signal: killed or K8s event OOMKilled	Memory limit too low
Dependency down	Logs show Connection refused to postgres/nats/redis	Upstream resource not ready
Migration failure	Logs show migration errors	Database schema issue
Config error	Logs show missing env var or config key	Missing ConfigMap/Secret
Pending	runtimeStatus: pending, no pod	Insufficient cluster resources

Phase 3: Root Cause Analysis

Apply 5-Whys to the identified failure:

1. Why is the service unhealthy? → {immediate cause}
2. Why did that happen? → {deeper cause}
3. Continue until root cause is found

Phase 4: Report

## Diagnosis: {service}

**Status:** runtime={status}, update={status}
**Pod:** {podName} (restarts: {count})

### Root Cause
{One sentence description}

### Evidence
- {Key log line or event}
- {Build error or condition}
- {Recent code change if relevant}

### Fix
{Specific action to take — command, code change, or config update}

### Prevention
{What to add/change to prevent recurrence}

Quick Fixes

For common issues, suggest the fix directly:

• Build error: Show the compiler error and the file/line to fix
• Dependency down: tilt_trigger {dependency} or check its logs
• OOMKilled: Increase memory limit in Helm values or Tiltfile
• Missing env var: Show which var is missing and where to set it