k8s-monitor

Kubernetes Training Monitor

Heuristics for monitoring training jobs on Kubernetes. This skill provides reference knowledge about common K8s patterns and failure modes — not rules or checklists. Use it to inform your reasoning about what to check in K8s environments.

K8s Evidence Collection

# Pod status
kubectl get pods -n <ns> -l job-name=<name> -o wide

# Recent logs
kubectl logs -n <ns> -l job-name=<name> --tail=20

# Error scan
kubectl logs -n <ns> -l job-name=<name> --tail=200 2>/dev/null | grep -i "error\|exception\|killed\|oom\|timeout\|nan\|inf" | tail -5

K8s Resource Checks

• Check PVC capacity: kubectl exec <pod> -n <ns> -- df -h /mount/path
• Disk >80% on PVC: delete old checkpoints or resize PVC.

Infrastructure Failures

Detect: pod not Running, RESTARTS > 0, pod evicted/preempted, NCCL errors, OOM-killed, node not ready, pod stuck in Pending/ContainerCreating >5 min.

Action:

1. kubectl describe pod <name> -n <ns> | tail -30 for events.
2. Full error from last 200 log lines.
3. Preempted: resubmit immediately (use same YAML, K8s picks new node).
4. OOM-killed: check if GPU OOM or container RAM OOM. Reduce batch_size (GPU) or increase memory request (RAM). If increased memory still cannot be scheduled, fall back to reduced config.
5. NCCL error: verify NCCL env vars, check if node changed (different GPU topology).

Unschedulable / Pending >5 min

The cluster cannot satisfy your resource request. Do NOT wait indefinitely. Diagnose and adapt:

kubectl describe pod <name> -n <ns> | grep -A3 "Events\|FailedScheduling"

Common reasons:

• Insufficient memory: reduce memory request (e.g. 256Gi to 192Gi). Estimate actual need from previous run's peak RSS.
• Insufficient GPU: requested GPU type unavailable. Try a different GPU type. Adjust config (batch_size, model parallel) to fit the new GPU's memory.
• Node taints / affinity: remove nodeSelector or add tolerations.
• PVC region mismatch: PVC is in one region, available GPUs in another.

Scheduling Escalation Ladder

Try in order, stop when it schedules:

1. Reduce memory request to minimum viable (peak RSS from last run + 20% headroom)
2. Switch GPU type (check available: kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu)
3. Drop PVC mount (use HF push for checkpoint persistence instead)
4. Reduce to 1 GPU (if model fits with gradient checkpointing + smaller batch)
5. Alert user: "Cannot schedule on K8s with any viable config. Need manual intervention."

After each adaptation, resubmit and set a 5-minute timer. If still Pending, try next rung on the ladder.