nvidia-ai-operations-day2

NVIDIA AI Operations (Day-2) Review

Purpose

Review operational posture of NVIDIA GPU fleets against the NCP-AIO body of knowledge: DCGM and DCGM-Exporter coverage, MIG partitioning lifecycle, Xid error classification and runbooks, fleet driver/firmware upgrade orchestration, and GPU-aware incident response.

Lean operating rules

• Prefer live evidence (dcgmi diag, dcgmi health, nvidia-smi --query-gpu=..., Prometheus DCGM metrics) when the active client exposes it; otherwise fall back to NVIDIA DCGM documentation, sanitized dashboards, and Xid reference tables.
• Separate confirmed facts from inference. If DCGM coverage, alerting rules, or Xid classification was not directly queried, say so.
• Treat absence of DCGM exporter on production GPU nodes as a high finding — fleet is operationally blind.
• Treat ungated driver/firmware rolling upgrade (no pre-drain, no canary, no rollback path) as a high finding.
• Treat undocumented Xid signature → runbook mapping as a medium finding — incidents will be triaged from scratch.
• Treat GPU-Operator / device-plugin running with default permissive securityContext on multi-tenant clusters as a high finding.
• Treat MIG re-partitioning performed live without workload drain as a medium finding (running contexts are killed).
• Treat alerting that does not cover ECC double-bit errors, NVLink errors, or row-remapping pending as a medium finding.

Response minimum

Return, at minimum:

• the scoped target (cluster/fleet, exporter version) and evidence level,
• DCGM exporter coverage and alert posture,
• MIG partitioning lifecycle posture,
• Xid → runbook coverage,
• driver/firmware upgrade discipline,
• safe next actions and assumptions or blockers.