nvidia-ai-networking-fabric-review

NVIDIA AI Networking Fabric Review

Purpose

Review NVIDIA AI fabric configuration against the NCP-AIN body of knowledge: rail-optimized topology, NCCL collective communication tuning (NCCL_TOPO, NCCL_IB_HCA, NCCL_NET_GDR_LEVEL), RoCEv2 lossless DCQCN/PFC, InfiniBand subnet manager and partitioning, adaptive routing, and tenant/job east-west isolation.

Lean operating rules

• Prefer live evidence (ibstat, ibdiagnet, nccl-tests all_reduce_perf baselines, ethtool -S, switch QoS counters) when the active client exposes it; otherwise fall back to NVIDIA Spectrum-X / Quantum InfiniBand documentation and sanitized topology diagrams.
• Separate confirmed facts from inference. If NCCL transport selection, PFC posture, or partition keys were not directly queried, say so.
• Treat a non-rail-optimized topology presented as rail-optimized as a critical finding — collective performance claims are wrong.
• Treat RoCEv2 deployed without PFC + ECN (DCQCN) as a high finding — lossless behavior is not provided; goodput collapses under congestion.
• Treat default NCCL settings on multi-rail systems as a medium finding when published baselines are missing — transport selection is unverified.
• Treat shared partition keys (PKey 0x7FFF / default) across tenants on a multi-tenant InfiniBand fabric as a high finding — east-west isolation is absent.
• Treat absence of nccl-tests baselines stored alongside the cluster spec as a medium finding — regressions cannot be detected.
• Treat subnet manager running on a single switch with no failover as a high finding for production fabrics.

Response minimum

Return, at minimum:

• the scoped target (fabric type, generation, scale) and evidence level,
• topology and rail posture,
• NCCL transport and tuning posture,
• lossless/PFC/ECN posture (RoCEv2) or PKey/partition posture (InfiniBand),
• tenant isolation posture,
• safe next actions and assumptions or blockers.