grounded-ai-guardrails

Cross-Cutting — AI / LLM Guardrails & Evaluation

Run during Specify (define AI requirements and acceptable behavior) and Implement (build guardrails and evals). Record in ai-guardrails.md. AI features need rigor that traditional checklists miss because they are non-deterministic and can fail confidently.

Evaluation

• Build a scored eval set, not just pass/fail asserts — accuracy, safety, format, refusal.
• Eval set covers the Phase 2 edge cases and known failure patterns.
• Set a quality bar the client approves as the AI definition of "done."
• Re-run evals on every prompt or model change to catch regressions.

Hallucination & grounding

• Ground outputs in retrieval or provided context; cite sources where claims are made.
• Define correct refusal behavior — when the model should say "I don't know."
• Constrain outputs to schemas or allow-lists where correctness matters.

Prompts & versioning

• Treat prompts and context templates as governed, versioned artifacts.
• Log prompt version alongside outputs for traceability.

Guardrails & control

• Filter inputs and outputs (injection, unsafe content, PII leakage).
• Human-in-the-loop approval for high-risk or irreversible actions.
• Bound agent tool access to least privilege.

Model & data risk

• Pin model versions; plan for deprecation and a fallback model.
• Monitor token cost, latency, and output quality in production.
• Control what data enters prompts, logs, and vendor retention (link to grounded-security-review).