spp — Supervised Prompt Producer
A Claude Code plugin for disciplined, human-in-the-loop supervised prompt
learning: it turns a labeled baseline into a production-grade prompt you can
defend in code review with evidence, not vibes.
spp handles classification (binary, multi-class, fixed-schema),
multi-field and hierarchical structured output, structured extraction
(variable-cardinality, span-grounded), and prompt decomposition (a managed
linear pipeline) — anything with a labeled ground truth and a mechanical metric.
It is deliberately not a generation, RAG, agentic, or prompt-search tool (see
DESIGN.md §7.1.3). For what shipped, see
CHANGELOG.md; for the roadmap and scope, DESIGN.md §7.1.
Why
Prompt engineering by feel produces prompts that look good and fail in
production — tuned against whatever rows the author remembered, shipping with
failures clustered where no one looked. Automated optimizers (DSPy, APE) go the
other way: they trust a metric and search, which works only if the metric is
honest — and a metric computed against one model on one labeled set rewards
learning the dataset's quirks and the model's style, both of which look like
generalization until you swap a model or a data slice.
spp targets the two failure modes those approaches miss:
- Baseline overfitting — the prompt fits your specific labels, not the class
definition. Scores high on what you tuned against, collapses on
similar-but-unseen data. A deal-breaker.
spp defends against it.
- Model overfitting — the prompt fits one model's instruction-following
style. Fine if you know it and ship accordingly; dangerous if it ships
unmarked.
spp documents and surfaces it.
The example that motivates both
The methodology comes from a hair-loss-discourse classifier that produced a
Qwen-locked prompt at test F1 = 0.941. Run cross-family, it split:
| Model | F1 |
|---|
| Qwen3-14B (optimized target) | 0.941 |
| GPT-4o full | ≈ 0.91 |
| GPT-4o-mini | ≈ 0.76 |
The failures were not random — they clustered, and were length-correlated rather
than purely capability-related: the prompt encoded a Qwen-specific length
tolerance the GPT family did not share. That is model overfitting, caught and
documented rather than shipped unmarked. Baseline overfitting is what a
less-disciplined run on the same labels would have produced — caught here by the
stratified split and the auditor.
How it works
Phase 1 Label baseline + adversarial label review
Phase 1.5 Stratified split (train / dev / sacred test)
Phase 2 Optimization loop:
propose edit from discrepancy analysis
→ AUDITOR: categorical or row-specific?
→ run on dev → overfitting guard
→ stop when dev plateaus or regresses
Phase 3 Final test on the sacred set, exactly once
→ REPORT.md + frozen prompt + documented limitations
Two properties are non-negotiable and are what separate spp from an automated
optimizer:
- Per-stage information isolation. Each cognitive stage of an iteration runs
in an isolated sub-agent with an explicit input allow-list: a discrepancy
stage that reads disagreed rows and abstracts them into clusters by ID; a
rule-edit stage that proposes the next prompt without ever seeing row
content; an auditor that reviews the edits but never sees the new scores.
State flows through files, not a shared context. The auditor's one question —
is each edit categorical (a class of rows with an articulable property) or
row-specific (a patch for one weird row)? — keeps the loop from fitting
rows it never saw.
- The sacred test set. Read exactly once, at finalization. The loop sees
train + dev only.
The pipeline is agentic, but the decisions that reshape it are human — the
kickoff that configures the run, a mid-loop redesign when an agent flags a
structural problem, and any change to the schema, ground truth, or model.
The four phases map to skills/run/phases/spp-{init,baseline,loop,finalize}.md;
the loop's internals are in spp-loop.md §4.
Automated vs. human
| Automated | You stay in the loop |
|---|
| Stratified split generation | Metric design |
| Running iterations against dev | Baseline labeling judgment |
| Discrepancy analysis | Decision criteria for ambiguous rows |
| Categorical-vs-row-specific auditing | Model selection |
REPORT.md generation | Whether an edit is generalized or reverted |
| Sacred-test-set protection | Production ship / no-ship |
When to use it
A good fit when most of these hold (match ~three of five and it's worth trying):
