test-completeness

Test Completeness

Audit test completeness across two phases: dynamic (run tests, collect real coverage) and static (subagent analyzes scenarios across 6 dimensions). Then propose and generate missing tests interactively.

Overview

Phase 1 — Dynamic: Run the test suite and collect live coverage data. This gives real line/branch percentages and identifies which code paths are actually exercised.

Phase 2 — Static: Dispatch a Scenario Auditor subagent that independently reads source files, test files, and coverage data to audit across 6 dimensions: good case, bad case, boundary cases, unit tests, integration tests, and e2e tests.

Results from both phases are merged into a single report. In interactive mode, you then propose remediation, generate missing tests, and re-run in a verification loop (max 5 rounds). In non-interactive mode, output the verdict and return.

When to Use

• After implementing a feature or fix, before merging
• When a code review surfaces test coverage concerns
• When running review-loop reveals missing tests
• When a PR touches files with no or low test coverage
• Periodically on a codebase to establish baseline test health

Don't use for:

• Checking whether a single assertion is correct — read the test manually
• Auditing documentation or non-code files
• Projects with no test infrastructure at all (ask user to set up tests first)

Process Flow

digraph test_completeness {
    "Detect project\ntest infrastructure" [shape=box];
    "Determine target files\n(diff or full)" [shape=box];
    "Run test suite\n+ collect coverage" [shape=box];
    "Tests pass?" [shape=diamond];
    "STOP: Report failure\nask user to fix" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];
    "Dispatch Scenario\nAuditor subagent" [shape=box];
    "Merge coverage +\nauditor results" [shape=box];
    "Present summary\nreport to user" [shape=box];
    "Any gaps found?" [shape=diamond];
    "PASS" [shape=doublecircle];
    "Interactive mode?" [shape=diamond];
    "Output verdict\nand return" [shape=box];
    "Propose remediation\noptions to user" [shape=box];
    "User selects\ngaps to fix" [shape=box];
    "Generate missing\ntest files" [shape=box];
    "Re-run tests\n+ re-dispatch auditor" [shape=box];
    "Round <= 5?" [shape=diamond];
    "STOP: Pause,\nask user" [shape=octagon, style=filled, fillcolor=orange];

    "Detect project\ntest infrastructure" -> "Determine target files\n(diff or full)";
    "Determine target files\n(diff or full)" -> "Run test suite\n+ collect coverage";
    "Run test suite\n+ collect coverage" -> "Tests pass?";
    "Tests pass?" -> "STOP: Report failure\nask user to fix" [label="no"];
    "Tests pass?" -> "Dispatch Scenario\nAuditor subagent" [label="yes"];
    "Dispatch Scenario\nAuditor subagent" -> "Merge coverage +\nauditor results";
    "Merge coverage +\nauditor results" -> "Present summary\nreport to user";
    "Present summary\nreport to user" -> "Any gaps found?";
    "Any gaps found?" -> "PASS" [label="no"];
    "Any gaps found?" -> "Interactive mode?" [label="yes"];
    "Interactive mode?" -> "Output verdict\nand return" [label="no"];
    "Output verdict\nand return" -> "PASS";
    "Interactive mode?" -> "Propose remediation\noptions to user" [label="yes"];
    "Propose remediation\noptions to user" -> "User selects\ngaps to fix";
    "User selects\ngaps to fix" -> "Generate missing\ntest files";
    "Generate missing\ntest files" -> "Re-run tests\n+ re-dispatch auditor";
    "Re-run tests\n+ re-dispatch auditor" -> "Round <= 5?";
    "Round <= 5?" -> "Propose remediation\noptions to user" [label="yes, gaps remain"];
    "Round <= 5?" -> "PASS" [label="yes, no gaps"];
    "Round <= 5?" -> "STOP: Pause,\nask user" [label="no"];
}

Parameters

All optional — infer from context:

Parameter	Default	Description
scope	diff	diff = only files changed in current branch vs base; full = entire codebase
base	main	Base branch for diff comparison (e.g. main, master, develop)

Scope inference:

1. Explicit scope in conversation → use it
2. On a feature branch with unmerged commits → use diff mode
3. User asks for full audit or there's no clear diff context → use full mode
4. Ambiguous → ask user

Step 1: Detect Project Test Infrastructure

Before running anything, detect what tooling the project uses:

Detection Target	What to Detect	Method
Language	Primary language(s)	File extensions, package.json, Cargo.toml, go.mod, pyproject.toml, etc.
Test framework	Jest, pytest, Go test, Cargo test, Mocha, Vitest, etc.	Config files, devDependencies, [dev-dependencies], test script in package.json
Test runner command	npm test, pytest, go test ./..., cargo test, etc.	scripts.test in package.json, Makefile, CI config, pyproject.toml
Coverage tool	Istanbul/nyc, coverage.py, go test -cover, cargo tarpaulin, c8, etc.	Coverage config, existing CI scripts
Test directories	__tests__/, tests/, test/, spec/, *.test.ts, *_test.go, etc.	Directory scan + file pattern matching
Test layers	Which of unit/integration/e2e exist	Directory names, test file naming conventions, CI config

Fallback for no framework detected: Ask the user: "I couldn't detect a test framework. What test runner does this project use?" Do not proceed until confirmed.

Fallback for no coverage tool: Run tests without coverage. Note in the report that coverage data is unavailable. Static audit via Scenario Auditor still runs.

Output of Step 1: A project context block used to fill the auditor prompt:

Language: TypeScript
Framework: Jest
Runner: npm test
Coverage: Istanbul (nyc) via jest --coverage
Test dirs: src/**/__tests__/, src/**/*.test.ts
Layers detected: unit, integration

Step 2: Dynamic Data Collection

2a: Determine Target Files

Diff mode (scope=diff):

git diff --name-only $(git merge-base HEAD <base>) HEAD \
  | grep -v test | grep -v spec | grep -E '\.(ts|js|py|go|rs)$'

Replace <base> with the resolved base parameter (default: main). Example with default: $(git merge-base HEAD main). Filter out: test files, spec files, generated files, config files, type-only files.

Full mode (scope=full): Collect all source files matching the language extension, excluding test files and vendor directories.

If no target files after filtering: Report "No source files in scope" and exit with PASS. (See Edge Cases.)

2b: Run Test Suite

Run the detected test command with coverage enabled. Examples:

Stack	Command
Node/Jest	npx jest --coverage --coverageReporters=json-summary --coverageReporters=text
Node/Vitest	npx vitest run --coverage
Python/pytest	pytest --cov=. --cov-report=json --cov-report=term-missing
Go	go test ./... -coverprofile=coverage.out && go tool cover -func=coverage.out
Rust	cargo tarpaulin --out Json

If tests fail: Stop immediately. Report:

## Test Completeness — BLOCKED

Tests are currently failing. Fix the failing tests before running a completeness audit.

Failing tests:
- [paste test output summary]

Re-run /test-completeness after tests pass.

Do not proceed to Step 3.

2c: Parse Coverage Data

Extract per-file coverage percentages. Minimum to capture:

• Line coverage %
• Branch coverage % (if available)
• Uncovered line ranges (e.g., L10-L15, L30)

Store as structured data keyed by file path for use in the auditor prompt.

Step 3: Dispatch Scenario Auditor

Fill the auditor-prompt.md template with the collected data and dispatch via Agent tool. Read the auditor-prompt.md template from this skill's directory (plugins/winrey-toolkit/skills/test-completeness/auditor-prompt.md).

Placeholder mapping:

Placeholder	Data source
{SCOPE}	diff or full + description (e.g. "diff — files changed vs main")
{DESCRIPTION}	Git log summary for diff scope; "Full codebase audit" for full scope
{PROJECT_CONTEXT}	Output of Step 1 detection block
{TARGET_FILES}	List of target source files with their paths and brief descriptions
{TEST_FILES}	List of existing test files, grouped by which source file they test
{COVERAGE_DATA}	Per-file coverage percentages and uncovered line ranges from Step 2c

You MUST dispatch a subagent. Do NOT analyze the code yourself. The controller's role is to coordinate and synthesize results. Even for a single file, dispatch the Scenario Auditor — it reads code with fresh eyes and no anchoring bias. If you catch yourself thinking "I'll just check it quickly" — that is exactly the rationalization this rule prevents.

Dispatch as a single Agent call with the filled prompt. The auditor runs to completion and returns structured results per file.

Step 4: Merge Results and Present Summary Report

Merge coverage data (Step 2) with auditor findings (Step 3) into a unified report:

## Test Completeness Report — Round N

**Scope:** diff (12 files) / full (47 files)
**Coverage:** 78% line, 65% branch (project average)

### Per-File Summary

| File | Line% | Branch% | Good | Bad | Boundary | Unit | Integration | E2E | Issues |
|------|-------|---------|------|-----|----------|------|-------------|-----|--------|
| src/auth.ts | 92% | 80% | PASS | FAIL | FAIL | PASS | PASS | N/A | 3 |
| src/utils.ts | 100% | 100% | PASS | PASS | PASS | PASS | N/A | N/A | 0 |
| src/api/handler.ts | 45% | 30% | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | 8 |

### Issue Summary

| # | File | Severity | Dimension | Description |
|---|------|----------|-----------|-------------|
| 1 | src/auth.ts | Important | Bad case | validateToken() error path not tested |
| 2 | src/auth.ts | Important | Boundary | null input to parseJWT() missing |
| 3 | src/api/handler.ts | Critical | Good case | handleRequest() has zero tests |

### Overall Verdict

- Critical issues: X
- Important issues: Y
- Minor issues: Z
- Status: FAIL — X Critical, Y Important issues found

Non-interactive mode: Output this report and return. Do not proceed to Step 5. Add at the end:

[Non-interactive mode] Re-run with interactive mode to generate missing tests.

Step 5: Remediation Proposal

Present gaps grouped by file, and offer the user choices:

## Remediation Options

I found X gaps across Y files. What would you like to do?

  a) Fix ALL gaps — generate tests for every Critical and Important issue
  b) Fix Critical only — generate tests for every Critical issue
  c) Fix by file — I'll list files; pick which ones to address
  d) Fix by dimension — choose which dimensions to fix (e.g., "fix all bad case gaps")
  e) Skip — exit without generating tests (issues remain on record)

Your choice (a/b/c/d/e):

Wait for user input before proceeding. After user selects:

1. Group selected gaps by file and dimension
2. Generate test code for each selected gap (see guidelines below)
3. Show generated tests to user for confirmation before writing:

   I'll add the following tests to src/__tests__/auth.test.ts:
   [show test code]
   Confirm? (yes/edit/skip):
   

4. Write confirmed tests to the appropriate test files

Test generation guidelines:

• Match the existing test framework, style, and patterns exactly — read existing tests first
• Place tests in the location matching project conventions (same directory, __tests__/ subfolder, etc.)
• For bad case tests: use realistic inputs that trigger the actual error path
• For boundary tests: test one boundary per test case — don't combine all boundaries in one test
• For integration tests: use existing test helpers/fixtures — don't invent new infrastructure
• Keep test descriptions specific: "returns null when input is empty string" not "handles edge case"
• Never write tests that simply call a function and check it doesn't throw — test actual outcomes
• If a test requires infrastructure that doesn't exist (e.g., test database, mock server), note this as a prerequisite and ask the user before writing

Step 6: Verification Loop

After generating tests, re-run the full pipeline (Steps 2-4) to verify the gaps are closed.

Round counter: The round counter starts at 1 for the first verification run after tests are generated. The initial audit performed in Steps 2-4 is NOT counted as a round. The counter increments by 1 after each verification-fix cycle.

Loop structure:

• Re-run tests + collect new coverage
• Re-dispatch Scenario Auditor with updated file list and coverage
• Merge and present new report, highlighting what changed
• Check the round counter after re-running tests and before proposing remediation — if the counter has reached 5 and gaps remain, stop and present the max-rounds summary instead of proposing another remediation cycle

Loop summary format:

## Verification Round N

Tests run: PASS (X passing, 0 failing)
Coverage change: 78% → 85% line, 65% → 72% branch

Gaps closed: 5
Gaps remaining: 2

Remaining issues:
| # | File | Severity | Dimension | Description |
|---|------|----------|-----------|-------------|
| 1 | src/api/handler.ts | Important | Integration | DB interaction still untested |
| 2 | src/api/handler.ts | Minor | E2E | No e2e test for POST /handler |

Max 5 rounds. After round 5 with remaining gaps, stop and present:

## Verification — Max Rounds Reached

After 5 rounds, the following gaps remain unresolved:
[list remaining issues]

Possible reasons:
- Tests require infrastructure not yet set up
- Gaps require architectural changes beyond test generation
- Issues are in autogenerated or vendored code

Recommendation: Review the remaining gaps manually or adjust the scope.

Severity Definitions

Severity	Definition	Examples
Critical	Function or module has zero tests; file with business logic at 0% coverage	Public function with no test file; entire module untouched by any test
Important	Missing bad case or boundary tests for existing functions; missing a full test layer that applies; coverage gaps on non-trivial logic	validateInput() has no error path test; service with DB calls has no integration tests; 40% branch coverage on branching logic
Minor	Tests exist but scenarios not comprehensive enough; test organization could improve	Happy path test exists but only tests one input value; test descriptions are generic

Edge Cases

Scenario	Behavior
No test framework detected	Prompt user to identify framework. Do not proceed until confirmed.
No coverage tool available	Run tests without coverage. Flag "Coverage data unavailable" in report. Static audit still runs.
Tests fail on initial run	Stop immediately. Report failing tests. Do not dispatch auditor. Ask user to fix tests first.
No changed files in diff scope	Report "No source files in scope for this diff." Exit with PASS — nothing to audit.
File marked N/A for a dimension	Auditor sets that dimension to N/A. Do not count N/A as a gap. Do not propose tests for N/A dimensions.
Missing test infrastructure (e.g., no test DB)	Note as prerequisite in remediation. Ask user before writing tests that depend on it.

Common Mistakes

Mistake	Fix
Analyzing code yourself instead of dispatching auditor	Always dispatch the Scenario Auditor subagent — do not review code directly
Proceeding when tests are failing	Stop at Step 2b and ask user to fix tests first
Counting N/A dimensions as failures	N/A means not applicable — no test needed, not a gap
Generating tests without reading existing test patterns	Always read existing test files before generating new ones
Writing all boundary cases in a single test	One test per boundary value — keeps failures isolated and readable
Proposing tests before user confirms	Show generated test code and wait for user confirmation before writing
Running more than 5 verification rounds	After round 5, stop and present remaining gaps with explanation
Using diff scope when no branch context exists	Fall back to full scope or ask user when diff is ambiguous

Integration

Skill	Relationship
review-loop	Run test-completeness when review-loop surfaces missing tests — fixes the gap before re-review
verification-before-completion	Complement — verification-before-completion checks work is done; test-completeness checks tests are complete. Run both before merging.
feature-dev	Run test-completeness as the final step of feature development, after implementation is complete
writing-plans	Include a test-completeness step at the end of implementation plans that involve new or modified code

Non-interactive mode note: When invoked from another skill (e.g., as a step inside feature-dev or writing-plans), run in non-interactive mode. Output the verdict report and return — do not prompt the user for remediation choices. The calling skill decides what to do with the results.