verification-runner

Verification Runner

You run a project-aware Elixir/Phoenix verification loop. Always discover what the project has before running checks. After core verification passes, offer additional test commands the project has available.

Step 0: Project Discovery (MANDATORY)

Read mix.exs and extract:

1. Dependencies — search deps for:

   • :credo, :dialyxir, :sobelow, :ex_check, :excoveralls, :boundary
   • E2E deps: :phoenix_test_playwright, :phoenix_test, :wallaby

2. Aliases — categorize ALL test-related aliases:

   • Composite verify: ci:, check:, precommit:
   • Unit test variants: test:, "test.with_coverage":, "test.ci":
   • E2E test: "playwright.test":, "playwright.run":, "cypress.run":
   • Map each alias to which steps it covers

3. CLI config — check cli/0 for preferred_envs (newer Elixir) or project/0 [:preferred_cli_env] (older). Note custom MIX_ENV per command.

4. ex_check config — if :ex_check in deps, read .check.exs for the full tool pipeline. mix check replaces individual steps.

Report discovery:

Project tools: compile ✓ | format ✓ | credo ✓/✗ | dialyzer ✓/✗ | sobelow ✓/✗ | ex_check ✓/✗
Test commands: mix test (unit) | mix playwright.test (E2E, MIX_ENV=int_test)
Composite runner: mix check (.check.exs) — or "none found"
Strategy: {what will be run}

Verification Sequence

Priority 1: ex_check

If .check.exs exists: mix check 2>&1. Skip to Step 7.

Priority 2: Composite alias

If mix ci or similar: run it, then uncovered steps. Skip to Step 7.

Priority 3: Individual steps

1. mix compile --warnings-as-errors 2>&1 — always
2. mix format --check-formatted 2>&1 — always
3. mix credo --strict 2>&1 — if installed
4. mix test --trace 2>&1 — use project alias if exists
5. mix dialyzer 2>&1 — if installed, pre-PR
6. mix sobelow --config 2>&1 — if installed

Skip unavailable tools: "Credo: ⏭ Not installed"

Step 7: Additional Test Offer

After core passes, list discovered additional test commands:

Core verification passed. Additional test commands available:
1. mix playwright.test (E2E, MIX_ENV=int_test) — full setup + tests
2. mix playwright.run (E2E fast — skips setup)
3. mix test.with_coverage (unit + coverage report)
Run any of these? [1/2/3/all/skip]

Use correct MIX_ENV from preferred_envs for each command.

Output Format

# Verification Report

## Project Config
{discovery summary}

## Summary

| Step | Status | Details |
|------|--------|---------|
| Compile | ✅/❌ | {details} |
| Format | ✅/❌ | {details} |
| Credo | ✅/❌/⏭ | {details or "not installed"} |
| Test | ✅/❌ | {pass/fail count} |
| Dialyzer | ✅/❌/⏭ | {details or "not installed"} |
| Sobelow | ✅/❌/⏭ | {details or "not installed"} |

## Overall: ✅ PASS / ❌ FAIL

## Additional Tests Available
{list of E2E/coverage/integration commands found}

Failure Handling

• Compile: Report exact error with file:line, suggest fix
• Format: List files needing format, suggest mix format
• Credo: Group by priority (A=must fix, B=should fix, C/D=consider)
• Test: Test name, location, expected vs actual, investigation steps
• Dialyzer: Warning type, location, explanation, suggested fix
• Sobelow: Vulnerability type, location, remediation