api-testing

API Testing Layers

Four layers (Layer 4 is conditional on the framework) plus a custom agent-friendliness suite. All applicable layers must pass before merging — no exceptions for "just this once".

Layer 1 — Spec drift detection

Validates the running application matches the OpenAPI spec: paths, operationIds, response schemas, x-agent-* extensions on every POST, format: date-time on timestamp fields. Catches drift between spec and code without needing a running server.

Run: pytest tests/test_openapi.py -v

Layer 2 — Schemathesis fuzzing

Generates valid (and edge-case) requests from the spec and validates responses conform to the spec. Finds edge cases hand-written tests miss. Catches empty-state envelope bugs, missing Retry-After on 429, 409 without conflicting_id.

Run: pytest tests/test_schemathesis.py -v Requires: running server at http://localhost:8000.

Layer 3 — Backward compatibility

oasdiff comparing the spec on the PR branch against the spec on main. Fails the PR on any breaking change in a stable (v1+) API.

Run in CI:

oasdiff breaking \
  <(git show origin/main:api/myservice/v1.yaml) \
  api/myservice/v1.yaml

Layer 4 — Generated spec compatibility (conditional)

For frameworks that auto-generate an OpenAPI spec at runtime (FastAPI, Flask-RESTX, Django REST Framework, etc.): run oasdiff comparing the runtime-generated spec against the canonical api/{service}/v1.yaml. Catches divergence introduced by code changes that update the generated spec without updating the canonical file.

Run: pytest tests/test_generated_spec.py -v

Skip Layer 4 only if the framework does not generate a spec at runtime. Do not skip it because "FastAPI generates the spec from the code anyway" — the canonical file is still the source of truth.

Custom tests — agent-friendliness

tests/test_agent_friendliness.py — not covered by schemathesis. Cover behaviours that require stateful or authenticated interaction:

• Idempotency deduplication — send the same request with the same Idempotency-Key twice; expect identical response, not a duplicate resource
• Token error codes — send an expired token; expect code: TOKEN_EXPIRED. Send a malformed token; expect code: TOKEN_INVALID.
• Pagination accuracy — fetch all pages; sum items lengths; assert equals total from first response
• 429 Retry-After header — trigger rate limit; assert Retry-After header present with integer value

Which layer catches what

Bug category	Layer 1	Layer 2	Layer 3	Layer 4	Custom
Endpoint missing from code	✓
Missing operationId	✓
Missing x-agent-* extensions on POST	✓
Timestamp fields missing format: date-time	✓
Optional response fields absent (null/absent convention)		✓
Response doesn't match schema	✓	✓
Empty list returns null or bare array	✓	✓
Edge case input handling		✓
Undocumented error responses		✓
Retry-After header absent on 429					✓
409 missing conflicting_id		✓
Breaking spec change vs main			✓
Code change breaks generated spec vs canonical				✓
Framework annotation drift				✓
Idempotency deduplication actually works					✓
TOKEN_EXPIRED vs TOKEN_INVALID distinction					✓
Pagination total matches actual page sum					✓

Not auto-testable — requires human review:

• x-agent-retryable annotation matches actual endpoint behaviour (you can test behaviour; you cannot prove the annotation describes it correctly)
• Token TTL documented accurately
• docs/agent-integration.md intent table is complete and accurate

Known schemathesis exclusions — document in docstrings, never skip silently

• /health from not_a_server_error — correctly returns 503 when dependencies are down
• Auth-exempt endpoints from missing_required_header
• Endpoints with custom validation from positive_data_acceptance

Any other exclusion must have a docstring explaining why. No bare @pytest.mark.skip. No # noqa without a reason.

Apply TDD when writing test files

When writing a new test file:

1. Write the failing test first
2. Run it — watch it fail with a specific error
3. Implement the endpoint (or fix the spec) to make the test pass
4. Never write the implementation and add the test retroactively

Anti-patterns

• Skipping schemathesis because it's slow
• Silently disabling checks with # noqa or @pytest.mark.skip without a docstring
• "Tests can wait until after merge"
• "I already manually tested every endpoint"
• Skipping Layer 4 on FastAPI services because "FastAPI generates the spec from the code anyway"
• Skipping custom agent-friendliness tests because they require a running server

Red flags — STOP

• "Layer 1 is enough, the others are overkill"
• "We can add tests in the next sprint"
• "Manual testing covered the cases"
• "It works, ship it"
• "Schemathesis is slow locally, I'll skip it and let CI run it"

References

• references/python-testing.md — full example test files (test_openapi.py, test_schemathesis.py, test_generated_spec.py, test_agent_friendliness.py), pyproject.toml schemathesis config, GitHub Actions CI for all layers
• SOFT REFERENCE: api-first:api-reviewing — testing coverage checklist that reviewers use to verify all layers are present