testing-expert

Software Testing Expert

Test behavior, not implementation — that's what survives refactors. Fast and deterministic beats exhaustive. A bug fixed without a failing test first will come back.

When to Use

• Writing unit/integration/e2e tests for new or existing code.
• Designing a test strategy or adopting TDD.
• Flaky, slow, or brittle tests that break on harmless changes.
• Mocking/stubbing, fixtures, fakes, and coverage decisions.

When NOT to Use

• LLM/model evaluation specifically → llm-testing-expert.
• Diagnosing a specific known failure → debugging-expert.
• Performance benchmarking → performance-expert.

Core Principles

1. Test the contract, not the internals

• Assert on observable behavior and public outputs/effects, not private methods or call counts. Tests coupled to implementation block refactoring and create false failures.
• Follow the test pyramid: many fast unit tests, fewer integration tests, a thin layer of e2e for critical user journeys. Push detail down to the cheapest level that can catch the bug.
• Cover edge cases and error paths, not just the happy path. Boundary values, empties, nulls, concurrency, and failures are where bugs live.

2. Good test shape

• Arrange–Act–Assert, one behavior per test, names that state the expectation (returns 401 when token expired).
• Tests must be deterministic and independent: no shared mutable state, no order dependence, no real clock/network/randomness leaking in.

3. Test doubles, sparingly

• Mock/stub/fake only at boundaries you don't own (network, DB, clock, filesystem, randomness). Never mock the unit under test.
• Inject those dependencies (time, IDs, I/O) so tests control them. Prefer fakes (in-memory impl) over brittle mock-verification chains. Over-mocking tests the mocks, not the code.

4. Reliability & coverage

• Eliminate flakiness at the source: fake time, await real conditions instead of sleep, control async ordering, isolate test data. Quarantine and fix flaky tests — don't retry-until-green.
• Coverage finds untested branches; it's a flashlight, not a goal. 100% coverage of trivial code ≠ quality; a critical untested branch at 90% is the real risk.

TDD loop

Red (write a failing test for the next behavior) → Green (simplest code to pass) → Refactor (clean up with the test as safety net). Especially valuable for bug fixes: reproduce as a failing test first.

Common Mistakes

• Asserting implementation details (private calls, exact internal state) → brittle; assert outcomes.
• sleep() to wait for async → flaky and slow; await a condition/event.
• Over-mocking → tests pass while real integration breaks; use fakes and some integration tests.
• Interdependent / order-dependent tests → spooky failures; isolate state.
• Testing only the happy path → bugs hide in errors/edges.
• Snapshot tests for everything → giant snapshots nobody reviews; snapshot small, stable output only.
• Chasing a coverage number → tests written to touch lines, not to assert behavior.

Examples

Behavior test with a controlled boundary (pytest)

def test_token_expires(monkeypatch):
    clock = {"now": 1000}
    monkeypatch.setattr(auth, "now", lambda: clock["now"])  # fake the clock

    token = auth.issue(ttl=60)        # valid until 1060
    assert auth.is_valid(token)

    clock["now"] = 1061               # advance time deterministically
    assert not auth.is_valid(token)   # asserts behavior, not internals

Bug → failing test first

def test_discount_never_negative():   # reproduces the reported bug
    assert price(items=[], coupon="HUGE") == 0   # was returning -5

See Also

• debugging-expert — turn a bug into a failing test, then fix.
• refactoring-expert — tests are the safety net for restructuring.
• github-master — run tests + coverage gates in CI.
• llm-testing-expert — evaluation for LLM systems.