From the-bulwark
Transforms mock-based test assertions into real output verification patterns. Converts T1-T4 violating tests to verify observable behavior.
How this skill is triggered — by the user, by Claude, or both
Slash command
/the-bulwark:assertion-patternsThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Transform T1-T4 violating assertions into real behavior verification. This skill provides
Transform T1-T4 violating assertions into real behavior verification. This skill provides pattern libraries for converting mock-based tests to tests that verify observable output.
Load this skill when:
Before checking T1-T4 violations, verify these prerequisites. Tests failing these checks are "testing nothing real" - they pass but provide zero confidence.
Test files MUST import functions from actual production modules.
| Valid | Invalid |
|---|---|
import { calculate } from '../src/calculator' | Function defined within test file |
from calculator import add | def add(a, b): return a + b in test |
Detection:
Violation Response:
"Test defines production logic inline. Move
{function_name}to production module and import it. Tests should verify production code, not self-defined code."
Test files MUST NOT contain functions representing production logic.
Allowed in test files:
test_*, it(), describe())make_*, create_*, setup_*)mock_*, fake_*, stub_*)@pytest.fixture)Not allowed:
src/Detection:
Functions in test files should follow specific naming patterns.
| Legitimate Prefixes | Purpose |
|---|---|
test_* | Test functions (pytest) |
_* (underscore) | Private helpers |
pytest_* | Pytest hooks |
make_*, create_* | Factory helpers |
mock_*, fake_*, stub_* | Mock factories |
setup_*, teardown_* | Lifecycle helpers |
Any other function definition raises suspicion - likely production code incorrectly placed in test files.
| Anti-Pattern (Mock) | Real Pattern |
|---|---|
expect(fn).toHaveBeenCalled() | const result = fn(); expect(result).toBe(expected) |
expect(fn).toHaveBeenCalledWith(arg) | const result = fn(arg); expect(result.field).toBeDefined() |
jest.spyOn(module, 'fn').mockReturnValue(x) | const result = module.fn(); expect(result).toBe(expected) |
Transformation example:
// BEFORE (T1 violation - mocking system under test)
jest.spyOn(calculator, 'add').mockReturnValue(5);
expect(calculator.add(2, 3)).toBe(5);
// AFTER (real verification)
expect(calculator.add(2, 3)).toBe(5); // Actually runs add()
| Anti-Pattern (Mock) | Real Pattern |
|---|---|
jest.spyOn(cp, 'spawn').mockReturnValue(mockProc) | const proc = spawn(...); await waitForReady(proc) |
expect(spawn).toHaveBeenCalled() | expect(await isPortOpen(PORT)).toBe(true) |
expect(spawn).toHaveBeenCalledWith(cmd, args) | const output = execSync(cmd); expect(output).toContain(expected) |
Transformation example:
// BEFORE (T1 violation - mocking spawn)
const mockProcess = { on: jest.fn(), stdout: mockStream };
jest.spyOn(child_process, 'spawn').mockReturnValue(mockProcess);
await startProxy();
expect(child_process.spawn).toHaveBeenCalledWith('proxy', ['--port', '8080']);
// AFTER (real verification)
await startProxy();
expect(await checkPort(8080)).toBe(true); // Port actually open
const response = await fetch('http://localhost:8080/health');
expect(response.status).toBe(200); // Proxy actually responds
| Anti-Pattern (Mock) | Real Pattern |
|---|---|
jest.mock('fs') | Use real fs with temp directory |
expect(fs.writeFile).toHaveBeenCalled() | expect(fs.existsSync(path)).toBe(true) |
expect(fs.readFile).toHaveBeenCalledWith(path) | const content = fs.readFileSync(path); expect(content).toContain('expected') |
Transformation example:
// BEFORE (T2 violation - call-only assertion)
await saveConfig(config);
expect(fs.writeFile).toHaveBeenCalled();
// AFTER (result verification)
await saveConfig(config);
expect(fs.existsSync(configPath)).toBe(true);
const saved = JSON.parse(fs.readFileSync(configPath, 'utf8'));
expect(saved.setting).toBe(config.setting);
| Anti-Pattern (Mock) | Real Pattern |
|---|---|
jest.mock('node-fetch') | Use MSW or test server |
expect(fetch).toHaveBeenCalledWith(url) | const resp = await fetch(url); expect(resp.status).toBe(200) |
| Mock response data | Actual response from test server |
Transformation example:
// BEFORE (T3 violation - mocking integration boundary)
jest.mock('node-fetch');
fetch.mockResolvedValue({ json: () => ({ id: 1 }) });
const user = await fetchUser(1);
expect(fetch).toHaveBeenCalledWith('/api/users/1');
// AFTER (real integration with MSW)
import { setupServer } from 'msw/node';
import { rest } from 'msw';
const server = setupServer(
rest.get('/api/users/:id', (req, res, ctx) => {
return res(ctx.json({ id: req.params.id, name: 'Test User' }));
})
);
beforeAll(() => server.listen());
afterAll(() => server.close());
const user = await fetchUser(1); // Real fetch, intercepted at network level
expect(user.name).toBe('Test User');
| Anti-Pattern (Mock) | Real Pattern |
|---|---|
expect(db.save).toHaveBeenCalled() | await db.save(data); const found = await db.find(id); expect(found).toBeDefined() |
expect(db.delete).toHaveBeenCalledWith(id) | await db.delete(id); expect(await db.find(id)).toBeNull() |
jest.mock('./database') | Use test database instance |
Transformation example:
// BEFORE (T2 violation - call-only)
await saveUser(user);
expect(db.insert).toHaveBeenCalledWith('users', user);
// AFTER (result verification)
await saveUser(user);
const saved = await db.findOne('users', { id: user.id });
expect(saved).toBeDefined();
expect(saved.email).toBe(user.email);
The system under test should NEVER be mocked. Remove the mock and verify actual behavior.
// T1 violation: Mocking the function being tested
jest.spyOn(calculator, 'add').mockReturnValue(5);
expect(calculator.add(2, 3)).toBe(5); // Always passes regardless of implementation
// Fixed: Test real implementation
expect(calculator.add(2, 3)).toBe(5); // Fails if add() is broken
Verifying a function was called is insufficient. Verify the RESULT of that call.
// T2 violation: Only checks call happened
await saveConfig(config);
expect(db.save).toHaveBeenCalled(); // Passes even if wrong data saved
// Fixed: Verify the actual saved data
await saveConfig(config);
const saved = await db.find(config.id);
expect(saved.value).toBe(config.value); // Fails if wrong data saved
Integration tests should test real integration. Use MSW, test servers, or in-memory DBs.
// T3 violation: Mocking HTTP in integration test
jest.mock('node-fetch');
const data = await fetchUserData(id); // Not testing real HTTP
// Fixed: Use MSW to intercept at network level
const server = setupServer(rest.get('/api/user/:id', handler));
const data = await fetchUserData(id); // Real fetch, real HTTP, controlled response
Integration tests should chain real outputs through the pipeline.
// T3+ violation: Using mock data instead of real output
const mockOrder = { id: 1, items: [{ sku: 'ABC', qty: 2 }] };
await processOrder(mockOrder); // Not testing real order creation
// Fixed: Chain real function outputs
const order = await createOrder({ sku: 'ABC', qty: 2 }); // Real order
await processOrder(order); // Processes real order data
const result = await getOrderStatus(order.id);
expect(result.status).toBe('processed');
| Violation | Pattern Category | Fix Strategy |
|---|---|---|
| T0.1 | Prerequisites | Move function to production, import it |
| T0.2 | Prerequisites | Extract non-test functions to src/ |
| T0.3 | Prerequisites | Rename or move functions |
| T1 | Function Call | Remove mock, call real function |
| T2 | Any category | Add result assertion after call |
| T3 | HTTP/DB/Process | Use MSW, test DB, or spawn real process |
| T3+ | Integration Chain | Chain real outputs, don't pass mocks |
Write diagnostic output to logs/diagnostics/assertion-patterns-{YYYYMMDD-HHMMSS}.yaml:
skill: assertion-patterns
timestamp: {ISO-8601}
diagnostics:
t0_checks:
t0_1_imports: pass|fail
t0_2_separation: pass|fail
t0_3_naming: pass|fail
patterns_applied: [T1, T2, T3]
transformations_suggested: 3
files_analyzed: 1
completion_status: success
Provides a checklist for code reviews covering functionality, security, performance, maintainability, tests, and quality. Use for pull requests, audits, team standards, and developer training.
npx claudepluginhub qball-inc/plugins-market --plugin the-bulwark