From qa-experimentation
Wraps Amplitude Experiment SDK testing patterns: client initialization with API key (or local-flags JSON), the fetch / variant API, exposure-event suppression in tests, and assignment-integrity tests. Use when writing tests for code that uses Amplitude Experiment for A/B testing or flag management. Composes guardrail-metrics-reference + peeking-problem-reference + ab-test-validity-checklist.
How this skill is triggered — by the user, by Claude, or both
Slash command
/qa-experimentation:amplitude-experiment-testThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Per
Per
amplitude.com/docs/experiment,
the Amplitude Experiment SDKs (server-side and client-side)
expose fetch + variant APIs: fetch the user's assigned
variants, then read each variant on demand.
Amplitude correlates exposure + outcome events via the same user ID space as Amplitude Analytics, so exposure-event suppression in tests is important to avoid polluting analytics.
ab-test-validity-checklist
Step 3.pip install amplitude-experiment # Python (server-side)
npm install --save-dev @amplitude/experiment-node-server
import * as Experiment from '@amplitude/experiment-node-server';
const client = Experiment.Experiment.initializeRemote(API_KEY, {
// Suppress real fetches in tests
fetchTimeoutMillis: 1000,
});
For fully-offline tests, use the local evaluation mode with a flags snapshot:
import { LocalEvaluationClient } from '@amplitude/experiment-node-server';
const localClient = new LocalEvaluationClient(API_KEY, {
flagConfigPollerIntervalMillis: 0, // No polling in tests
});
// Or provide flags directly via the local-evaluation-config fixture
await localClient.start(localFlagConfigJson);
const user = { user_id: 'user-1', device_id: 'dev-1' };
test('user variant from local eval', async () => {
const variants = localClient.evaluate(user);
expect(variants['checkout-experiment'].value).toBe('treatment-a');
});
Amplitude Experiment's standard pattern is via the flag config:
override the flag's default-variant for a specific user ID by
modifying the local-eval fixture. Alternatively, mock the
evaluate method:
import { jest } from '@jest/globals';
test('user in treatment', () => {
jest.spyOn(localClient, 'evaluate').mockReturnValue({
'checkout-experiment': { value: 'treatment-a' } as any,
});
const variants = localClient.evaluate(user);
expect(variants['checkout-experiment'].value).toBe('treatment-a');
});
Default behavior fires an exposure event on variant() read.
Suppress per
amplitude.com/docs/experiment:
// In test setup:
const client = Experiment.Experiment.initializeRemote(API_KEY, {
// Disable automatic exposure tracking
automaticExposureTracking: false,
});
test('deterministic assignment', () => {
const v1 = localClient.evaluate({ user_id: 'user-1' });
const v2 = localClient.evaluate({ user_id: 'user-1' });
expect(v1).toEqual(v2);
});
npm test
jobs:
amplitude-experiment-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v4
- run: npm ci
- run: npm test
env:
AMPLITUDE_API_KEY: ${{ secrets.AMPLITUDE_TEST_KEY }}
For fully-offline CI: skip the env var and use local-eval with checked-in flag config JSON.
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Tests use prod Amplitude key | Test users pollute analytics | Use test workspace + dev key |
| Exposure events enabled in CI | Spurious exposure tracking | automaticExposureTracking: false |
| Mocking variant() result without testing the fetch | Misses fetch-network bugs | Test both layers separately |
| Local-eval flag JSON not committed | Test flakes when prod changes | Commit fixture |
Skipping client.stop() / cleanup | Network handles leak | Always teardown |
| Different user-ID space between test + analytics | Amplitude correlation broken | Match the prod user-ID strategy |
guardrail-metrics-reference,
peeking-problem-reference,
ab-test-validity-checklist.statsig-test,
optimizely-test,
vwo-test.npx claudepluginhub testland/qa --plugin qa-experimentationProvides UI/UX resources: 50+ styles, color palettes, font pairings, guidelines, charts for web/mobile across React, Next.js, Vue, Svelte, Tailwind, React Native, Flutter. Aids planning, building, reviewing interfaces.
Fetches up-to-date documentation from Context7 for libraries and frameworks like React, Next.js, Prisma. Use for setup questions, API references, and code examples.