amplitude-experiment-test

amplitude-experiment-test

Overview

Per amplitude.com/docs/experiment, the Amplitude Experiment SDKs (server-side and client-side) expose fetch + variant APIs: fetch the user's assigned variants, then read each variant on demand.

Amplitude correlates exposure + outcome events via the same user ID space as Amplitude Analytics, so exposure-event suppression in tests is important to avoid polluting analytics.

When to use

• Tests for code that reads an Amplitude Experiment variant.
• Suppressing exposure events in non-production test runs.
• Assignment-integrity tests per ab-test-validity-checklist Step 3.

Authoring

Install

pip install amplitude-experiment           # Python (server-side)
npm install --save-dev @amplitude/experiment-node-server

Initialize (server-side)

import * as Experiment from '@amplitude/experiment-node-server';

const client = Experiment.Experiment.initializeRemote(API_KEY, {
  // Suppress real fetches in tests
  fetchTimeoutMillis: 1000,
});

For fully-offline tests, use the local evaluation mode with a flags snapshot:

import { LocalEvaluationClient } from '@amplitude/experiment-node-server';

const localClient = new LocalEvaluationClient(API_KEY, {
  flagConfigPollerIntervalMillis: 0,   // No polling in tests
});

// Or provide flags directly via the local-evaluation-config fixture
await localClient.start(localFlagConfigJson);

Fetch + read variant

const user = { user_id: 'user-1', device_id: 'dev-1' };

test('user variant from local eval', async () => {
  const variants = localClient.evaluate(user);
  expect(variants['checkout-experiment'].value).toBe('treatment-a');
});

Force a variant for a test

Amplitude Experiment's standard pattern is via the flag config: override the flag's default-variant for a specific user ID by modifying the local-eval fixture. Alternatively, mock the evaluate method:

import { jest } from '@jest/globals';

test('user in treatment', () => {
  jest.spyOn(localClient, 'evaluate').mockReturnValue({
    'checkout-experiment': { value: 'treatment-a' } as any,
  });

  const variants = localClient.evaluate(user);
  expect(variants['checkout-experiment'].value).toBe('treatment-a');
});

Suppress exposure events in tests

Default behavior fires an exposure event on variant() read. Suppress per amplitude.com/docs/experiment:

// In test setup:
const client = Experiment.Experiment.initializeRemote(API_KEY, {
  // Disable automatic exposure tracking
  automaticExposureTracking: false,
});

Assignment integrity tests

test('deterministic assignment', () => {
  const v1 = localClient.evaluate({ user_id: 'user-1' });
  const v2 = localClient.evaluate({ user_id: 'user-1' });
  expect(v1).toEqual(v2);
});

Running

npm test

CI integration

jobs:
  amplitude-experiment-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm test
        env:
          AMPLITUDE_API_KEY: ${{ secrets.AMPLITUDE_TEST_KEY }}

For fully-offline CI: skip the env var and use local-eval with checked-in flag config JSON.

Anti-patterns

Anti-pattern	Why it fails	Fix
Tests use prod Amplitude key	Test users pollute analytics	Use test workspace + dev key
Exposure events enabled in CI	Spurious exposure tracking	automaticExposureTracking: false
Mocking variant() result without testing the fetch	Misses fetch-network bugs	Test both layers separately
Local-eval flag JSON not committed	Test flakes when prod changes	Commit fixture
Skipping client.stop() / cleanup	Network handles leak	Always teardown
Different user-ID space between test + analytics	Amplitude correlation broken	Match the prod user-ID strategy

Limitations

• Local-evaluation mode is feature-limited. Some flag types (CMAB, multi-armed bandit) aren't supported offline.
• Mocking variant() loses targeting-rule fidelity. Use real local-eval when targeting matters.
• Exposure suppression is binary. Can't selectively suppress per-test.
• Doesn't validate Amplitude's results analysis. Platform-side statistics separate.

References

• Amplitude Experiment docs: amplitude.com/docs/experiment.
• Local evaluation: amplitude.com/docs/experiment/general/evaluation/local-evaluation.
• Companion catalogs: guardrail-metrics-reference, peeking-problem-reference, ab-test-validity-checklist.
• Sibling SDKs: statsig-test, optimizely-test, vwo-test.