From qa-experimentation
Wraps Optimizely Feature Experimentation SDK testing patterns: client initialization with a datafile (offline-friendly), the decide / decideAll API (Optimizely Feature Experimentation, the v5 API), forced-decisions for per-test arm pinning, OptimizelyUserContext + activate / track events, and assignment-integrity tests. Use when writing tests for Optimizely-instrumented application code. Composes guardrail-metrics-reference + peeking-problem-reference + ab-test-validity-checklist.
How this skill is triggered — by the user, by Claude, or both
Slash command
/qa-experimentation:optimizely-testThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Optimizely Feature Experimentation (Optimizely Full Stack /
Optimizely Feature Experimentation (Optimizely Full Stack / Optimizely X) uses a datafile - a JSON blob describing all flags, experiments, and audiences - that the SDK fetches and evaluates locally. Per docs.developers.optimizely.com/feature-experimentation/docs/python-sdk, the SDK supports datafile-based testing: load a fixture datafile in tests, no network call.
The current API surface is decide (single flag) / decideAll
(all flags) per the v5 SDK.
pip install optimizely-sdk # Python
npm install --save-dev @optimizely/optimizely-sdk
Per Optimizely docs, the datafile path is the canonical offline approach:
import json
from optimizely import optimizely
# Load a checked-in datafile fixture
with open("tests/fixtures/optimizely-datafile.json") as f:
datafile = json.load(f)
client = optimizely.Optimizely(json.dumps(datafile))
The datafile is downloadable from the Optimizely UI or via the Optimizely API; commit a version-specific copy to the repo for deterministic tests.
def test_get_decision_for_user():
user = client.create_user_context("user-1", {"plan": "premium"})
decision = user.decide("new_checkout_flow")
assert decision.enabled is True
assert decision.variation_key == "treatment_a"
from optimizely.optimizely_user_context import OptimizelyDecisionContext
def test_force_user_to_treatment():
user = client.create_user_context("user-1")
context = OptimizelyDecisionContext(flag_key="new_checkout_flow", rule_key=None)
user.set_forced_decision(context, OptimizelyForcedDecision(variation_key="treatment_a"))
decision = user.decide("new_checkout_flow")
assert decision.variation_key == "treatment_a"
def test_assignment_deterministic():
user_a1 = client.create_user_context("user-1")
user_a2 = client.create_user_context("user-1")
d1 = user_a1.decide("flag-x")
d2 = user_a2.decide("flag-x")
assert d1.variation_key == d2.variation_key
def test_decide_all_returns_consistent_set():
user = client.create_user_context("user-1")
decisions_1 = user.decide_all()
decisions_2 = user.decide_all()
assert decisions_1.keys() == decisions_2.keys()
def test_conversion_event_emitted():
captured_events = []
# Optimizely supports a notification listener
client.notification_center.add_notification_listener(
notification_type="TRACK", notification_callback=lambda *args: captured_events.append(args)
)
user = client.create_user_context("user-1")
user.track_event("checkout_completed")
assert len(captured_events) == 1
pytest tests/optimizely/
jobs:
optimizely-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v5
- run: pip install -e ".[test]"
- run: pytest tests/optimizely/
Datafile lives in the repo; no SDK key needed for tests.
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Tests with live SDK key | Production data polluted; rate-limited | Use datafile fixture |
| Datafile not version-controlled | Tests flake when prod config changes | Commit the fixture |
| Forced decisions leak across tests | Cross-test pollution | Per-test user context; reset before assertion |
Skipping client.shutdown / network listener cleanup | Goroutine / handle leak | Always cleanup |
| Asserting on variation IDs not keys | IDs change per environment | Use variation_key |
| Manual event tracking in tests vs notification listener | Misses platform-emitted events | Use the listener |
| Tests rely on real decide-network roundtrip | Slow; non-deterministic | Datafile + offline |
guardrail-metrics-reference,
peeking-problem-reference,
ab-test-validity-checklist.statsig-test,
vwo-test,
amplitude-experiment-test.Provides UI/UX resources: 50+ styles, color palettes, font pairings, guidelines, charts for web/mobile across React, Next.js, Vue, Svelte, Tailwind, React Native, Flutter. Aids planning, building, reviewing interfaces.
Fetches up-to-date documentation from Context7 for libraries and frameworks like React, Next.js, Prisma. Use for setup questions, API references, and code examples.
npx claudepluginhub testland/qa --plugin qa-experimentation