From kibana-testing-tools
Strategic test suite optimizer for Cypress→Scout migration. Analyzes entire test coverage, identifies blind spots, refactors for performance and DRY principles, then generates optimized Scout suites. Not a 1:1 converter.
How this skill is triggered — by the user, by Claude, or both
Slash command
/kibana-testing-tools:cypress-to-scout-migratorThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
**Purpose:** Strategic test suite optimizer that analyzes Cypress test coverage holistically, identifies optimization opportunities and blind spots, then generates performant, DRY Scout test suites. This is NOT a 1:1 migrator—it's a quality upgrade.
Purpose: Strategic test suite optimizer that analyzes Cypress test coverage holistically, identifies optimization opportunities and blind spots, then generates performant, DRY Scout test suites. This is NOT a 1:1 migrator—it's a quality upgrade.
Context: Kibana is migrating from Cypress to Scout. This is an opportunity to not just convert tests, but to optimize them for performance, maintainability, and coverage. You want to leverage Scout/Playwright best practices (test.step, parallelism, DRY) while catching blind spots in existing coverage.
Philosophy:
Automatic activation triggers:
Manual invocation:
/cypress-to-scout-migrator
Goal: Build a complete feature coverage map, identify redundancy and blind spots, then design an optimized test suite.
find x-pack/{plugins,solutions}/<target-path> -name "*.cy.ts" -type f
For each .cy.ts file, extract:
Build a coverage matrix:
| Feature | User Flow | Assertions | Current Tests | Setup Cost | Redundancy? | Missing Coverage |
|---|---|---|---|---|---|---|
| Alert triage | View alert details | Alert renders with correct fields | alerts.cy.ts (test 1), alert_details.cy.ts (test 2) | Medium (2 API calls) | ✅ YES (2 tests, same assertions) | Missing: 403 forbidden test |
| Alert triage | Assign alert to analyst | Assignment succeeds, assignee visible | alerts.cy.ts (test 3) | Medium (2 API calls) | ❌ NO | Missing: bulk assignment |
| Alert triage | Close alert with comment | Status=closed, comment visible | alerts.cy.ts (test 4) | Medium (2 API calls) | ❌ NO | — |
| Detection rules | Create custom rule | Rule saved, appears in list | rules.cy.ts (test 1), rule_creation.cy.ts (test 1) | High (3 API calls + archive load) | ✅ YES (2 tests, same flow) | Missing: duplicate name error, validation |
| Detection rules | Edit rule schedule | Schedule updated | rules.cy.ts (test 5) | High (3 API calls + archive load) | ❌ NO | — |
| Detection rules | Delete rule | Rule removed from list | MISSING | — | ❌ BLIND SPOT | — |
1. Redundant Tests → Consolidate with test.step()
Pattern: Multiple tests with same setup, single assertion each
Example:
// ❌ Current: 5 separate tests, 5x setup overhead
test('can view alert details', async () => {
await setup(); // 10s
await expect(alertPanel).toBeVisible();
});
test('can assign alert', async () => {
await setup(); // 10s (repeated!)
await expect(assignButton).toBeEnabled();
});
// ✅ Optimized: 1 multi-step test, 1x setup
test('alert triage workflow', async ({ pageObjects }) => {
await test.step('view alert details', async () => {
await expect(alertPanel).toBeVisible();
});
await test.step('assign alert to analyst', async () => {
await assignButton.click();
await expect(assignee).toHaveText('Alice');
});
await test.step('close alert with comment', async () => {
await closeButton.click();
await expect(status).toHaveText('Closed');
});
});
Savings: 80% reduction in execution time (1x setup vs 5x)
2. Overlapping Coverage → Merge into Single Test
Pattern: Different files testing same feature from slightly different angles
Example:
alerts.cy.ts tests alert renderingalert_details.cy.ts tests same rendering with different datatest.step() for data variations3. Missing Coverage (Blind Spots) → Add to Plan
Pattern: Feature has create/edit but no delete/error path tests
Common blind spots:
Action: Add these scenarios to optimized suite
4. Expensive Setup → Global Setup Hook
Pattern: Tests load large archives or create complex data via UI
Example:
// ❌ Current: 10 tests all load same 500MB archive in beforeEach
test.beforeEach(async ({ esArchiver }) => {
await esArchiver.load('x-pack.es_archives.security'); // 50s per test!
});
// ✅ Optimized: Load once in global setup, shared across tests
// In scout.config.ts
globalSetupHook('Load security archive', async ({ esArchiver }) => {
await esArchiver.loadIfNeeded('x-pack.es_archives.security'); // 50s once!
});
Savings: 90% setup time reduction (50s once vs 50s x 10 = 500s)
5. Wrong Test Layer → Move to API Tests
Pattern: UI test validating API response shape (should be API test)
Example:
// ❌ Current: Slow UI test (5s)
test('alert enrichment shows correct data', async ({ page }) => {
await page.goto('/alerts/123');
const response = await page.evaluate(() => fetch('/api/alerts/123/enrich'));
expect(response.body).toMatchObject({ /* exact schema */ });
});
// ✅ Optimized: Fast Scout API test (0.5s)
apiTest('returns enrichment data', async ({ apiClient }) => {
const response = await apiClient.get('api/alerts/123/enrich');
expect(response).toHaveStatusCode(200);
expect(response.body).toMatchObject({ /* exact schema */ });
});
Savings: 10x faster, more reliable, correct test layer
6. UI Setup → API Fixtures
Pattern: Tests create data by clicking through forms (slow, brittle)
Example:
// ❌ Current: Setup via UI (20s)
test.beforeEach(async ({ page }) => {
await page.goto('/rules/create');
await page.fill('[data-test-subj="ruleNameInput"]', 'Test Rule');
await page.click('[data-test-subj="saveButton"]');
// ... 10 more clicks
});
// ✅ Optimized: Setup via API (2s)
test.beforeEach(async ({ apiServices }) => {
testRule = await apiServices.securitySolution.createRule({
name: 'Test Rule',
type: 'query',
query: 'event.action: "process_start"',
});
});
Savings: 90% setup time reduction, more reliable
Output format:
📊 Coverage Analysis: x-pack/solutions/security/plugins/security_solution
## Current State
- **Total Cypress tests:** 47 files, 189 test cases
- **Estimated execution time:** 12.5 minutes (sequential)
- **Test duplication rate:** 28% (53 redundant assertions)
- **Blind spots identified:** 14 missing scenarios
## Optimization Opportunities
### 1. Consolidate Redundant Tests (High Impact)
**Impact:** -19 tests, -6min execution time
| Feature | Current Tests | Optimized Tests | Savings |
|---------|---------------|-----------------|---------|
| Alert triage - view details | 3 tests (same setup) | 1 multi-step test | -2 tests, -40s |
| Rule creation | 5 tests (same setup) | 1 multi-step test | -4 tests, -2min |
| Timeline operations | 8 tests (same setup) | 2 multi-step tests | -6 tests, -3min |
| Case management | 11 tests (overlapping) | 4 multi-step tests | -7 tests, -1.5min |
### 2. Fix Expensive Setup (High Impact)
**Impact:** -4min setup overhead
| Issue | Current Cost | Optimized Approach | Savings |
|-------|-------------|-------------------|---------|
| 10 tests load `x-pack.es_archives.security` in beforeEach | 50s x 10 = 500s | Load once in global setup | -450s |
| 5 tests create users via UI | 20s x 5 = 100s | Use API in beforeEach | -75s |
### 3. Coverage Blind Spots (Critical)
**Action Required:** Add 14 missing test scenarios
| Feature | Missing Scenarios |
|---------|-------------------|
| Detection rules | Delete rule, duplicate name error, invalid config validation |
| Alerts | Bulk operations (assign all, close all), filter persistence |
| Timeline | Export timeline, import timeline, share timeline |
| Cases | Case attachments (add/remove), case status transitions, case tags |
| Endpoints | Endpoint isolation (success + error), endpoint policy change |
### 4. Wrong Test Layer (Medium Impact)
**Impact:** -2min execution time, +reliability
| Test | Current Layer | Correct Layer | Reason |
|------|---------------|---------------|--------|
| `alert_enrichment.cy.ts` | UI (validates API response shape) | Scout API test | No UI behavior, just data validation |
| `rule_execution_stats.cy.ts` | UI (checks exact aggregation values) | Scout API test | Testing API correctness, not UI |
### 5. Scout Best Practices Violations
**Fix during migration:**
- ❌ **Manual waits:** 23 tests use `cy.wait(5000)` → Replace with explicit readiness signals
- ❌ **Duplicate selectors:** Same `data-test-subj` hardcoded in 15 tests → Extract to page objects
- ❌ **No error path coverage:** 80% of tests only test happy path
- ❌ **Admin-only auth:** 35 tests use `loginAsAdmin()` → Use minimal RBAC roles
- ❌ **Setup via UI:** 18 tests create data by clicking through forms → Use API fixtures
## Proposed Optimized Suite
### Metrics
- **New test count:** 28 Scout spec files (61 test cases)
- **Execution time:** 4.2 minutes (parallel, 4 workers)
- **Coverage improvement:** +14 scenarios (blind spots), -53 redundant assertions
- **Test reduction:** -67% test count, -66% execution time
- **Quality improvements:**
- ✅ All tests use `test.step()` for multi-step flows
- ✅ All setup via API fixtures (fast, reliable)
- ✅ Parallel-safe (isolated spaces via `spaceTest`)
- ✅ DRY: shared page objects + fixtures
- ✅ RBAC-aware: minimal permissions, forbidden path coverage
### Test Suite Breakdown
#### UI Tests (18 spec files, 42 test cases)
1. **Alert Triage Workflows** (3 files)
- `alert_details_viewer.spec.ts` — view alert (happy + error), assignment, comments (multi-step)
- `alert_triage_analyst.spec.ts` — bulk operations, filtering, status transitions
- `alert_enrichment_ui.spec.ts` — UI rendering of enrichment data (NOT data correctness)
2. **Detection Rules** (4 files)
- `rule_creation.spec.ts` — create custom rule (multi-step: config → schedule → save)
- `rule_editing.spec.ts` — edit rule, duplicate name error, validation
- `rule_list_operations.spec.ts` — filter, sort, bulk enable/disable
- `rule_deletion.spec.ts` — delete single, bulk delete, forbidden path
3. **Timeline** (3 files)
- `timeline_operations.spec.ts` — create, add events, filter, save (multi-step)
- `timeline_sharing.spec.ts` — export, import, share
- `timeline_viewer.spec.ts` — read-only view, forbidden edit path
4. **Cases** (5 files)
- `case_creation.spec.ts` — create case, add description, save (multi-step)
- `case_attachments.spec.ts` — add/remove attachments, forbidden path
- `case_status_transitions.spec.ts` — open → in-progress → closed (multi-step)
- `case_tags.spec.ts` — add/remove tags, tag filtering
- `case_viewer.spec.ts` — read-only view, forbidden edit path
5. **Endpoints** (3 files)
- `endpoint_isolation.spec.ts` — isolate endpoint (success + error scenarios)
- `endpoint_policy.spec.ts` — apply policy, verify applied
- `endpoint_list.spec.ts` — filter, sort, bulk operations
#### API Tests (10 spec files, 19 test cases)
1. **Alert Enrichment Data** — validate enrichment API responses (not UI)
2. **Rule Execution Stats** — validate aggregation correctness (not UI)
3. **Timeline Export Format** — validate export JSON schema
4. **Case API RBAC** — validate 403 responses for under-privileged roles
5. **Endpoint Isolation API** — validate isolation state transitions
6. ... (5 more API tests moved from UI layer)
---
## Next Step: Review & Approval
**Questions for you:**
1. **Approve consolidation strategy?** (19 redundant tests → 6 multi-step flows)
2. **Approve blind spot additions?** (14 new scenarios)
3. **Approve test layer moves?** (2 UI tests → Scout API tests)
4. **Any features to deprioritize?** (reduce scope if needed)
**Once approved**, I'll proceed to Phase 2: Implementation Plan.
Goal: Generate detailed implementation plan with pattern mappings, fixture/page object designs, and batch breakdown.
1. Page Objects (for DRY, reusable UI interactions)
// test/scout_ui/pages/alert_details_page.ts
export class AlertDetailsPage {
constructor(private readonly page: ScoutPage) {}
async goto(alertId: string) {
await this.page.goto(`/app/security/alerts/${alertId}`);
await this.page.testSubj.waitForSelector('alertDetailsPanel-loaded', { state: 'visible' });
}
async assignToUser(username: string) {
await this.page.testSubj.click('alertAssignButton');
await this.page.testSubj.fill('assignUserInput', username);
await this.page.testSubj.click('assignUserConfirm');
await this.page.testSubj.waitForSelector('assignmentSuccessToast', { state: 'visible' });
}
async closeWithComment(comment: string) {
await this.page.testSubj.click('alertCloseButton');
await this.page.testSubj.fill('closeCommentInput', comment);
await this.page.testSubj.click('closeAlertConfirm');
await expect(this.page.testSubj.locator('alertStatus')).toHaveText('Closed');
}
}
2. API Services (for setup/teardown)
// test/scout_api/services/security_solution_service.ts
export class SecuritySolutionService {
async createRule(config: RuleConfig) {
const response = await this.apiClient.post('api/detection_engine/rules', {
body: config,
headers: COMMON_HEADERS,
});
return response.body;
}
async deleteRule(ruleId: string) {
await this.apiClient.delete(`api/detection_engine/rules/${ruleId}`, {
headers: COMMON_HEADERS,
});
}
}
3. Fixtures (for test setup/auth)
// test/scout_ui/fixtures/index.ts
export const test = base.extend<{
alertDetailsPage: AlertDetailsPage;
securitySolutionService: SecuritySolutionService;
}>({
alertDetailsPage: async ({ page }, use) => {
await use(new AlertDetailsPage(page));
},
securitySolutionService: async ({ apiClient }, use) => {
await use(new SecuritySolutionService(apiClient));
},
});
| Cypress Pattern | Scout Equivalent | Notes |
|---|---|---|
cy.visit(url) | await page.goto(url) | Add wait for loaded state |
cy.get(selector).click() | await page.testSubj.click(testSubj) | Prefer data-test-subj |
cy.get(selector).type(text) | await page.testSubj.fill(testSubj, text) | fill() replaces, type() appends |
cy.contains(text) | await page.getByText(text) | Scoped within container |
cy.wait(ms) | await page.waitForSelector() | Never use waitForTimeout() |
cy.intercept(route) | await page.route(route, handler) | Or use apiClient for API tests |
cy.request(url) | await apiClient.get(url) | Use Scout's apiClient fixture |
| Custom commands | Scout page objects | Convert to page object methods |
beforeEach setup via UI | beforeEach setup via API | Use apiServices |
Archive load in beforeEach | Global setup hook | Load once, share across tests |
cy.should('be.visible') | await expect(locator).toBeVisible() | Playwright auto-waits |
Strategy: 5-7 tests per PR, grouped by feature area
Batch 1: Alert Triage (3 UI tests + 1 API test)
alert_details_viewer.spec.tsalert_triage_analyst.spec.tsalert_enrichment_ui.spec.tsalert_enrichment_api.spec.tsBatch 2: Detection Rules (4 UI tests + 2 API tests)
rule_creation.spec.tsrule_editing.spec.tsrule_list_operations.spec.tsrule_deletion.spec.tsrule_execution_stats_api.spec.tsrule_rbac_api.spec.tsBatch 3-5: Timeline, Cases, Endpoints (similar structure)
Output: Detailed implementation plan for each batch with:
Goal: Implement optimized Scout tests, validate locally, create PR.
For each test in batch:
test.describe + testtest.step() for multi-step flowsapiServices for setup/teardownType check:
yarn test:type_check --project test/scout_ui/tsconfig.json
Lint:
node scripts/eslint --fix test/scout_ui/<new-file>.spec.ts
Run tests (20-50 runs for flake detection):
node scripts/scout run-tests --arch stateful --domain classic \
--config test/scout_ui/scout.config.ts \
--testFiles test/scout_ui/<new-file>.spec.ts \
--repeat 50
Auto-fix common issues:
await → Add to all async callsgetByRole() or add data-test-subj to sourcewaitForTimeout)Invoke skill:
/scout-best-practices-reviewer
Fix any issues raised, then proceed to PR.
PR structure:
## Summary
Migrates Cypress E2E tests to Scout (Playwright) for <plugin/solution> with strategic optimization.
## Tests Migrated & Optimized (Batch 1/5)
### UI Tests (3 files, 7 test cases)
- ✅ `alert_details_viewer.spec.ts` — Consolidated 3 Cypress tests into 1 multi-step flow
- ✅ `alert_triage_analyst.spec.ts` — Consolidated 5 Cypress tests into 1 multi-step flow
- ✅ `alert_enrichment_ui.spec.ts` — New test (UI rendering only, not data validation)
### API Tests (1 file, 2 test cases)
- ✅ `alert_enrichment_api.spec.ts` — Moved from Cypress UI test (data validation)
### Coverage Improvements
- ✅ Added forbidden path test (viewer tries to assign alert → 403)
- ✅ Added bulk operations test (assign all, close all)
## Optimization Metrics
| Metric | Before (Cypress) | After (Scout) | Δ |
|--------|------------------|---------------|---|
| Test files | 8 | 4 | -50% |
| Test cases | 23 | 9 | -61% |
| Execution time | 3.5min (sequential) | 1.2min (parallel, 4 workers) | -66% |
| Setup overhead | 160s (archive loaded 8x) | 50s (global setup hook) | -69% |
| Coverage gaps | 2 missing scenarios | 0 (added) | +100% |
## Quality Improvements
- ✅ All tests use `test.step()` for multi-step flows (no redundant setup)
- ✅ All setup via API fixtures (fast, reliable)
- ✅ Parallel-safe (isolated spaces via `spaceTest`)
- ✅ DRY: shared page objects in `test/scout_ui/pages/alert_details_page.ts`
- ✅ RBAC-aware: minimal permissions, forbidden path coverage
- ✅ No manual waits (`cy.wait(5000)` → explicit readiness signals)
## Pattern Mapping Applied
| Cypress | Scout |
|---------|-------|
| `cy.get('[data-test-subj="foo"]')` | `page.testSubj.locator('foo')` |
| `cy.request('/api/...')` | `apiClient.get('/api/...')` |
| `cy.wait(5000)` | `page.testSubj.waitForSelector('loaded')` |
| `beforeEach` setup via UI | `beforeEach` setup via API |
| Archive load in `beforeEach` | Global setup hook |
## Validation
- ✅ Type checks pass
- ✅ Linting passes
- ✅ All tests pass locally (50/50 runs, no flakes)
- ✅ Behavior matches Cypress (verified via screenshots)
- ✅ scout-best-practices-reviewer approved
## Next Steps
- **Do NOT delete Cypress files in this PR** (delete in follow-up after Scout tests proven stable in CI)
- CI will run Scout tests automatically
- Monitor Buildkite for any environment-specific failures
- Proceed with Batch 2 after this PR merges
---
**Migration batch:** 1 of 5 (28 total tests, 61 test cases)
**Generated by:** @cypress-to-scout-migrator
Commit message:
test(scout): optimize & migrate Cypress tests to Scout (batch 1/5)
Consolidates 23 Cypress tests into 9 optimized Scout tests with:
- Multi-step flows via test.step (no redundant setup)
- API fixtures for fast setup (not UI clicks)
- Parallel execution via spaceTest (4 workers)
- Coverage improvements (forbidden paths, bulk ops)
Performance: -66% execution time (3.5min → 1.2min)
Quality: +2 missing scenarios, DRY page objects, RBAC-aware
Part of strategic Cypress → Scout migration initiative.
Stop when:
Max iterations: 10 iterations per batch (if tests fail/flake, fix and re-run)
npx claudepluginhub patrykkopycinski/patryks-treadmill-claude-plugins --plugin kibana-testing-toolsGuides creation, editing, and verification of skills for AI coding agents using test-driven development with subagent scenarios. Use when authoring or debugging skills.