perf-optimizer

Performance Optimizer

Description

Identifies and fixes performance bottlenecks in Kibana build, test, and CI workflows. Analyzes slow operations, diagnoses root causes, suggests targeted optimizations, and measures impact.

Triggers

• "optimize build time"
• "why is this test slow"
• "reduce CI time"
• "analyze performance"
• "speed up [build|test|CI]"
• "profile [webpack|jest|scout]"

Core Capabilities

1. Detect Slow Operations

• Build analysis: Webpack bundle size, compilation time, plugin loading
• Test profiling: Jest/Scout execution time, setup/teardown duration
• CI timing: Buildkite agent hours, step duration, parallelism efficiency
• Bootstrap analysis: yarn kbn bootstrap dependency resolution

2. Analyze Bottlenecks

• Bundle bloat: Large packages (>5MB), duplicate dependencies, dead code
• Test inefficiency: Expensive setup (>30s), redundant ES archive loads, serial execution
• CI waste: Redundant steps, under-utilized parallelism, cache misses
• Startup overhead: Lazy loadable code, synchronous imports, unused plugins

3. Suggest Optimizations

• Code splitting: Dynamic imports, route-based chunks, lazy components
• Caching: Webpack cache, Jest cache, Scout ES archive snapshots
• Parallelization: Increase Jest/Scout workers, Buildkite agent count
• Tree-shaking: Replace lodash with lodash-es, eliminate side effects
• Global setup: Shared test fixtures, reusable ES instances

4. Measure Impact

• Before/after metrics: Timing, bundle size, agent-hours
• Cost analysis: Agent-hour cost, developer time saved
• ROI calculation: One-time effort vs ongoing savings

Instructions

Phase 1: Detection & Diagnosis

When user requests performance analysis:

1. Identify scope

   • Build performance → webpack analysis
   • Test performance → Jest/Scout profiling
   • CI performance → Buildkite analytics
   • Bootstrap performance → dependency graph analysis

2. Gather baseline metrics

   # Build: Generate webpack stats
   STATS_JSON=true node scripts/build_kibana_platform_plugins.js --focus <plugin>
   
   # Test: Profile Jest with --verbose --detectOpenHandles
   yarn test:jest --config <config> --verbose --detectOpenHandles
   
   # Scout: Check test timings in Scout HTML report
   # Look for tests with >30s duration in timeline
   
   # CI: Query Buildkite GraphQL API for recent builds
   # Group by step, calculate p50/p95/p99 timing
   

3. Run automated analysis

   • Use webpack-bundle-analyzer for bundle visualization
   • Parse Jest --json output for slow tests
   • Check CI logs for repeated expensive operations
   • Profile Node.js with --cpu-prof if needed

Phase 2: Root Cause Analysis

Bundle Bloat Patterns

• Symptom: Plugin bundle >5MB
• Causes:
  • Lodash imported entire library (450KB)
  • Moment.js with all locales (160KB)
  • Multiple versions of same dependency
  • Unminified source maps in production
• Detection:

  # Find large dependencies
  cat webpack-stats.json | jq '.assets | sort_by(.size) | reverse | .[0:10]'
  
  # Check for duplicate packages
  yarn dedupe --check
  
  # Analyze specific plugin
  npx webpack-bundle-analyzer webpack-stats.json
  

Test Inefficiency Patterns

• Symptom: Test suite takes >5min
• Causes:
  • Each test loads ES archive independently
  • Expensive setup in beforeEach (should be in beforeAll)
  • Serial execution (single worker)
  • Unused Playwright browser context
• Detection:

  # Profile Jest test timing
  yarn test:jest --config <config> --verbose 2>&1 | grep "PASS\|FAIL" | awk '{print $2, $NF}'
  
  # Check Scout parallelism
  grep "workers:" <scout_config>
  
  # Identify expensive setup
  # Look for "beforeEach" with >1s operations
  grep -r "beforeEach" --include="*.test.ts" | xargs grep -l "es.indices.create\|kibanaServer.importExport"
  

CI Waste Patterns

• Symptom: Build takes >60min
• Causes:
  • Bootstrap runs on every agent (not cached)
  • Tests run serially (should parallelize)
  • Artifacts uploaded/downloaded unnecessarily
  • Redundant type checks across steps
• Detection:

  # Check Buildkite step timing
  buildkite-agent step get <step-key> --format json | jq '.duration'
  
  # Find steps running bootstrap
  grep -r "yarn kbn bootstrap" .buildkite/
  
  # Check parallelism
  grep "parallelism:" .buildkite/pipeline.yml
  

Phase 3: Optimization Strategies

Build Optimizations

1. Code Splitting

// Before: Synchronous import (bloats main bundle)
import { HeavyComponent } from './heavy_component';

// After: Dynamic import (separate chunk)
const HeavyComponent = lazy(() => import('./heavy_component'));

2. Tree-Shaking

// Before: Imports entire lodash (450KB)
import _ from 'lodash';

// After: Tree-shakeable imports (20KB)
import { debounce, throttle } from 'lodash-es';

3. Webpack Cache

// webpack.config.js
module.exports = {
  cache: {
    type: 'filesystem',
    buildDependencies: {
      config: [__filename],
    },
  },
};

4. Plugin Lazy Loading

// kibana.jsonc
{
  "type": "plugin",
  "plugin": {
    "id": "myPlugin",
    "requiredPlugins": ["data"],
    // Only load when route accessed
    "optionalPlugins": ["maps", "lens"]
  }
}

Test Optimizations

1. Shared ES Archives

// Before: Each test loads archive (5s per test)
beforeEach(async () => {
  await esArchiver.load('security_solution/alerts');
});

// After: Load once, reuse across tests (5s total)
beforeAll(async () => {
  await esArchiver.load('security_solution/alerts');
});
afterAll(async () => {
  await esArchiver.unload('security_solution/alerts');
});

2. Increase Parallelism

// jest.config.js
module.exports = {
  // Before: 1 worker (serial)
  maxWorkers: 1,

  // After: 50% of CPU cores
  maxWorkers: '50%',
};

// scout.config.ts
export default {
  // Before: 1 worker
  workers: 1,

  // After: Match available CPU cores
  workers: Math.max(1, Math.floor(os.cpus().length * 0.75)),
};

3. Global Setup Hook

// jest.config.js
module.exports = {
  globalSetup: '<rootDir>/global_setup.ts',
  globalTeardown: '<rootDir>/global_teardown.ts',
};

// global_setup.ts
export default async () => {
  // Start ES once for all tests
  const es = await startElasticsearch();
  process.env.ES_URL = es.url;
};

4. Scout ES Snapshot Caching

// Use Scout's built-in snapshot system
test.beforeAll(async ({ kbnClient }) => {
  // Load data once, snapshot the ES state
  await kbnClient.importExport.load(...);
  await kbnClient.savedObjects.create(...);
});

// Subsequent tests restore from snapshot (faster)

CI Optimizations

1. Cache Bootstrap

# .buildkite/pipeline.yml
steps:
  - label: "Bootstrap"
    command: "yarn kbn bootstrap"
    # Cache node_modules across builds
    plugins:
      - cache#v1:
          key: "v1-yarn-{{ checksum 'yarn.lock' }}"
          paths:
            - "node_modules"
            - ".yarn/cache"

2. Parallelize Tests

# Before: Serial execution
steps:
  - label: "Test Suite"
    command: "yarn test:jest"

# After: Parallel execution
steps:
  - label: "Test Suite"
    command: "yarn test:jest"
    parallelism: 10
    # Split tests across 10 agents

3. Incremental Type Checking

# Only check changed files
steps:
  - label: "Type Check"
    command: |
      CHANGED_FILES=$(git diff --name-only origin/main...HEAD | grep '\.tsx\?$')
      for file in $CHANGED_FILES; do
        # Find nearest tsconfig
        TSCONFIG=$(find $(dirname $file) -name tsconfig.json -type f | head -n1)
        yarn test:type_check --project $TSCONFIG
      done

4. Conditional Steps

# Skip tests if no code changed
steps:
  - label: "API Tests"
    if: |
      git diff --name-only origin/main...HEAD | grep -E "src/.*\.ts$"
    command: "yarn test:jest --config jest.integration.config.js"

Phase 4: Impact Measurement

Metrics to Track

1. Build Performance

   • Bundle size (MB): Before vs After
   • Compilation time (seconds): Before vs After
   • First load time (seconds): Before vs After

2. Test Performance

   • Suite duration (minutes): Before vs After
   • Test count: No change expected
   • Failure rate: No change expected

3. CI Performance

   • Pipeline duration (minutes): Before vs After
   • Agent hours consumed: Before vs After
   • Cost ($): Before vs After (agent-hour rate)

4. Developer Experience

   • Local build time: Before vs After
   • Test feedback loop: Before vs After
   • Merge queue time: Before vs After

Measurement Template

## Performance Optimization Results

### Scope
- **Target**: [Build/Test/CI]
- **Package**: [Plugin/Package name]
- **Date**: [YYYY-MM-DD]

### Baseline (Before)
- Metric 1: [value + unit]
- Metric 2: [value + unit]
- Metric 3: [value + unit]

### Optimizations Applied
1. [Optimization name]: [Brief description]
2. [Optimization name]: [Brief description]

### Results (After)
- Metric 1: [value + unit] (**-XX%**)
- Metric 2: [value + unit] (**-XX%**)
- Metric 3: [value + unit] (**-XX%**)

### Cost-Benefit Analysis
- **Implementation time**: X hours
- **Time saved per build/test/CI run**: Y minutes
- **Runs per day**: Z
- **Total time saved per day**: Y * Z minutes
- **ROI breakeven**: X hours / (Y * Z hours/day) = N days

### Recommendations
- [Next optimization opportunity]
- [Monitoring to prevent regression]

Phase 5: Implementation & Validation

1. Create optimization branch

   git checkout -b perf/optimize-<target>-<date>
   

2. Apply optimizations incrementally

   • One optimization per commit
   • Measure impact after each change
   • Document reasoning in commit message

3. Validate no regressions

   # Build: Verify bundle size
   STATS_JSON=true node scripts/build_kibana_platform_plugins.js
   
   # Test: Verify all tests pass
   yarn test:jest --config <config>
   
   # Type check
   yarn test:type_check --project <tsconfig>
   
   # Lint
   node scripts/eslint --fix $(git diff --name-only)
   

4. Create PR with metrics

   • Include before/after comparison
   • Link to Buildkite builds showing improvement
   • Document any trade-offs (e.g., lazy loading = slight delay on first use)

Kibana-Specific Patterns

Bootstrap Optimization

# Problem: Bootstrap takes 10+ minutes
# Cause: Re-resolves dependencies every time

# Solution: Use Yarn PnP (Plug'n'Play)
yarn config set nodeLinker pnp
yarn install

# Or: Cache node_modules in CI (see CI Optimizations above)

Scout Parallelism

// scout.config.ts
import os from 'os';

export default {
  // Match CPU cores (leave 2 for system)
  workers: Math.max(1, os.cpus().length - 2),

  // Reuse browser contexts (faster than fresh launch)
  fullyParallel: false,

  // Keep browser alive between tests
  reuseExistingServer: !process.env.CI,
};

ES Archive Caching

// Use Scout's snapshot system instead of manual archiving
import { test } from '@kbn/scout';

test.describe('Alerts', () => {
  test.beforeAll(async ({ kbnClient, esClient }) => {
    // Load data once
    await esClient.indices.create({ index: 'alerts' });
    await kbnClient.savedObjects.create(...);

    // Scout automatically snapshots ES state
  });

  test.afterAll(async ({ esClient }) => {
    // Cleanup
    await esClient.indices.delete({ index: 'alerts' });
  });

  // Each test gets fresh ES state from snapshot (fast)
  test('test 1', async () => { ... });
  test('test 2', async () => { ... });
});

Plugin Lazy Loading

// Only load expensive plugins when needed
// kibana.jsonc
{
  "type": "plugin",
  "plugin": {
    "id": "myPlugin",
    "requiredPlugins": ["data"],
    // Lazy load (don't block startup)
    "optionalPlugins": ["maps", "lens", "ml"],
    "requiredBundles": ["kibanaReact"]
  }
}

// In plugin code
export class MyPlugin {
  setup(core, plugins) {
    // Register route that lazy loads Maps
    core.http.route({
      path: '/app/my-plugin/map',
      handler: async () => {
        // Only load Maps when this route accessed
        const maps = await core.plugins.getPlugin('maps');
        return maps ? maps.renderMap() : null;
      },
    });
  }
}

Example Workflows

Example 1: Optimize Security Solution Build

User Input: "Why is the Security Solution build so slow?"

Agent Actions:

1. Generate webpack stats

   cd x-pack/solutions/security
   STATS_JSON=true node scripts/build_kibana_platform_plugins.js --focus securitySolution
   

2. Analyze bundle

   npx webpack-bundle-analyzer webpack-stats.json
   

3. Findings:

   • lodash imported 47 times (450KB)
   • moment.js with all locales (160KB)
   • react-beautiful-dnd duplicated (120KB)

4. Optimizations:

   • Replace import _ from 'lodash' with import { debounce } from 'lodash-es'
   • Add webpack plugin: new MomentLocalesPlugin({ localesToKeep: ['en'] })
   • Run yarn dedupe to remove duplicate react-beautiful-dnd

5. Results:

   • Bundle size: 8.2MB → 5.8MB (-29%)
   • Build time: 4m 20s → 3m 10s (-27%)
   • First load time: 3.2s → 2.1s (-34%)

Example 2: Speed Up Scout Tests

User Input: "reduce CI time for detection engine tests"

Agent Actions:

1. Profile test suite

   node scripts/scout run-tests \
     --arch stateful \
     --domain security \
     --config x-pack/test/security_solution_scout/detection_engine.scout.config.ts \
     --reporter html
   

2. Findings (from HTML report timeline):

   • 45 tests, 12m 30s total
   • Each test loads ES archive (5s per test = 3m 45s wasted)
   • Running on 1 worker (serial execution)
   • Browser launched 45 times (20s startup overhead)

3. Optimizations:

   • Move ES archive load to beforeAll (5s once vs 5s * 45)
   • Increase workers from 1 to 4 (4x parallelism)
   • Enable browser context reuse

4. Code changes:

   // detection_engine.scout.config.ts
   export default {
     workers: 4, // Was: 1
     fullyParallel: false, // Reuse browser contexts
   };
   
   // detection_engine.test.ts
   test.beforeAll(async ({ esClient, kbnClient }) => {
     // Load once
     await esArchiver.load('security_solution/detection_engine');
   });
   
   test.afterAll(async ({ esClient }) => {
     await esArchiver.unload('security_solution/detection_engine');
   });
   

5. Results:

   • Suite duration: 12m 30s → 3m 45s (-70%)
   • CI agent hours: 12.5 → 3.75 (-70%)
   • Cost savings: $0.50/agent-hour * 8.75 hours = $4.38 per run

Example 3: Optimize CI Pipeline

User Input: "analyze performance of platform builds"

Agent Actions:

1. Query Buildkite API for recent builds

   query {
     pipeline(slug: "elastic/kibana") {
       builds(first: 20, branch: "main") {
         edges {
           node {
             jobs {
               ... on JobTypeCommand {
                 label
                 runnableAt
                 finishedAt
               }
             }
           }
         }
       }
     }
   }
   

2. Findings:

   • "Bootstrap" step: 8m 30s (runs on every agent)
   • "Type Check" step: 12m (checks all packages, not just changed)
   • Tests run serially: 45m total (could parallelize)
   • Artifacts uploaded unnecessarily: 2m overhead

3. Optimizations:

   • Cache bootstrap (see CI Optimizations above)
   • Incremental type checking (only changed packages)
   • Parallelize tests across 10 agents
   • Skip artifact upload for non-release builds

4. Results:

   • Pipeline duration: 58m → 22m (-62%)
   • Agent hours: 58 → 30 (-48%)
   • Cost savings: $0.50/hour * 28 hours = $14 per build
   • Estimated monthly savings: $14 * 100 builds = $1,400/month

Validation Checklist

Before declaring optimization complete:

• Metrics collected (before/after)
• All tests pass (no regressions)
• Type check passes
• Linting passes
• Build succeeds
• Local development workflow verified
• CI builds pass
• Cost-benefit analysis documented
• Monitoring alerts configured (prevent regression)

Edge Cases & Warnings

Don't Over-Optimize

• Optimizing code that runs once (e.g., one-time migration) has low ROI
• Prefer simple, readable code over premature optimization
• Profile first, optimize second

Watch for Regressions

• Lazy loading can introduce UI delays (first load slower)
• Aggressive caching can mask bugs (stale data)
• Parallel tests can introduce flakiness (race conditions)
• Code splitting can break source maps (harder debugging)

Kibana-Specific Gotchas

• Bootstrap cache must be invalidated when dependencies change
• Scout snapshots don't work with ES cross-cluster search
• Webpack cache breaks when plugins change (must clear)
• Plugin lazy loading requires careful dependency management

Success Criteria

Optimization is successful when:

1. Metrics improve by ≥20% (anything less may not be worth effort)
2. No regressions in functionality or test coverage
3. ROI breakeven within 2 weeks (implementation time recovered)
4. Monitoring in place to prevent future regression
5. Documentation updated so others can apply same patterns

References

• Webpack Bundle Analyzer: https://github.com/webpack-contrib/webpack-bundle-analyzer
• Jest Performance: https://jestjs.io/docs/troubleshooting#tests-are-slow-when-run-in-debug-mode
• Scout Config: https://github.com/elastic/kibana/blob/main/packages/kbn-scout/README.md
• Buildkite Analytics: https://buildkite.com/docs/analytics
• Kibana Performance Docs: https://github.com/elastic/kibana/blob/main/PERFORMANCE.md