Harness Chaos

Harness Chaos

Chaos engineering, fault injection, and resilience validation. Systematically introduces failures to verify that systems degrade gracefully, recover automatically, and maintain availability under real-world fault conditions.

When to Use

• Validating resilience of a service before a production launch or major milestone
• Testing circuit breakers, retries, timeouts, and fallback mechanisms under failure
• Preparing for a game day exercise with defined failure scenarios
• NOT when the service has no resilience mechanisms implemented (implement them first with harness-resilience)
• NOT when testing application logic correctness (use harness-tdd or harness-integration-test instead)
• NOT when performing load or performance testing (use harness-load-testing instead)

Process

Phase 1: PLAN -- Define Failure Modes and Steady-State Hypotheses

1. Map the system architecture. Identify:

   • Services and their dependencies (databases, caches, message queues, third-party APIs)
   • Network boundaries (service-to-service calls, DNS resolution, load balancers)
   • Infrastructure components (Kubernetes pods, Docker containers, cloud services)
   • Data stores and their replication topology

2. Define steady-state behavior. Establish measurable indicators of normal operation:

   • Response time P50 and P99 (e.g., P99 < 500ms)
   • Error rate (e.g., < 0.1% 5xx responses)
   • Throughput (e.g., > 100 requests/second)
   • Business metrics (e.g., orders processed per minute, messages delivered)

3. Enumerate failure modes. For each dependency, define what can go wrong:

   • Network failures: latency injection, packet loss, DNS resolution failure, connection reset
   • Service failures: dependency returns 500, dependency is unreachable, dependency responds slowly
   • Infrastructure failures: pod crash, node failure, disk full, memory exhaustion
   • Data failures: database connection pool exhaustion, replication lag, cache eviction storm

4. Scope the blast radius. For each experiment, define:

   • Target: which specific service or instance is affected
   • Duration: how long the fault persists (start small: 30 seconds to 2 minutes)
   • Magnitude: what percentage of traffic or instances are affected
   • Abort conditions: when to immediately stop the experiment (e.g., customer-visible error rate > 5%)

5. Prioritize experiments by risk and value. Start with:

   • High likelihood, high impact failures (database connection loss, primary cache failure)
   • Failures with existing but untested resilience mechanisms (circuit breakers, retries)
   • Failures that have caused production incidents historically

6. Document the experiment plan. For each experiment, write:

   • Hypothesis: "When [fault] is injected into [target], the system [expected behavior]"
   • Steady-state metric: what to monitor
   • Abort criteria: when to terminate the experiment
   • Rollback procedure: how to immediately restore normal operation

Phase 2: INJECT -- Apply Fault Injection

1. Select the chaos tooling. Based on the infrastructure:

   • Chaos Toolkit: framework-agnostic, declarative experiment definitions (JSON/YAML)
   • Gremlin: SaaS platform for enterprise chaos engineering
   • Litmus: Kubernetes-native chaos engineering
   • Toxiproxy: network-level fault injection proxy
   • tc (traffic control): Linux kernel network delay/loss injection
   • Custom middleware: application-level fault injection via feature flags or interceptors

2. Configure the experiment. Write the experiment definition:

   • Steady-state probe: how to verify normal operation before and after
   • Fault action: what fault to inject (latency, error, crash, resource exhaustion)
   • Rollback action: how to undo the fault if the experiment must abort
   • Duration and magnitude parameters

3. Verify the pre-experiment steady state. Before injecting any fault:

   • Run the steady-state probe and confirm it passes
   • Record baseline metrics (latency, error rate, throughput)
   • Confirm monitoring and alerting are active and visible to the experiment operator

4. Inject the fault. Execute the experiment:

   • Start with the smallest blast radius (single instance, short duration)
   • Monitor real-time metrics during injection
   • Be prepared to abort immediately if abort criteria are met

5. Verify the abort mechanism works. Before running experiments with larger blast radius:

   • Test that the rollback action successfully removes the injected fault
   • Confirm the system returns to steady state after rollback
   • Measure recovery time

Phase 3: OBSERVE -- Monitor System Behavior Under Fault

1. Collect metrics during the experiment. Capture:

   • Response latency distribution (P50, P95, P99) for the affected service and its consumers
   • Error rates (HTTP status codes, exception counts, queue dead letters)
   • Circuit breaker state transitions (closed, open, half-open)
   • Retry counts and backoff behavior
   • Resource utilization (CPU, memory, connections, threads)

2. Verify the steady-state hypothesis. Compare observed metrics against the hypothesis:

   • Hypothesis holds: the system degraded gracefully as expected. Record as a successful experiment.
   • Hypothesis violated: the system behaved worse than expected. This is a finding. Record the specific deviation.

3. Check for cascading failures. Monitor downstream services:

   • Did the fault in service A cause service B to fail?
   • Did retry storms amplify the failure?
   • Did the load balancer route traffic away from the faulty instance?
   • Did the circuit breaker open before the caller's timeout?

4. Record the timeline. Document:

   • T+0: fault injected
   • T+N: first detection by monitoring/alerting
   • T+N: circuit breaker opens (if applicable)
   • T+N: system reaches degraded but stable state
   • T+N: fault removed
   • T+N: system returns to full steady state (recovery time)

5. Terminate the experiment. Remove the injected fault and verify:

   • The system returns to steady state within the expected recovery time
   • No data loss or corruption occurred during the experiment
   • All queued/retried operations complete successfully

Phase 4: IMPROVE -- Analyze Findings and Strengthen Resilience

1. Classify findings. For each experiment:

   • Passed: system behavior matched the hypothesis. No action needed.
   • Finding -- minor: system recovered but slower than expected. Add to backlog.
   • Finding -- major: system experienced partial outage or data issue. Immediate action required.
   • Finding -- critical: cascading failure or data loss. Stop experiments and prioritize fix.

2. Recommend resilience improvements. For each finding:

   • Missing circuit breaker: implement circuit breaker with appropriate thresholds
   • Insufficient timeout: reduce timeout to prevent thread pool exhaustion
   • No fallback: implement graceful degradation (cached response, default value, feature flag)
   • Retry storm: implement exponential backoff with jitter and retry budget
   • No health check: add readiness/liveness probes that detect the specific failure mode

3. Update runbooks and incident response documentation. For each experiment:

   • Add the failure mode to the incident response playbook
   • Document the detection time, recovery procedure, and expected recovery time
   • Update alerting thresholds if monitoring did not detect the failure promptly

4. Plan follow-up experiments. Based on findings:

   • Re-run failed experiments after resilience improvements are implemented
   • Increase blast radius for successful experiments (from single instance to multiple instances)
   • Combine faults (e.g., database slowdown plus cache failure simultaneously)

5. Run harness validate. Confirm the project passes all harness checks after any code changes made for resilience improvements.

6. Generate an experiment report. Summarize:

   • Experiments run, hypotheses tested, and pass/fail outcomes
   • Findings with severity classification
   • Resilience improvements recommended or implemented
   • Recovery time measurements
   • Recommended next experiments

Graph Refresh

If a knowledge graph exists at .harness/graph/, refresh it after code changes to keep graph queries accurate:

harness scan [path]

Harness Integration

• harness validate -- Run in IMPROVE phase after resilience changes are implemented. Confirms project health.
• harness check-deps -- Run after INJECT phase setup to verify chaos tooling dependencies do not leak into production bundles.
• emit_interaction -- Used at checkpoints to present experiment plans for human approval before fault injection, and to present findings for prioritization.
• Grep -- Used in PLAN phase to find circuit breaker configurations, retry policies, timeout settings, and fallback implementations.
• Glob -- Used to locate infrastructure configuration files, Kubernetes manifests, and Docker Compose definitions.

Success Criteria

• Every critical dependency has at least one chaos experiment testing its failure mode
• Steady-state hypotheses are defined with measurable metrics before experiments run
• The system degrades gracefully under every tested fault (no cascading failures, no data loss)
• Recovery time after fault removal is measured and meets the defined SLA
• Findings are documented with severity, root cause, and recommended fix
• Abort mechanisms are tested and confirmed functional before expanding blast radius
• harness validate passes after resilience improvements

Examples

Example: Chaos Toolkit Experiment for Database Latency

PLAN -- Experiment definition:

{
  "title": "Database latency does not cause cascading timeout failures",
  "description": "Inject 2-second latency on PostgreSQL connections and verify the order service responds within 5 seconds using cached data",
  "steady-state-hypothesis": {
    "title": "Order service responds within SLA",
    "probes": [
      {
        "type": "probe",
        "name": "order-api-responds",
        "tolerance": true,
        "provider": {
          "type": "http",
          "url": "http://localhost:3000/api/orders/health",
          "timeout": 5
        }
      }
    ]
  },
  "method": [
    {
      "type": "action",
      "name": "inject-db-latency",
      "provider": {
        "type": "process",
        "path": "toxiproxy-cli",
        "arguments": "toxic add -t latency -a latency=2000 postgresql"
      }
    },
    {
      "type": "probe",
      "name": "check-order-response-time",
      "provider": {
        "type": "http",
        "url": "http://localhost:3000/api/orders?limit=10",
        "timeout": 5
      }
    }
  ],
  "rollbacks": [
    {
      "type": "action",
      "name": "remove-db-latency",
      "provider": {
        "type": "process",
        "path": "toxiproxy-cli",
        "arguments": "toxic remove -n inject-db-latency_latency_downstream postgresql"
      }
    }
  ]
}

Example: Litmus Chaos for Kubernetes Pod Failure

INJECT -- Pod kill experiment:

# litmus/pod-kill-experiment.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-pod-kill
  namespace: staging
spec:
  appinfo:
    appns: staging
    applabel: app=order-service
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'
            - name: PODS_AFFECTED_PERC
              value: '50'

OBSERVE -- Expected behavior timeline:

T+0s:   Fault injected - 50% of order-service pods killed
T+3s:   Kubernetes detects pod failure, starts replacement pods
T+5s:   Load balancer routes traffic to surviving pods
T+8s:   Response latency increases from 50ms to 200ms (surviving pods absorb load)
T+15s:  Replacement pods pass readiness probe, rejoin the pool
T+20s:  Latency returns to baseline (50ms)
T+60s:  Experiment ends

Result: PASSED - System maintained availability throughout.
        P99 latency spiked to 450ms (within 500ms SLA).
        Zero 5xx errors observed. No data loss.

Rationalizations to Reject

Rationalization	Reality
"Our circuit breakers are already tested in unit tests — we don't need chaos experiments"	Unit tests verify that circuit breaker code executes. Chaos experiments verify that the circuit breaker actually opens under real load, that the fallback produces an acceptable user-facing response, and that monitoring detects the transition. These are different things.
"We can't run chaos experiments in production — it's too risky"	Avoiding chaos experiments does not reduce risk — it defers the discovery of failure modes to real incidents. Chaos experiments in staging with defined abort criteria and short durations are lower risk than discovering failure modes at 2am during a real outage.
"The experiment passed in staging so we know it'll work in production"	Staging differences in traffic volume, data distribution, and infrastructure scale can mask failure modes. Staging experiments validate the mechanism; production experiments (with tightly scoped blast radius) validate the system under real conditions.
"We injected the fault and the system recovered — the experiment is done"	Recovery alone does not validate resilience. The experiment must also confirm: detection time (did monitoring catch it?), recovery time (did it meet the SLA?), and no data loss or corruption. A system that recovers after 10 minutes of data inconsistency has not passed.
"We have runbooks for these failure modes, so game days aren't necessary"	A runbook that has never been executed under pressure is a hypothesis, not a procedure. Game days reveal whether runbooks are complete, whether on-call engineers can execute them accurately under stress, and whether the estimated recovery times are realistic.

Gates

• No chaos experiments without abort criteria. Every experiment must define conditions under which it is immediately terminated. Running an experiment that you cannot stop is reckless, not engineering.
• No production experiments without staging validation. Run every experiment in staging first. Only after the experiment passes in staging and the team is confident in the abort mechanism should it be considered for production.
• No expanding blast radius without successful small-scope runs. Start with a single instance or a small percentage of traffic. Only increase scope after the smaller experiment passes and recovery is confirmed.
• No experiments during incidents or peak traffic. Chaos experiments must not be run when the system is already under stress or during known high-traffic periods. Schedule experiments during low-traffic windows.

Escalation

• When an experiment causes unexpected data loss or corruption: Immediately abort, restore from backup, and halt all chaos experiments. Conduct a post-mortem to understand why the data protection mechanisms failed. Do not resume experiments until the data safety gap is addressed.
• When monitoring does not detect the injected fault: This is a finding -- the monitoring is inadequate. Do not interpret "no alerts" as "the system handled it well." Escalate to the observability team to add detection for the specific failure mode before re-running the experiment.
• When the team is reluctant to run chaos experiments: Start with the least risky experiment in a non-production environment. Use the results to demonstrate value. Chaos engineering requires organizational buy-in; do not force it.
• When cascading failures are discovered: This is a critical finding. The service mesh or dependency chain lacks isolation. Escalate immediately with a recommendation for circuit breakers, bulkheads, or service mesh fault injection policies.