state-consistency-investigator

State Consistency Investigator

Role: Diagnose state consistency failures across services: stale reads, replication lag, event ordering, cache-vs-source divergence. Dispatched by: debug-conductor — never invoked directly by user. Evidence type: StateEvidence (see skills/_shared/evidence-schema.md)

When to Use

Dispatched by debug-conductor when the incident involves data inconsistency — stale reads, replication lag, cache staleness, event reordering, or cross-service state divergence.

Investigation Protocol

Step 1: Identify the Consistency Boundary

1. Which data entities are affected? (user profiles, order records, account settings)
2. Which services read this data? (source of truth vs. replicas vs. caches)
3. What consistency model is expected? (strong, eventual, read-after-write)
4. What propagation paths exist? (DB replication, event bus, cache invalidation, API sync)

Step 2: Cross-Source Comparison

For each affected entity:

1. Query the source of truth (primary DB) — what's the current value?
2. Query each consumer (replica DB, cache, downstream service) — what do they see?
3. Record discrepancies with timestamps:

{
  "inconsistencies": [
    {
      "entity": "customer:C-1001:phone",
      "sourceA": "primary-db",
      "sourceB": "replica-db",
      "valueA": "+1-555-0199",
      "valueB": "+1-555-0100"
    }
  ]
}

Step 3: Replication Lag Assessment

1. Measure current replication lag (primary → replica)
2. Check historical lag: is it steady or spiking?
3. Compare lag SLA vs. actual:
   • Average within SLA but P99 outside? (intermittent issue)
   • Lag correlated with load patterns? (peak hours)
4. Check if lag explains the observed inconsistency window

Step 4: Event Ordering Audit

1. Inspect the event/message bus for the affected entity
2. Verify events were published in correct order
3. Verify events were consumed in correct order
4. Check for:
   • Duplicate events (at-least-once delivery)
   • Lost events (check consumer acknowledgment)
   • Reordered events (partition key mismatch, consumer parallelism)

Step 5: Cache Behavior Analysis

1. Check cache TTL vs. invalidation timing
2. Trace the invalidation flow: event received → cache key deleted → next read
3. Identify: does the cache-miss read go to source of truth or replica?
   • Critical pattern: Cache invalidation works, but re-fill reads stale replica → re-caches stale data
4. Check cache hit rate: sudden drops may indicate invalidation storms

Step 6: Produce Evidence

{
  "inconsistencies": [ /* cross-source comparison results */ ],
  "replicationLag": { "primaryToReplica": 4500, "measured": "ISO-8601" },
  "eventOrdering": [
    { "expected": ["update", "invalidate", "read"], "actual": ["update", "read", "invalidate"], "divergencePoint": 1 }
  ],
  "staleReads": [
    { "query": "SELECT phone FROM profiles WHERE id=C-1001", "staleValue": "+1-555-0100", "currentValue": "+1-555-0199", "lagMs": 4500 }
  ]
}

Stop Conditions

• Inconsistency source identified (replication lag, cache re-fill, event ordering)
• 3 cross-source comparison checks completed
• Token budget exhausted
• Wall-clock limit (5 minutes)

Escalation Conditions

• Multiple unrelated inconsistencies with no common cause (may indicate data corruption)
• Replication lag within SLA but inconsistency persists (not a lag issue — something else)
• Event bus shows correct delivery but consumer state is wrong (consumer bug)

Common Patterns This Investigator Detects

Pattern	Evidence Shape
Stale replica read	Primary has new value, replica has old; lag > consistency window
Cache re-fill from replica	Cache invalidated correctly, but refill reads stale replica
Event reordering	Events consumed in different order than published
Invalidation race	Read arrives between invalidation and re-fill → cache miss → stale read
Dual-write inconsistency	Two services write same entity independently; values diverge
Eventual consistency window violation	Consumer reads during propagation delay

Failure Modes

Mode	Symptom	Recovery
Timing sensitivity	Bug only appears under specific timing	Test with artificial delays
Stale read	Reading from replica during lag	Read from primary for verification
Event ordering	Assuming ordered delivery	Check for out-of-order events explicitly