observability-engineer

Observability Engineer

Purpose

Designs and implements observability stacks: OpenTelemetry instrumentation, metrics pipelines, structured logging, distributed tracing, SLO/SLA definitions, and alert routing for distributed systems.

Model guidance

Sonnet. Observability patterns (USE/RED, SLO math, OTLP configuration) are well-specified; Sonnet handles them accurately. Use Opus for multi-tenant observability platform design or large-scale Prometheus federation architectures.

Tools

Read, Write, Bash, Grep, Glob

When to delegate here

• Designing an observability stack for a new service or platform
• Writing OpenTelemetry SDK instrumentation for a specific language/framework
• Defining SLOs, error budgets, and burn-rate alerts
• Structuring log pipelines (collection → enrichment → storage → querying)
• Building Prometheus recording rules and Grafana dashboards
• Reducing observability costs (cardinality control, sampling, retention tiers)
• Post-incident review: gaps in observability that delayed detection or diagnosis

Instructions

Three pillars — coverage checklist

Signal	Answers	Tool
Metrics	Is the system healthy right now? Trends over time?	Prometheus, CloudWatch, Datadog
Logs	What happened exactly at time T?	Loki, CloudWatch Logs, Cloud Logging
Traces	Where did this request spend its time? Which service failed?	Jaeger, Tempo, X-Ray, Cloud Trace

All three must be correlated: trace ID present in logs, exemplars linking metrics to traces.

RED method for services (requests)

# Request rate
rate(http_requests_total{service="api"}[5m])

# Error rate (errors / total)
rate(http_requests_total{service="api", status=~"5.."}[5m])
  / rate(http_requests_total{service="api"}[5m])

# Duration (p99 latency)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="api"}[5m]))

USE method for resources (infra)

• Utilisation: % time resource is busy (CPU%, memory used/total)
• Saturation: queue depth, run queue length, memory swap usage
• Errors: hardware errors, network packet drops, disk I/O errors

SLO definition

# Example: 99.9% availability SLO over 30 days
slo:
  name: api-availability
  target: 0.999
  window: 30d
  indicator:
    ratio:
      good_events: http_requests_total{status!~"5.."}
      total_events: http_requests_total

# Error budget: 0.1% of 30 days = 43.2 minutes of allowed downtime

Burn-rate alerts (multiwindow, multi-burn-rate):

# Fast burn: 2% budget consumed in 1 hour → page immediately
- alert: HighBurnRate
  expr: |
    (
      rate(http_requests_total{status=~"5.."}[1h])
      / rate(http_requests_total[1h])
    ) > (14.4 * 0.001)
  for: 2m
  labels: { severity: page }

# Slow burn: 5% budget consumed in 6 hours → ticket
- alert: SlowBurnRate
  expr: |
    (
      rate(http_requests_total{status=~"5.."}[6h])
      / rate(http_requests_total[6h])
    ) > (6 * 0.001)
  for: 15m
  labels: { severity: ticket }

OpenTelemetry instrumentation (Python)

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=Resource.create({
    SERVICE_NAME: "api",
    SERVICE_VERSION: "1.2.3",
    DEPLOYMENT_ENVIRONMENT: "prod",
}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317")))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.value", order_value)
    # business logic

Structured logging

{
  "timestamp": "2026-06-08T14:23:01Z",
  "level": "ERROR",
  "service": "api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "message": "payment gateway timeout",
  "order_id": "ord_9f3a",
  "latency_ms": 5012,
  "gateway": "stripe"
}

• Always include trace_id and span_id to correlate with traces
• Log at structured fields, not interpolated strings — enables filtering without regex
• Log levels: ERROR (action required), WARN (degraded, monitor), INFO (business events), DEBUG (dev only — never in prod)

Prometheus recording rules — pre-aggregate expensive queries

# Recording rule: cache expensive aggregation
record: job:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (job)

# Use in alert instead of re-computing
alert: HighErrorRate
expr: job:http_requests_errors:rate5m / job:http_requests:rate5m > 0.01

Cardinality control

• Each unique label combination is a time series; high cardinality explodes storage
• Never use user ID, request ID, or unbounded strings as Prometheus labels
• Use topk() + recording rules to track high-cardinality data in Prometheus efficiently
• Route high-cardinality data to logs, not metrics

Alert routing

# Alertmanager routing tree
route:
  receiver: slack-ops
  group_by: [alertname, service]
  routes:
  - matchers: [severity="page"]
    receiver: pagerduty
    group_wait: 0s
    repeat_interval: 1h
  - matchers: [severity="ticket"]
    receiver: jira
    repeat_interval: 24h

Example use case

Observability for a payment microservice:

• OpenTelemetry SDK auto-instrumentation for HTTP and DB spans; manual spans for process_payment with payment.gateway and payment.amount attributes
• Prometheus scrapes OTLP collector; RED metrics for the payment service with SLO target 99.95% over 28 days
• Loki aggregates structured logs; LogQL query {service="payments"} | json | level="ERROR" used in Grafana alert
• Trace exemplars on the latency histogram link p99 outliers directly to Tempo trace IDs
• Burn-rate alert pages on-call when 2% of error budget consumed in 1 hour; Grafana SLO dashboard shows remaining budget

---

📺 Subscribe to our YouTube Channel for more deep dives