observability-best-practices

Python observability best practices for async-first services using OpenTelemetry — covers SDK bootstrap with `setup_telemetry(component)` and an `OTEL_EXPORTER_OTLP_ENDPOINT` off-switch contract, single-shot idempotency guard, per-component `service.name` resource attribute, OTLP HTTP/protobuf exporters with `BatchSpanProcessor` / `BatchLogRecordProcessor` / `PeriodicExportingMetricReader`, auto-instrumentation (`GrpcAioInstrumentorServer/Client`, `SQLAlchemyInstrumentor`, `FastAPIInstrumentor`, `LoggingInstrumentor`), the critical `set_logging_format=False` + custom `_inject_trace_context` log hook + `_CleanLoggingHandler` pattern that bridges OTel SDK and structlog, two-tier log field taxonomy (Tier-1 structured `key=value` indexed in Loki + Tier-2 raw payload after `|` separator), `<service>.<business_key>` span attribute prefix for cross-service Tempo search, sensitive-field redaction via copy-and-replace at the proto/JSON serialization boundary, status taxonomy mapping, metrics catalogue conventions with strict cardinality control (no `transaction_id` / `user_id` labels), and tail-based sampling design. Use this skill whenever the user is writing, reviewing, debugging, or designing OpenTelemetry / observability code in a Python backend — including any work involving imports from `opentelemetry` / `opentelemetry.instrumentation.*` / `opentelemetry.sdk.*`, `setup_telemetry()` calls, `tracer.start_as_current_span()`, `meter.create_counter()` / `create_histogram()`, `LoggingInstrumentor`, structlog with trace_id binding, span attribute setting, OTLP exporter configuration, Loki / Tempo / Grafana / Mimir integration, distributed tracing across gRPC services, log correlation via `trace_id` / `span_id`, sensitive-field redaction in logs, metric cardinality decisions, alert rule design, or Sentry-with-OTel integration. Trigger on any review of `telemetry.py`, `*_logger.py`, log-decorator modules, or Grafana dashboard / Prometheus alert YAML in a Python project. Do NOT use for proprietary observability SDKs without OTel (raw New Relic / Datadog / Honeycomb agents), browser/frontend tracing, log analysis questions unrelated to instrumentation, or generic "what's a span" intro questions.