langfuse-monitoring

Langfuse Monitoring

This skill covers operating the data once it's in Langfuse: visualizing it (dashboards), extracting it (metrics API), analyzing eval scores (score analytics), and alerting. It does not cover emitting the data (that's instrumentation — vendored langfuse skill) or designing the scores (that's the langfuse-evaluation skill).

Operating principles

1. Distill judgment, fetch facts. This skill carries what to monitor and how to structure it. For exact UI steps and API schemas, fetch live docs by appending .md to the page URL (e.g. https://langfuse.com/docs/metrics/features/metrics-api.md).
2. You can only monitor what you instrumented. Cost, latency, userId, tags, release/version must be on the traces to slice by them. If a needed dimension is missing, the fix is upstream (instrumentation → vendored langfuse skill), not here.

Workflow

1. Frame the monitoring goal

Identify the metric family — cost, latency, quality, or volume — and the dimensions to slice by (trace name/feature, user, model, tags, release). See references/dashboards.md for the metric/dimension model.

2. Pick the surface

• Visual, shareable, exploratory → dashboards. Start with curated (Latency/Cost/Usage), then build custom widgets. references/dashboards.md.
• Programmatic / scheduled / billing / embed elsewhere → references/metrics-api.md (use v2).
• Eval scores specifically (distributions, trends, judge-vs-human agreement) → references/score-analytics.md.

3. Build the standard dashboards

For a new production app, stand up the three durable dashboards (production health, cost optimization, quality/UX) from references/dashboards.md.

4. Set up alerting

Disambiguate first: Spend Alerts = your Langfuse Cloud bill, not app cost. For app-level cost/latency/quality alerts, build a Metrics-API-driven check. See references/alerting.md.

Bundled resources

• references/dashboards.md — metrics & dimensions, curated vs custom dashboards, the widget model, and the three standard dashboards to build (health / cost / quality).
• references/metrics-api.md — programmatic metrics: use v2, the query model (view/metrics/dimensions/filters/timeDimension), and v1→v2 migration gotchas.
• references/score-analytics.md — zero-config eval-score analysis: distributions, trends, and judge-vs-human agreement metrics (MAE/RMSE, Cohen's Kappa/F1).
• references/alerting.md — Spend Alerts (Cloud billing) vs application-level alerting (Metrics API + your own check); how to alert on cost/latency/quality.

Hand-off map

Need	Where
Dashboards, metrics extraction, score analytics, alerting	this skill
Emitting cost/latency/userId/tags on traces (instrumentation)	vendored langfuse skill
Designing the scores/evaluators being monitored	langfuse-evaluation skill
Onboarding-time spend-alert setup / production-readiness checklist	langfuse-setup skill
Formal judge calibration (vs lightweight score analytics)	vendored langfuse skill judge-calibration.md
Exact dashboard UI / metrics API schema	live docs (.md-append)