okareo

Configure or review Okareo production monitoring — checks on live traffic, baselines, drift, and alerts, for a text or voice agent.

Guided onboarding to Okareo — verify the connection, explain the model, and walk through a first simulation or evaluation. For users new to Okareo or asking how to get started.

Build an Okareo scenario set — either composed synthetically from scratch or captured from production traces, logs, and incidents.

Start an Okareo simulation against an agent — simulated multi-turn conversations against a text agent, or simulated calls against a voice agent.

Stress-test an agent or chatbot before production by running simulated multi-turn conversations against it with Okareo. Use this skill whenever the user wants to simulate users, run synthetic conversations, red-team an agent, probe for failure modes, or check how an agent behaves across many personas — including requests like "simulate users talking to my agent", "how does the bot handle an angry customer", "find where the agent breaks", or "run conversations before we ship". Use it even when the user does not say "Okareo" but is clearly trying to exercise an agent with synthetic conversations.

Design, run, and analyze evaluations of an LLM app's generated output with Okareo — scoring a model or prompt against a scenario set with checks. Use this skill whenever the user wants to test or benchmark output quality, catch regressions, or compare prompt or model versions — including requests like "evaluate my model", "benchmark output quality", "did this prompt change make things worse", "compare GPT-4 and Claude on our test set", or "run our evals". Use it even when the user does not say "Okareo" but is clearly trying to score an LLM's output against expected results. For generating synthetic conversations against an agent use `agent-simulation`; to build the scenario set first use `scenario-design` or `scenario-from-traces`.

Set up and interpret production monitoring of an LLM app, agent, or voice agent with Okareo — checks running on live traffic, quality baselines, and alerts on regressions or drift. Use this skill whenever the user wants to monitor production, watch live quality, catch regressions in the wild, set up alerts, or investigate a drift in behavior — including requests like "monitor my agent in production", "watch my voice agent's live calls", "alert me when quality drops", "why did responses get worse this week", or "track our model in the wild". Use it even when the user does not say "Okareo" but is clearly trying to observe a live LLM or voice system.

Onboard a developer to Okareo — verify the MCP connection works, explain the Target / Driver / Scenario model, and walk through a first simulation or evaluation. Use this skill whenever the user is new to Okareo, asks how to get started or set up, or asks what Okareo can do — including requests like "get me started with Okareo", "how do I set up Okareo", "what can Okareo do", "I want to try Okareo", or "help me run my first test". Use it even when the user does not say "Okareo" but is clearly new to AI evaluation and trying to test an agent or model for the first time. Not for a user who already has a configured target and a specific task — route straight to the matching skill instead.

Design a synthetic test scenario set from scratch with Okareo — diverse, edge-case inputs covering real workflows, user roles, and stress conditions. Use this skill whenever the user wants to create a test set from scratch, expand coverage, or generate scenarios — including requests like "create a test set for my agent", "generate scenarios to test this", "we need more test coverage", "build evals from scratch", or "what cases should I test". Use it even when the user does not say "Okareo" but is clearly trying to build synthetic test cases for an LLM app or agent. Not for converting production traffic into a test set (use `scenario-from-traces`) and not for running the set (use `evaluation`).