ds-conduct

ds-conduct : the crew's conductor

When this applies

Starting fresh: user has data and a fuzzy request but no clear workflow yet. The conductor figures out the right crew and sequence before any analysis runs.

Typical triggers:

• "Where do I start with this data?"
• "Orchestrate the analysis."
• "Look at my data and tell me what to do."
• "Set up the analysis for me."
• "I have a CSV — what should I do?"
• User uploads files but has not named a specific skill or method.

Skip this skill when the user already knows what they want (e.g. "run ds-spike on payments.csv to compute churn"). Call the named skill directly.

The cardinal rule

ds-conduct is a conductor, not an analyst. It never re-implements the skills it orchestrates. If it needs to know something from the data, it calls data-profile. If it needs intent clarified, it calls ds-clarify. If it needs analysis, it dispatches ds-star-plus or ds-spike. All substance lives in the specialist skills.

---

How to run it

Stage 1 — Peek (fast, read-only)

Inspect every data file structurally without running analysis: row count, column names + dtypes, null rates, unique counts for key columns, timestamp detection, candidate keys, file relationships (shared columns across files). Reuse the fast analyzer from ../data-profile/SKILL.md (description-only mode, not a full profile). Goal: learn enough to ask informed questions.

Produce a compact internal summary:

payments.csv: 2.8M rows | columns: payment_id (uuid, unique), merchant_id (int, 97% coverage),
created_at (datetime, range 2023-01–2024-12), amount_eur (float, 2% null), status (3 values)
merchants.json: 18K records | merchant_id (int, key), signup_date, country, tier, is_active (bool)
Shared key: merchant_id (both files)
Triggered patterns: timestamp, shared key, target-shaped column (is_active), internal report language

Many files (data-lake / N > ~100). When the peek surfaces too many files to load wholesale, do not route on filename or embedding guesswork. Apply the column-level retrieval protocol — ../ds-star-plus/references/retrieval.md Stage 3 (column-name overlap, low-cardinality value containment, join-key reachability, with the recall-biased keep rule) — to pick the relevant file-set from the digests you already produced, then route on that scoped set. A file dropped here is unrecoverable downstream, so keep anything strong on either embedding or structure.

Stage 2 — Grill (data-aware)

Using the trigger catalog (references/trigger_catalog.md), identify every pattern that fired in Stage 1. Formulate one targeted question per detected pattern, grounded in what was actually seen ("I see a created_at column — is this a time-series problem?" not "do you have time data?"). Extend with the clarification checklist from ../ds-clarify/references/clarify_checklist.md for any additional ambiguities.

Ask all triggered questions in a single batched message. Do not trickle questions one per round. Record every answer — unanswered items block plan assembly.

Stage 3 — Assemble plan

Fill in the workflow plan template (references/workflow_plan_template.md) with the answers from Stage 2. Determine:

• Which skills, in what order
• ds-spike N + personas, if ensemble selected
• Whether ds-model is needed (prediction task) — if so, its first step is the feasibility & leakage gate (../ds-model/references/feasibility_gate.md): estimate the achievable ceiling (forecastability for time-series) and scan for leakage before modeling
• The analysis-spec.md path that ds-clarify will write
• Explicit checkpoint points and backtrack conditions at each handoff

Fold in distilled rules (if a store exists). Read the rules store (../ds-memory/SKILL.md Mode 4 — ExpeL distillation; rules.jsonl) and keep any rule whose task-signature family overlaps this task. Let matches shape the plan and the questions you raise — e.g. a rule "confirm cancellation handling before summing revenue" becomes both a plan step and a Stage-2 question. Rules are advisory; they never override the user or the verifier.

Escalation to search (hard single-number tasks). When a task is a single high-stakes number that greedy solving keeps missing, route it to ../ds-search/SKILL.md (tree-search). If a search_experience.jsonl store exists, pass the matching prior experience as the seed (its Step 0) so the tree starts from past wins and avoids recorded dead-ends — search multiplies cost, so seeding it well is what keeps the escalation affordable.

Present the filled plan to the user in plain language. Do not begin execution until the user approves.

Stage 4 — Confirm → Execute with checkpoints

Show the assembled plan. Wait for explicit user approval. On approval, execute skill by skill, pausing at each checkpoint to surface the output before proceeding.

At each checkpoint:

1. Show the output from the completed skill (profile summary, spec contents, or analysis result).
2. Ask "Continue to [next step]?" or surface any anomaly that warrants re-asking.
3. If a checkpoint output reveals a problem — bad data quality, wrong join, ambiguous spec — backtrack to the appropriate stage (see backtrack table in the workflow template) rather than pressing forward with a flawed foundation.

---

Output

A conducted analysis: profile → spec → analysis result (consensus + minority report if spike was used), plus a run manifest listing:

• Each skill invoked (in order)
• The analysis-spec.md path
• Any assumptions recorded during clarification
• Checkpoint decisions and any backtrack events

---

What this cannot guarantee

This skill is markdown driving a model across a long context. No process can enforce that the verifier, routing, and oscillation-handling steps actually ran as specified.

Mitigation: emit a structured runlog.json with these fields after each run:

• rounds (int): number of loop iterations
• verifier_verdicts (list): [{"round": N, "score": 1-4}, ...] for each round
• oscillated (bool): whether oscillation was detected
• subrun_cost_usd (list): cost per sub-run (ds-spike only; empty list for single-solver skills)

Validate it:

python3 ../ds-spike/scripts/runlog.py runlog.json

If it reports errors, the loop was not followed; do not present the answer as verified. This makes deviations detectable after the fact, not impossible.

Quick reference

Resource	Path
Trigger catalog	references/trigger_catalog.md
Workflow plan template	references/workflow_plan_template.md
Evidence / theory	references/evidence.md
Clarification checklist	../ds-clarify/references/clarify_checklist.md
Spec template	../ds-clarify/references/spec_template.md

Pairs with: data-profile · ds-clarify · ds-star-plus · ds-spike · ds-model