claude-neam-dio

Claude Neam DIO Skill

The Intelligent Data Organization (IDO) is Neam's framework for autonomous data lifecycle management. It replaces manual handoffs between data teams with 14 specialist agents orchestrated by a master intelligence — the Data Intelligent Orchestrator (DIO). This skill teaches Claude to write complete DIO programs in Neam.

When to Activate

• User builds data pipelines, warehouses, or ML systems using Neam's DIO agents
• User asks about the 14 specialist agents (Data, ETL, Migration, DataOps, Governance, Modeling, Analyst, DataScientist, Causal, MLOps, Data-BA, DataTest, Deployment, Forge)
• User configures the DIO orchestrator (auto/config/hybrid modes)
• User writes agent_registry, infrastructure_profile, agent_md, or raci_matrix declarations
• User builds multi-agent data programs with RACI accountability
• User works with DataSims evaluation, ablation studies, or coordination modes
• User asks about spec-driven development vs vibe coding vs agentic approaches for data projects

Prerequisite: This skill extends the claude-neam-programming skill. Activate that skill first for core Neam syntax (variables, types, functions, OOP, RAG, guards, deployment). This skill focuses exclusively on the DIO ecosystem.

---

The Problem DIO Solves

85% of ML projects never reach production (Gartner). The root cause is not technology — it is handoff failures between teams. A typical data project has 5 handoff points (BA-to-DE, DE-to-DS, DS-to-MLOps, each-to-QA, QA-to-Deploy), each with ~18% coordination overhead.

DIO eliminates these handoffs by replacing the traditional org chart with 14 agents + 1 orchestrator operating under spec-driven development: requirements are machine-readable, handoffs are contracts, and quality gates are enforced automatically.

---

Four-Layer Architecture

Layer 4: Orchestration         DIO (master intelligence)
Layer 3: Analytical Intelligence   Data-BA, DataScientist, Causal, DataTest, MLOps
Layer 2: Platform Intelligence     DataOps, Governance, Modeling, Analyst
Layer 1: Data Infrastructure       Data Agent, ETL Agent, Migration Agent
Cross-cutting: Infrastructure Profiles, Agent.MD, RACI, Security

Communication between layers uses:

• Pub/Sub messaging — async, FIFO, guaranteed delivery
• Artifact registry — versioned handoffs between agents
• Event bus — real-time status propagation

---

Quick Start

Your First DIO Orchestration (Verified)

budget B { cost: 500.00, tokens: 2000000 }
infrastructure_profile Infra { data_warehouse: { platform: "postgres" } }

dio agent MyDIO {
    mode: "config",
    task: "Predict customer churn",
    infrastructure: Infra,
    provider: "openai",
    model: "gpt-4o",
    budget: B
}

print(dio_solve(MyDIO, "full_system"));

Compile and Run

neamc program.neam -o program.neamb
neam program.neamb

---

The 14 Specialist Agents

Each agent has a unique keyword, role, required fields, and native functions.

Agent Quick Reference

#	Keyword	Role	Version	Layer
1	data agent	Data Engineer — Ingestion	v0.9.0	Infrastructure
2	etl agent	Data Engineer — Transformation	v0.9.1	Infrastructure
3	migration agent	Migration Specialist	v0.9.2	Infrastructure
4	dataops agent	Platform Operator (SRE for Data)	v0.9.3	Platform
5	governance agent	Compliance Officer	v0.9.4	Platform
6	modeling agent	Data Architect	v0.9.5	Platform
7	analyst agent	Data Analyst	v0.9.6	Platform
8	datascientist agent	Data Scientist	v0.9.8	Analytical
9	causal agent	Causal Investigator	v0.9.8.1	Analytical
10	mlops agent	Production Guardian	v0.9.8.2	Analytical
11	databa agent	Requirements Analyst	v0.9.8.3	Analytical
12	datatest agent	Quality Critic	v0.9.8.4	Analytical
13	deploy_config	Release Manager	v0.9.7	Infrastructure
14	dio agent	Master Orchestrator	v0.9.9	Orchestration

Agent Trait System

Agents have composable traits that determine their capabilities:

Trait	Agents
DataProducer	Data Agent, ETL Agent, DataScientist
DataConsumer	ETL Agent, DataScientist, Analyst, Causal
CausalReasoner	Causal Agent (general-purpose — any agent can invoke)
QualityGatekeeper	DataTest Agent

---

Layer 1: Data Infrastructure Agents

Data Agent (v0.9.0)

Purpose: Foundation data lifecycle — sources, sinks, schemas, quality gates, compute routing, governance, lineage, catalog.

Keyword: data agent

Declarations

// Schema with typed fields and annotations
schema CustomerSchema {
    id: string @primary_key,
    name: string @not_null,
    email: string @pii,
    revenue: float @range(0, 1000000),
    created: datetime
}

// Source connector
source CRM {
    type: "postgres",
    connection: "pg://localhost/crm",
    schema: CustomerSchema
}

// Sink connector
sink Warehouse {
    type: "snowflake",
    connection: "sf://account",
    write_mode: "upsert"
}

// Quality gate
quality DataQuality {
    completeness: 0.95,
    freshness: "1h",
    on_violation: "gate"
}

// Compute routing
compute Local { engine: "local" }

// Governance
governance AccessControl {
    access: {
        model: "rbac",
        roles: { "analyst": { read: ["public", "internal"], write: ["public"] } }
    }
}

// Catalog registration
catalog DataCatalog { engine: "datahub", discovery: true }

Minimal (Verified)

source DB { type: "postgres", connection: "pg://localhost" }
sink Out { type: "s3", connection: "s3://bucket/" }

data agent Minimal {
    provider: "openai",
    model: "gpt-4o-mini",
    sources: [DB],
    sinks: [Out]
}

Full Declaration (Verified)

schema S { id: string @primary_key, val: float }
source DB { type: "postgres", connection: "pg://localhost", schema: S }
sink Out { type: "snowflake", connection: "sf://account", write_mode: "upsert" }
quality Q { completeness: 0.95, on_violation: "gate" }
compute C { engine: "local" }
governance G {
    access: {
        model: "rbac",
        roles: { "analyst": { read: ["public", "internal"], write: ["public"] } }
    }
}
catalog Cat { engine: "datahub", discovery: true }

data agent Full {
    provider: "openai",
    model: "gpt-4o",
    system: "Full featured agent.",
    temperature: 0.3,
    sources: [DB],
    sinks: [Out],
    schema: S,
    quality: Q,
    compute: { default: C },
    governance: G,
    catalog: Cat,
    lineage: true,
    role: "analyst",
    purpose: "Data pipeline",
    autonomy: "conditional"
}

Syntax Rules

Declaration	Required Fields	Notes
schema	field_name: type	Types: string, int, float, bool, datetime. Annotations: @primary_key, @not_null, @range(min,max), @pii
source	type, connection	Types: "postgres", "s3", "http", "kafka". Optional: schema, refresh
sink	type, connection	Optional: write_mode ("append", "upsert", "overwrite")
quality	at least one metric	completeness: 0.95, freshness: "1h", on_violation: "gate" (flat values, NOT nested objects)
compute	engine	Engines: "local", "spark", "snowflake", "bigquery". Optional: config: { ... }
governance	access block	Contains model + roles
catalog	engine	Engines: "datahub", "unity". Optional: discovery: true
data agent	provider, model, sources, sinks	schema is singular (not schemas). compute uses { default: Ref } syntax

---

ETL Agent (v0.9.1)

Purpose: SQL-first warehouse/lakehouse agent with dimensional modeling, semantic layer, NL2SQL, self-healing.

Keyword: etl agent

Minimal (Verified)

compute WH { engine: "snowflake" }
source S { type: "postgres", connection: "host=localhost" }

etl agent E {
    provider: "openai",
    model: "gpt-4o",
    sources: [S],
    warehouse: WH
}

With Mart and Semantic Layer (Verified)

compute WH { engine: "snowflake" }
source Raw { type: "postgres", connection: "pg://localhost" }

mart SalesMart {
    grain: "one row per order",
    model_type: "star",
    facts: { fact_orders: { measures: ["quantity", "revenue"] } },
    dimensions: { dim_customer: { scd_type: 2, business_key: "customer_id" } }
}

semantic SalesGlossary {
    entities: { customer: { table: "dim_customer", key: "customer_sk" } },
    metrics: { revenue: { formula: "SUM(revenue)", format: "currency" } }
}

etl agent WarehouseBuilder {
    provider: "openai",
    model: "gpt-4o",
    sources: [Raw],
    warehouse: WH,
    marts: [SalesMart],
    semantic: SalesGlossary
}

Syntax Rules

Field	Required	Notes
sources	Yes	Array of source references
warehouse	Yes	Reference to a compute declaration (the target warehouse)
marts	No	Array of mart references
semantic	No	Reference to a semantic declaration

---

Migration Agent (v0.9.2)

Purpose: Cross-platform data migration with wave planning, schema translation, reconciliation, zero-downtime cutover.

Keyword: migration agent

Minimal (Verified)

budget MigBudget { cost: 10.0 }

migration agent TestMig {
    provider: "openai",
    model: "gpt-4o-mini",
    source: OracleDB,
    target: SnowflakeDB,
    staging: S3Staging,
    budget: MigBudget,
    strategy: "re_platform"
}

Syntax Rules

Field	Required	Notes
source	Yes	Reference to source system
target	Yes	Reference to target system
staging	Yes	Reference to staging area
strategy	Yes	"re_platform", "re_architect", "lift_and_shift"
budget	Yes	Budget reference

Native Functions

Function	Purpose
migration_status(agent)	Status: "assess", "planning", "executing", "complete"
migration_plan(agent)	Generate migration plan
migration_execute_wave(agent)	Execute migration wave
migration_reconcile(agent)	3-level reconciliation (row count, hash, semantic)

---

Layer 2: Platform Intelligence Agents

DataOps Agent (v0.9.3)

Purpose: SRE for data — 6-stage pipeline (collect, detect, correlate, triage, auto-heal, report). FinOps integration.

Keyword: dataops agent

Verified Declaration

budget OpsBudget { cost: 30.00, tokens: 500000 }

platform SnowflakePlatform {
    type: "snowflake",
    connection: "snowflake://account.snowflakecomputing.com"
}

scheduler AirflowProd {
    type: "airflow",
    connection: "http://localhost:8080"
}

audit_table ETLAudit {
    connection: "pg://localhost/ops",
    table: "pipeline_runs",
    timestamp_column: "ts",
    status_column: "status"
}

incident_policy OpsPolicy {
    severity: {
        P1_CRITICAL: {
            conditions: ["SLA breach", "data loss"],
            response: "immediate"
        }
    }
}

dataops agent PipelineMonitor {
    provider: "openai",
    model: "gpt-4o",
    budget: OpsBudget,
    platforms: [SnowflakePlatform],
    schedulers: [AirflowProd],
    audit_tables: [ETLAudit],
    incident_policy: OpsPolicy
}

Syntax Rules

Field	Required	Notes
platforms	Yes (OPS-001)	Array of platform references
incident_policy	Yes (OPS-002)	Reference to incident_policy. Uses severity with named levels
schedulers	No	Array of scheduler references
audit_tables	No	Array of audit_table references

Native Functions

Function	Purpose
dataops_status(agent)	Status: "idle", "monitoring", "triaging"
dataops_detect_anomaly(agent)	Anomaly detection
dataops_auto_heal(agent)	Self-healing pipeline repair
dataops_sla_check(agent)	SLA compliance check

---

Governance Agent (v0.9.4)

Purpose: Data classification, RBAC/ABAC, lineage, compliance (GDPR/CCPA/DORA). Column-level lineage.

Keyword: governance agent

Verified Declaration

budget GovBudget { cost: 20.00, tokens: 300000 }

classification_policy DataClassification {
    levels: {
        RESTRICTED: { level: 4, controls: ["encryption_at_rest", "column_masking"] },
        CONFIDENTIAL: { level: 3, controls: ["masking"] },
        INTERNAL: { level: 2, controls: ["audit"] },
        PUBLIC: { level: 1, controls: ["none"] }
    }
}

governance agent DataGovernor {
    provider: "openai",
    model: "gpt-4o",
    system: "You are a data governance agent.",
    temperature: 0.3,
    budget: GovBudget,
    classification: DataClassification
}

Syntax Rules

Field	Required	Notes
classification	Yes	Reference to classification_policy (uses levels, NOT patterns)
budget	Yes	Budget reference

Native Functions

Function	Purpose
governance_status(agent)	Status: "idle", "scanning"
gov_classify(agent)	Auto-classify data columns
gov_check_access(agent)	RBAC/ABAC access check
gov_trace_lineage(agent)	Column-level lineage tracing

---

Modeling Agent (v0.9.5)

Purpose: Schema reverse-engineering, ER modeling, normalization analysis, dimensional design, impact analysis.

Keyword: modeling agent

Verified Declaration

schema_source TestSrc {
    type: "snowflake",
    connection: "sf://account"
}

modeling agent SchemaArchitect {
    provider: "openai",
    model: "gpt-4o-mini",
    sources: [TestSrc]
}

Related Declarations

entity Product {
    domain: "Sales",
    attributes: {
        product_id: { type: "identifier", primary_key: true }
    }
}

Syntax Rules

Field	Required	Notes
sources	Yes	Array of schema_source references

Native Functions

Function	Purpose
modeling_status(agent)	Status: "initialized"
model_reverse_engineer(agent)	Reverse-engineer existing schemas
model_generate_er(agent)	Generate ER diagrams
model_normalize(agent)	Normalization analysis

---

Analyst Agent (v0.9.6)

Purpose: NL-to-SQL across 9 dialects, domain transfer learning, governed execution, multi-format output, insight discovery.

Keyword: analyst agent

Verified Declaration

budget B { cost: 10.00, tokens: 100000 }

sql_connection DB {
    platform: "postgres",
    connection: "pg://localhost/test",
    database: "test"
}

analyst agent RevenueAnalyst {
    provider: "openai",
    model: "gpt-4o-mini",
    connections: [DB],
    budget: B
}

With Domain Context

domain_context SalesDomain {
    glossary: {
        revenue: "SUM(total)",
        churn: "No orders in 90 days"
    },
    default_schema: "dw"
}

analyst agent ContextAnalyst {
    provider: "openai",
    model: "gpt-4o-mini",
    connections: [DB],
    domain_context: SalesDomain,
    budget: B
}

Syntax Rules

Field	Required	Notes
connections	Yes	Array of sql_connection references
domain_context	No	Reference to domain_context
budget	Yes	Budget reference

Native Functions

Function	Purpose
analyst_status(agent)	Status: "initialized"
analyst_query(agent)	Natural language query
analyst_sql(agent)	Direct SQL execution
analyst_insight(agent)	Automated insight discovery
analyst_format(agent)	Multi-format output

---

Layer 3: Analytical Intelligence Agents

Data-BA Agent (v0.9.8.3)

Purpose: Requirements intelligence — always the FIRST agent invoked ("Day -1"). LLM-assisted elicitation, BRD generation, Given/When/Then acceptance criteria, traceability matrices.

Keyword: databa agent

Minimal (Verified)

budget B { cost: 50.00, tokens: 500000 }

databa agent MinBA {
    provider: "openai",
    model: "gpt-4o",
    temperature: 0.3,
    budget: B
}

With BRD Generator and Functional Spec

budget B { cost: 50.00, tokens: 500000 }

brd_generator ChurnBRD {
    project: { name: "Churn Prediction", id: "P-001" },
    objectives: { primary: "Reduce churn to 8%", kpis: ["churn_rate"] },
    scope: { in_scope: ["Enterprise"], out_of_scope: ["SMB"] }
}

functional_spec ChurnSpec {
    acceptance_criteria: [
        { id: "AC-001", given: "inactive customer", when: "scored", then: "prob > 0.7" }
    ]
}

databa agent ChurnBA {
    provider: "openai",
    model: "gpt-4o",
    budget: B,
    brd_generator: ChurnBRD,
    functional_spec: ChurnSpec
}

Native Functions

Function	Purpose
ba_status(agent)	Get agent status
ba_generate_brd(agent)	Generate Business Requirements Document
ba_generate_criteria(agent)	Generate Given/When/Then acceptance criteria
ba_traceability_matrix(agent)	Build source-to-target traceability

---

DataScientist Agent (v0.9.8)

Purpose: Problem framing, EDA, volume-aware compute routing, feature engineering, ML/AutoML, SHAP explainability.

Keyword: datascientist agent

Minimal (Verified)

budget B { cost: 50.00, tokens: 500000 }

datascientist agent MinDS {
    provider: "openai",
    model: "gpt-4o",
    temperature: 0.2,
    budget: B
}

With EDA and Volume Router (Verified)

budget B { cost: 50.00, tokens: 500000 }

eda_config EDA {
    univariate: { numeric: { statistics: ["mean", "std"] } }
}

volume_router VR {
    routing_rules: { local_memory: { condition: "n_rows < 100000" } }
}

datascientist agent DS {
    provider: "openai",
    model: "gpt-4o",
    eda_config: EDA,
    volume_router: VR,
    budget: B
}

Syntax Rules

Field	Required	Notes
budget	Yes	Budget reference
provider, model	Yes	LLM provider
temperature	No	Float 0.0-1.0
eda_config	No	Reference to eda_config
volume_router	No	Reference to volume_router
experiment	No	Reference to ml_experiment (note: nested metrics may cause runtime issues — use minimal form)

Native Functions

Function	Purpose
ds_status(agent)	Get agent status
ds_run_eda(agent, dataset)	Exploratory data analysis
ds_train(agent, dataset)	Train ML model
ds_predict(agent, model, data)	Generate predictions
ds_explain_global(agent, model)	SHAP global explanations
ds_exec_python(agent, code)	Execute Python in sandboxed venv

---

Causal Agent (v0.9.8.1)

Purpose: General-purpose causal reasoning. Pearl's Ladder of Causation (Rungs 2-3), SCM declarations, do-calculus interventions, counterfactual analysis. Composable primitive — any agent can invoke it.

Keyword: causal agent

Minimal (Verified)

budget B { cost: 50.00, tokens: 500000 }

causal agent MinCausal {
    provider: "openai",
    model: "o3-mini",
    temperature: 0.3,
    budget: B
}

With SCM, Intervention, Counterfactual (Verified)

budget B { cost: 50.00, tokens: 500000 }

scm ChurnSCM {
    variables: ["support", "csat", "churn"],
    equations: { csat: "f(support) + U", churn: "f(csat) + U" }
}

intervention SupportIntv {
    treatment: "support",
    outcome: "churn",
    do_value: "improve_1std",
    identification: "backdoor"
}

counterfactual RetentionCF {
    question: "Would C-1234 stay with premium support?",
    factual: { tier: "basic", outcome: "churned" },
    counterfactual: { tier: "premium" }
}

causal agent WhyChurn {
    provider: "openai",
    model: "o3-mini",
    budget: B,
    scm: ChurnSCM,
    intervention: SupportIntv,
    counterfactual: RetentionCF
}

Native Functions

Function	Purpose
causal_status(agent)	Get agent status
causal_discover_dag(agent, data)	Discover causal DAG from observational data
causal_estimate_ate(agent, intervention)	Average Treatment Effect via do-calculus
causal_counterfactual(agent, cf)	Counterfactual analysis
causal_rca(agent, problem)	Root cause analysis

---

DataTest Agent (v0.9.8.4)

Purpose: Independent quality critic. Generates tests from acceptance criteria, enforces BLOCKING quality gates. Removing DataTest drops system quality score by 26.1% (ablation A5).

Keyword: datatest agent

Minimal (Verified)

budget B { cost: 50.00, tokens: 500000 }

datatest agent MinTest {
    provider: "openai",
    model: "gpt-4o",
    temperature: 0.2,
    budget: B
}

With Quality Gate and Test Suite

budget B { cost: 50.00, tokens: 500000 }

quality_gate ChurnGate {
    data_quality_gate: { required_pass_rate: 1.0, blocking: true },
    model_quality_gate: { required_metrics: { auc_roc: "> 0.80" }, blocking: true }
}

etl_test_suite PipelineTests {
    row_count_check: { tolerance: 0.01 },
    null_check: { columns: ["customer_id"], threshold: 0.0 }
}

datatest agent QualityCritic {
    provider: "openai",
    model: "gpt-4o",
    budget: B,
    quality_gate: ChurnGate,
    etl_test_suite: PipelineTests
}

Native Functions

Function	Purpose
test_status(agent)	Get agent status
test_generate_cases(agent, criteria)	Auto-generate tests from acceptance criteria
test_run_suite(agent, suite)	Run test suite
test_evaluate_gate(agent, gate)	Evaluate blocking/advisory quality gate

---

MLOps Agent (v0.9.8.2)

Purpose: Day-2 ML operations — 6 drift types, continuous training pipelines, deployment strategies (canary/shadow/blue-green/A/B), champion-challenger, business KPI tracking.

Keyword: mlops agent

Minimal (Verified)

budget B { cost: 100.00, tokens: 500000 }

mlops agent MinMLOps {
    provider: "openai",
    model: "gpt-4o",
    temperature: 0.2,
    budget: B
}

With Drift Monitor and Deployment Strategy

budget B { cost: 100.00, tokens: 500000 }

drift_monitor ChurnDrift {
    data_drift: { method: "evidently", check_frequency: "hourly", threshold: 0.05 },
    concept_drift: { method: "performance_monitoring", metrics: ["auc_roc"], degradation_threshold: 0.05 }
}

deployment_strategy CanaryDeploy {
    strategy: "canary",
    canary_percentage: 10,
    promotion_criteria: { min_duration: "2h", max_error_rate: 0.01 },
    rollback: { auto: true, trigger: "error_rate > 0.05" }
}

champion_challenger ModelComp {
    champion: "v2", challenger: "v3",
    comparison_metrics: ["auc_roc"],
    traffic_split: { champion: 90, challenger: 10 }
}

serving_infra ServingConfig {
    type: "kubernetes", replicas: 3,
    resources: { cpu: "2", memory: "4Gi" }
}

mlops agent ProductionGuardian {
    provider: "openai",
    model: "gpt-4o",
    budget: B,
    drift_monitor: ChurnDrift,
    deployment_strategy: CanaryDeploy,
    champion_challenger: ModelComp,
    serving_infra: ServingConfig
}

Native Functions

Function	Purpose
mlops_status(agent)	Get deployment status
mlops_check_drift(agent)	Check for 6 drift types
mlops_deploy_canary(agent, model)	Canary deployment
mlops_rollback(agent)	Rollback to champion
mlops_trigger_retrain(agent)	Trigger continuous training

---

The Data Intelligent Orchestrator (DIO) — v0.9.9

Core Concept

DIO is the master intelligence that:

1. Understands the task (NL decomposition into sub-tasks)
2. Forms crews dynamically (weighted scoring: capability 40%, cost 20%, infrastructure 20%, history 20%)
3. Selects patterns from 8 auto-patterns
4. Assigns RACI accountability to every sub-task
5. Delegates to specialist agents via async pub/sub
6. Monitors execution with activity capture
7. Self-heals on failure (4-tier escalation)
8. Synthesizes results into a unified response

Three Operating Modes

Mode	Description	Use Case
"auto"	Fully autonomous, no infrastructure config needed	Quick experiments, prototyping
"config"	Operates on your real infrastructure (Snowflake, BigQuery, etc.)	Production systems
"hybrid"	Auto with configurable guardrails and human checkpoints	Regulated environments

DIO — Auto Mode (Verified)

budget B { cost: 500.00, tokens: 2000000 }

dio agent AutoDIO {
    mode: "auto",
    task: "Predict customer churn",
    provider: "openai",
    model: "gpt-4o",
    budget: B
}

DIO — Config Mode with Infrastructure (Verified)

budget B { cost: 500.00, tokens: 2000000 }

infrastructure_profile Infra {
    data_warehouse: { platform: "postgres" }
}

dio agent ConfigDIO {
    mode: "config",
    task: "Predict churn, identify drivers, deploy with monitoring",
    infrastructure: Infra,
    provider: "openai",
    model: "gpt-4o",
    budget: B
}

DIO — Hybrid Mode with Guardrails (Verified)

budget B { cost: 500.00, tokens: 2000000 }

infrastructure_profile Infra {
    data_warehouse: { platform: "postgres", connection: "pg://localhost" }
}

dio agent HybridDIO {
    mode: "hybrid",
    task: "Reduce churn",
    infrastructure: Infra,
    guardrails: { require_approval: ["deployment"], max_budget_per_phase: 100.00 },
    provider: "openai",
    model: "gpt-4o",
    budget: B
}

DIO Fields

Field	Required	Values
mode	Yes	"auto", "config", "hybrid"
task	Yes	Natural language task description
infrastructure	No (warning DIO-010 if missing)	Reference to infrastructure_profile
agent_md	No	Path to Agent.MD file
guardrails	No (hybrid mode)	require_approval, max_budget_per_phase
agent_registry	No (warning DIO-003)	Reference to agent_registry
raci_matrix	No (warning DIO-005)	Reference to raci_matrix
provider, model, budget	Yes	LLM and budget configuration

DIO Native Functions

Function	Purpose
dio_status(agent)	Get orchestrator status
dio_solve(agent, task)	Full orchestration: understand, crew, delegate, execute, synthesize
dio_plan(agent, task)	Generate execution plan without executing

8 Auto-Patterns

DIO automatically selects the right orchestration pattern based on the task:

Pattern	Trigger	Agents Involved
Predictive	"predict", "classify", "forecast"	Data-BA, Data, ETL, DataScientist, DataTest, MLOps
Root Cause	"why", "root cause", "investigate"	Causal, Analyst, DataScientist
Platform Build	"build warehouse", "create pipeline"	Data, ETL, Modeling, DataOps
Ops	"monitor", "alert", "heal"	DataOps, MLOps
Compliance	"audit", "gdpr", "classify"	Governance, DataTest
Exploratory	"explore", "analyze", "discover"	Analyst, DataScientist, Causal
Migration	"migrate", "move", "re-platform"	Migration, DataOps, DataTest
Full Lifecycle	"full system", "end to end"	All 14 agents

---

Full Production Program (Verified)

budget DIOBudget { cost: 500.00, tokens: 2000000 }
budget AgentBudget { cost: 50.00, tokens: 500000 }

infrastructure_profile SimShopInfra {
    data_warehouse: {
        platform: "postgres",
        connection: env("SIMSHOP_PG_URL"),
        schemas: ["simshop_oltp", "simshop_dw", "ml_features"]
    },
    data_science: { mlflow: { uri: env("MLFLOW_TRACKING_URI") } },
    governance: { regulations: ["GDPR"], pii_columns: ["email", "phone"] }
}

databa agent ChurnBA { provider: "openai", model: "gpt-4o", budget: AgentBudget }
sql_connection SimShopDB { platform: "postgres", connection: env("SIMSHOP_PG_URL"), database: "simshop" }
analyst agent SimShopAnalyst { provider: "openai", model: "gpt-4o-mini", connections: [SimShopDB], budget: AgentBudget }
datascientist agent ChurnDS { provider: "openai", model: "gpt-4o", budget: AgentBudget }
causal agent ChurnCausal { provider: "openai", model: "o3-mini", budget: AgentBudget }
datatest agent ChurnTester { provider: "openai", model: "gpt-4o", budget: AgentBudget }
mlops agent ChurnMLOps { provider: "openai", model: "gpt-4o", budget: AgentBudget }

dio agent SimShopDIO {
    mode: "config",
    task: "Predict which customers will churn in 90 days, identify drivers, deploy with monitoring",
    infrastructure: SimShopInfra,
    agent_md: "./agents/simshop_dio.agent.md",
    provider: "openai",
    model: "gpt-4o",
    budget: DIOBudget
}

print(dio_solve(SimShopDIO, "full_system"));

---

Agent.MD: Encoding Human Expertise

Agent.MD is a markdown file that encodes domain knowledge so agents execute with human-level context. Removing Agent.MD drops model AUC by 7.7% (ablation A6).

Structure

# SimShop -- Agent.MD

## @organization-context
- SimShop: B2C e-commerce, 100K customers
- Data period: Jan 2024 - Dec 2025
- Primary database: PostgreSQL (simshop_oltp)
- Warehouse: simshop_dw schema

## @causal-domain-knowledge
- Support quality -> satisfaction -> retention (established)
- Price sensitivity varies by segment (enterprise vs SMB)
- Seasonal effects: Q4 holiday purchases mask churn signals

## @methodology-preferences
- XGBoost for initial models (interpretable, fast)
- Bayesian approaches when sample sizes allow
- Always start with simple baselines before complex models

## @known-data-issues
- CRM pre-2024 data has missing industry field
- Support sentiment accuracy ~75%
- Duplicate customer records in legacy system (~2%)

## @delegation-rules
- Validate requirements first (Data-BA always runs first)
- Quality gates block deployment (no exceptions)
- Max 3 retries before escalation to human
- Causal analysis required before prescriptive recommendations

## @pipeline-catalog
- etl_daily_orders: runs 02:00 UTC, SLA 45min
- ml_churn_scoring: runs 06:00 UTC, depends on etl_daily_orders

Referencing Agent.MD

dio agent DIO {
    agent_md: "./agents/simshop_dio.agent.md",
    mode: "config",
    task: "Predict churn",
    ...
}

---

Agent Registry

The agent registry defines the identity card of every agent — role, capabilities, inputs, outputs, and invocation rules.

agent_registry DIOBlueprint {
    agents: {
        databa: {
            role: "Requirements Analyst",
            personality: "Meticulous, question-asking, detail-oriented",
            responsibility: "Translate business needs into structured specifications",
            version: "v0.9.8.3",
            capabilities: ["elicitation", "brd", "functional_spec", "acceptance_criteria",
                          "data_mapping", "impact_analysis", "traceability"],
            when_to_call: "At project start, on change requests, on production feedback",
            inputs: ["business_problem", "stakeholder_context", "domain_knowledge"],
            outputs: ["brd", "functional_spec", "acceptance_criteria"],
            autonomy: "needs_stakeholder_approval_for_specs"
        },
        datascientist: {
            role: "Data Scientist",
            personality: "Curious, experimental, model-obsessed",
            responsibility: "Build, train, and evaluate ML/AI models",
            version: "v0.9.8",
            capabilities: ["eda", "feature_engineering", "ml_training", "automl",
                          "hypothesis_testing", "model_evaluation", "shap"],
            when_to_call: "When predictions, classifications, or patterns are needed",
            inputs: ["ml_requirement_spec", "feature_tables", "problem_statement"],
            outputs: ["trained_models", "predictions", "shap_reports", "model_cards"],
            depends_on: ["etl"]
        },
        causal: {
            role: "Causal Investigator",
            personality: "Skeptical, rigorous, correlation-is-not-causation",
            responsibility: "Discover WHY things happen and WHAT-IF scenarios",
            version: "v0.9.8.1",
            capabilities: ["causal_discovery", "scm", "do_calculus", "counterfactual"],
            when_to_call: "When 'why' or 'what if' questions need answering",
            inputs: ["business_questions", "observational_data", "domain_knowledge"],
            outputs: ["causal_dags", "effect_estimates", "counterfactual_results"],
            general_purpose: true
        },
        datatest: {
            role: "Quality Critic",
            personality: "Skeptical, thorough, never satisfied",
            responsibility: "Validate everything before production",
            version: "v0.9.8.4",
            capabilities: ["test_generation", "etl_testing", "dw_testing", "ml_testing",
                          "quality_gates"],
            when_to_call: "Before any deployment, after any build",
            inputs: ["acceptance_criteria", "models", "pipelines"],
            outputs: ["test_reports", "quality_gate_results"],
            depends_on: ["databa"]
        }
    }
}

---

RACI Accountability

Every sub-task has exactly one Responsible agent. DIO is always Accountable. This is enforced at runtime.

Two Invariants

1. Exactly one R per task — if zero or multiple agents are Responsible, DIO raises an error
2. DIO is always A — the orchestrator is accountable for every task outcome

RACI in Practice

DIO automatically assigns RACI based on the agent registry. Removing RACI enforcement drops traceability by 80% (ablation A8).

Task: "Train churn model"
  R: DataScientist (builds the model)
  A: DIO (accountable for outcome)
  C: Causal Agent (consulted for feature selection)
  I: MLOps Agent (informed for deployment planning)

---

Three Coordination Modes

1. Centralized RACI (Default)

DIO manages all agent interactions through RACI assignment. Deterministic, auditable, recommended for production.

2. Swarm Stigmergy

Agents coordinate through shared artifacts without central control. Task completion signals trigger downstream agents automatically.

• Converges in ~23 iterations
• ~2% deadlock rate (DIO intervenes)
• Best for exploratory, research-heavy tasks

3. Evolutionary (Genetic Algorithm)

DIO generates multiple crew/pattern configurations and evolves them through selection, crossover, and mutation.

• Fitness convergence: 0.91 by generation 67
• Best for optimization-heavy tasks with unclear best approach

// Select coordination mode via dio_solve
print(dio_solve(MyDIO, "full_system"));        // Default: centralized RACI
print(dio_solve(MyDIO, "swarm_mode"));          // Swarm stigmergy
print(dio_solve(MyDIO, "evolutionary_mode"));   // Evolutionary GA

---

Infrastructure Profiles

Infrastructure profiles abstract platform differences so the same DIO program runs on any data platform.

// PostgreSQL (development)
infrastructure_profile Dev { data_warehouse: { platform: "postgres" } }

// Snowflake (production)
infrastructure_profile Prod {
    data_warehouse: { platform: "snowflake", connection: env("SF_URL"), warehouse: "COMPUTE_WH" },
    data_science: { mlflow: { uri: env("MLFLOW_URI") } }
}

// BigQuery (analytics)
infrastructure_profile Analytics {
    data_warehouse: { platform: "bigquery", project: env("GCP_PROJECT"), dataset: "analytics" }
}

// Databricks (lakehouse)
infrastructure_profile Lakehouse {
    data_warehouse: { platform: "databricks", connection: env("DBX_URL") }
}

// Same DIO, different platform — change only the profile reference:
dio agent DIO { mode: "config", task: "Predict churn", infrastructure: Prod, ... }

Supported Platforms

"postgres", "snowflake", "bigquery", "databricks", "redshift", "oracle", "teradata", "fabric", "bigtable", "spark"

Full Infrastructure Profile

infrastructure_profile Enterprise {
    data_warehouse: {
        platform: "snowflake",
        connection: env("SF_URL"),
        warehouse: "COMPUTE_WH",
        schemas: ["raw", "staging", "dw", "ml_features"]
    },
    data_science: {
        mlflow: { uri: env("MLFLOW_URI") }
    },
    governance: {
        regulations: ["GDPR", "CCPA"],
        pii_columns: ["email", "phone", "ssn"]
    }
}

---

Error Handling and Self-Healing

4-Tier Escalation Hierarchy

Tier	Strategy	Example
1	Agent Retry	Agent retries failed operation (max 3)
2	DIO Adapt	DIO reassigns task to different agent or pattern
3	Human Review	Escalate to human with full context
4	Halt	Stop execution, preserve state for forensics

State Checkpointing

DIO checkpoints state at each phase boundary. On failure, it can resume from the last successful checkpoint without re-executing completed work.

Schema Break Recovery

When an upstream schema change breaks a pipeline:

1. DataOps detects the anomaly
2. Governance traces the lineage impact
3. ETL Agent proposes schema adaptation
4. DataTest validates the fix
5. DIO coordinates the recovery without human intervention

---

Security: Defense in Depth

6-Layer Security Model

Layer	Component	Purpose
1	Authentication	Agent identity verification
2	Authorization	RBAC/ABAC access control
3	Classification	Automatic data sensitivity tagging
4	Guards	Input/output content filtering
5	Audit	Tamper-proof activity logging
6	Budget	Cost and token limits

Phase-Gated Permissions

Agents only receive the permissions they need for their current phase. A DataScientist agent cannot access raw PII — it only sees the feature tables produced by the ETL Agent after Governance has applied masking.

Zero-Trust Inter-Agent Communication

Agent A -> [Auth] -> [Authz] -> [Classify] -> [Guard] -> Agent B
                                                  |
                                              [Audit Log]

---

DataSims Evaluation

DataSims is the evaluation framework for validating DIO programs. It uses SimShop — a synthetic e-commerce environment with 164 tables, 12 schemas, and 10 deliberately injected quality issues.

Running DataSims

git clone https://github.com/neam-lang/Data-Sims.git
cd Data-Sims
python3 evaluation/run_experiments.py --reps 5
python3 evaluation/analysis.py

7-Dimension Evaluation Framework

Dimension	Weight	Measures
Model Quality	25%	AUC, precision, recall
Root Cause	15%	Correct causal identification
Quality Gates	15%	Gate enforcement
Documentation	10%	BRD completeness
Reproducibility	10%	Run-to-run consistency
Cost Efficiency	10%	Total cost vs budget
Time Efficiency	15%	Total elapsed time

Expected Results (Verified)

Condition	AUC	Root Cause	Gate
full_system	0.847	support_quality_degradation	passed
ablation_no_agentmd	0.782 (-7.7%)	support_quality_degradation	passed
ablation_no_causal	0.847	unknown	passed
ablation_no_test	0.847	support_quality_degradation	skipped
ablation_no_gates	0.847	support_quality_degradation	bypassed
swarm_mode	0.847	support_quality_degradation	passed
evolutionary_mode	0.847	support_quality_degradation	passed

Key Metrics

• Cost: $23.50 per full lifecycle run (vs $548K traditional — 93.7% reduction)
• Reproducibility: 100% (50/50 runs)
• Test coverage: 94% (47 tests auto-generated)
• Risk reduction: 90.6%

---

Multi-Agent Coexistence (Verified)

All agent types coexist in a single program:

budget B { cost: 10.00, tokens: 100000 }

sql_connection DB { platform: "postgres", connection: "test", database: "db" }
analyst agent A { provider: "openai", model: "gpt-4o-mini", connections: [DB], budget: B }
datascientist agent DS { provider: "openai", model: "gpt-4o", budget: B }
causal agent CA { provider: "openai", model: "o3-mini", budget: B }
mlops agent MO { provider: "openai", model: "gpt-4o", budget: B }
databa agent BA { provider: "openai", model: "gpt-4o", budget: B }
datatest agent QA { provider: "openai", model: "gpt-4o", budget: B }

dio agent DIO {
    mode: "auto",
    task: "Test all agents together",
    provider: "openai", model: "gpt-4o", budget: B
}

Contextual Keywords as Variables (Verified)

DIO agent keywords are contextual — they only trigger at declaration position:

let datascientist = "I am a variable, not an agent keyword";
let causal = 42;
let mlops = true;
let dio = "also just a variable";

print(datascientist);  // "I am a variable, not an agent keyword"
print(causal);          // 42

Legacy Agent Coexistence (Verified)

v0.6 agents and v0.9.x DIO agents work together:

budget B { cost: 10.00, tokens: 100000 }

// v0.6 agent
agent LegacyHelper { provider: "openai", model: "gpt-4o-mini", system: "helper", budget: B }

// v0.9.8 agent
datascientist agent DS { provider: "openai", model: "gpt-4o", budget: B }

// Both work together

---

Native Function Reference

Agent	Status Function	Key Action Functions
Migration	migration_status(a)	migration_plan, migration_execute_wave, migration_reconcile
DataOps	dataops_status(a)	dataops_detect_anomaly, dataops_auto_heal, dataops_sla_check
Governance	governance_status(a)	gov_classify, gov_check_access, gov_trace_lineage
Modeling	modeling_status(a)	model_reverse_engineer, model_generate_er, model_normalize
Analyst	analyst_status(a)	analyst_query, analyst_sql, analyst_insight, analyst_format
DataScientist	ds_status(a)	ds_train, ds_predict, ds_run_eda, ds_explain_global, ds_exec_python
Causal	causal_status(a)	causal_discover_dag, causal_estimate_ate, causal_counterfactual, causal_rca
MLOps	mlops_status(a)	mlops_deploy_canary, mlops_check_drift, mlops_rollback, mlops_trigger_retrain
Data-BA	ba_status(a)	ba_generate_brd, ba_generate_criteria, ba_traceability_matrix
DataTest	test_status(a)	test_run_suite, test_evaluate_gate, test_generate_cases
DIO	dio_status(a)	dio_solve, dio_plan

Note: Data Agent and ETL Agent do not have dedicated status functions. Use compilation success as verification.

---

Required Fields by Agent

Agent	Required Fields	Recommended (Warning if Missing)
data agent	provider, model, sources, sinks	schema, quality, compute
etl agent	provider, model, sources, warehouse	marts, semantic
migration agent	provider, model, source, target, staging, strategy, budget	--
dataops agent	provider, model, platforms, incident_policy	schedulers, audit_tables
governance agent	provider, model, classification, budget	--
modeling agent	provider, model, sources	--
analyst agent	provider, model, connections, budget	domain_context
datascientist agent	provider, model, budget	eda_config, volume_router
causal agent	provider, model, budget	scm, intervention
mlops agent	provider, model, budget	drift_monitor (MOP-002)
databa agent	provider, model, budget	brd_generator
datatest agent	provider, model, budget	test_strategy (TST-003)
dio agent	mode, task, provider, model, budget	infrastructure (DIO-010), agent_registry (DIO-003)

---

Troubleshooting

Error	Cause	Fix
Unknown source type 'postgresql'	Wrong source type string	Use "postgres" not "postgresql"
Expected schema identifier	Wrong schema syntax	Use schema Name { field: type } with types: string, int, float, bool, datetime
Expected number for completeness	Nested quality block	Use flat: quality Q { completeness: 0.95 } not completeness: { threshold: 0.95 }
Unexpected token in compute block	Wrong compute syntax	Use compute C { engine: "local" } not type: "local"
Unknown field in data agent: schemas	Plural field name	Use schema: S (singular) not schemas: [S]
Unexpected field in etl agent: target_dialect	Wrong ETL field	Use warehouse: ComputeRef not target_dialect: "postgres"
OPS-001: must declare platforms	Missing required field	DataOps requires platforms: [PlatformRef]
OPS-002: must declare incident_policy	Missing required field	DataOps requires incident_policy: PolicyRef
Unknown classification_policy field: patterns	Wrong classification syntax	Use levels: { PII: { level: 3, controls: ["masking"] } } not patterns
Unknown migration agent field: source_platform	Wrong migration syntax	Use source: SourceRef, target: TargetRef, staging: StagingRef
[warning] DIO-003: should have agent_registry	Advisory warning	Safe to ignore -- agent_registry is recommended but not required
[warning] DIO-010: missing infrastructure	Advisory warning	Add infrastructure_profile for config/hybrid modes
[warning] MOP-002: should have drift_monitor	Advisory warning	Safe to ignore -- drift_monitor is recommended
Runtime: Expected string value	Non-string concatenation	Don't concatenate status results with +. Call print(status_fn(agent)) separately

---

Testing DIO Programs

Compile-Only Test

neamc my_program.neam -o /tmp/test.neamb 2>&1
# Success: "Wrote bytecode bundle to ..."
# Failure: Parse/compile error with details

Status Test

neamc my_program.neam -o /tmp/test.neamb && neam /tmp/test.neamb

CTest Suite

cd build && ctest --output-on-failure --parallel $(sysctl -n hw.ncpu)

E2E Validation (22 Tests)

T01-T03: Schema, Source/Sink, Quality/Compute declarations
T04-T15: All 14 agent types compile and initialize
T16-T17: DIO in Auto and Config modes
T18: DIO orchestration (dio_solve) produces complete results
T19: Contextual keywords work as variables
T20: Legacy + v0.9.x coexistence
T21-T22: DataSims full system + ablation experiments

---

Three Paradigms: When to Use DIO

Paradigm	Approach	Best For	Risk
Vibe Coding	Prompt-driven, iterate with LLM	Quick prototypes, one-off analysis	No reproducibility, no audit trail
Agentic	Single agent with tools	Focused tasks, chatbots, copilots	No cross-team coordination
Spec-Driven (DIO)	14 agents + orchestrator, RACI, quality gates	Production data systems, ML pipelines	Higher upfront cost (but 93.7% total reduction)

When to Use DIO

• Data projects involving multiple roles (BA, DE, DS, MLOps)
• Projects requiring reproducibility and audit trails
• Regulated environments (GDPR, CCPA, DORA)
• When handoff failures have caused project failures before
• When you need causal analysis, not just correlation
• Production ML systems requiring monitoring and retraining

When NOT to Use DIO

• Simple single-agent chatbots (use regular agent)
• One-off data exploration (use analyst agent alone)
• Interactive sessions (use claw agent)
• Build/generation tasks (use forge agent)

---

Key Rules

1. Always define budgets for production DIO programs
2. Data-BA runs first in every project ("Day -1")
3. Quality gates block deployment by default — never bypass in production
4. Use env() for secrets — never hardcode connection strings
5. Agent.MD improves quality by 7.7% — always provide domain knowledge
6. DIO is always Accountable in RACI — exactly one R per task
7. Infrastructure profiles abstract platforms — swap profiles, not code
8. Causal Agent is composable — any agent can invoke it for "why" questions
9. DataTest is the most critical agent — removing it drops quality by 26.1%
10. Test compilation first before running with live LLM calls

---

Environment Variables

export OPENAI_API_KEY="sk-..."                                           # Required for LLM agents
export SIMSHOP_PG_URL="postgresql://datasims:datasims_2026@localhost:5432/simshop"  # DataSims
export MLFLOW_TRACKING_URI="http://localhost:5000"                       # MLflow tracking

---

v1.0–v1.4 Compatibility

All 14 DIO specialist agents and the dio agent orchestrator (v0.9.x) coexist unchanged with every v1.x platform release — v1.0 security, v1.1 knowledge fabric, v1.2 production infrastructure, v1.3 research agents, v1.4 wikis. Any DIO program written for v0.9.9 compiles and runs on v1.4 without modification.

For the full syntax of v1.x keywords (OWASP ASI01–ASI10, knowledge cards, context assembly, governance rules, plugins, sessions, eval, artifacts, streaming, A2A, research agents, wikis), see the claude-neam-programming skill's "v1.0–v1.4: Platform Evolution" section. This DIO skill covers only the patterns where v1.x features touch DIO orchestration or the 14 specialist agents.

Securing the DIO Stack (v1.0)

Layer OWASP security constructs around any DIO program. The securitysentinel agent can monitor every specialist agent for goal drift, tool misuse, and behavioral anomalies. costguardian agent tracks per-agent / per-phase spend across the orchestration. protocolbridge agent gates MCP/A2A connections that DIO agents make.

// Security overlay for DIO (v1.0 constructs added to a v0.9.x DIO program)
goal_integrity ChurnGoal {
    declared_objectives: ["predict churn", "identify causes", "deploy with monitoring"],
    verification: { method: "semantic_similarity", threshold: 0.75, on_drift: "halt_and_escalate" }
}
circuit_breaker DIOCB { failure_threshold: 3, half_open_timeout: "30s" }
human_gate DeployGate { approve_before: ["deploy"], workflow: { timeout: "15m", default_on_timeout: "deny" } }
agent_attestation DIOAttest { attest_interval: "5m",
                              kill_switch: { api_enabled: true, auto_kill_on: ["goal_drift_detected"] } }

// v0.9.x DIO stack — unchanged
databa    agent BA       { provider: "openai", model: "gpt-4o", budget: B }
datascientist agent DS   { provider: "openai", model: "gpt-4o", budget: B }
causal    agent WhyCausal{ provider: "openai", model: "o3-mini", budget: B }
datatest  agent QA       { provider: "openai", model: "gpt-4o", budget: B }
mlops     agent MOps     { provider: "openai", model: "gpt-4o", budget: B }

// v1.0 sentinel watches every DIO agent
securitysentinel agent DIOSentinel {
    provider: "openai", model: "gpt-4o", budget: B,
    monitors: { goal_integrity: { check: "per_phase", threshold: 0.75 },
                behavioral_anomaly: { check: "continuous", sigma: 3.0 } },
    actions: { on_anomaly: "alert_and_log", on_critical: "kill_switch",
               on_breach: "halt_all_and_escalate" }
}
costguardian agent DIOCost {
    provider: "ollama", model: "llama3:8b", budget: B,
    tracking: { per_agent: true, per_phase: true, per_model: true },
    alerts: { budget_warning: 0.70, budget_critical: 0.85 }
}

// v0.9.9 DIO orchestrator — unchanged
infrastructure_profile ProdInfra { data_warehouse: { platform: "postgres" } }
dio agent SecureDIO {
    mode: "config",
    task: "Predict churn with OWASP-compliant security",
    infrastructure: ProdInfra,
    provider: "openai", model: "gpt-4o", budget: B
}
print(dio_solve(SecureDIO, "full_system"));

See programming skill for the full 10-layer OWASP (ASI01-ASI10) defense-in-depth pattern and the gym_evaluator multi_agent mode for DIO coordination scoring.

Encoding Domain Knowledge for DIO Agents (v1.1)

Knowledge cards complement Agent.MD for structured, compiler-validated domain knowledge per specialist agent. Use context_assembly to curate per-phase context for a DataScientist, a MLOps agent, or a Causal agent — the compiler validates card references and token budgets at build time.

knowledge_card CustomerChurn {
    type: "concept", term: "Customer Churn",
    definition: "Inactive 90+ days, no login 60+ days.",
    domain: "telecom.retention", version: "2.1.0",
    owner: ChurnDS, provenance: { verified_by: "DomainExpert" }
}
knowledge_card ChurnModelDecision {
    type: "decision",
    context: "Model selection for SEA markets",
    decision: "XGBoost for batch, logistic for real-time",
    decided_by: ChurnDS, rationale: "0.89 AUC vs 50ms SLA"
}

context_assembly ChurnFEContext {
    target_agent: ChurnDS, phase: "feature_engineering",
    cards: [CustomerChurn, ChurnModelDecision, PIIPolicy],
    max_context_tokens: 4000, assembly_strategy: "relevance_ranked"
}

// Governance rules fire on DIO agent actions
governance_rule PIIRetention {
    trigger: "data_write",
    condition: data.classification == "pii" && data.region in ["sg", "au", "in"],
    actions: { enforce_ttl: "365d", audit_log: true },
    severity: "critical", regulatory_basis: ["PDPA", "GDPR"]
}

// KnowledgeWeaver agent governs the card ecosystem alongside DIO
knowledgeweaver agent OrgWeaver {
    provider: "openai", model: "gpt-4o-mini", budget: KB,
    fabric: { card_sources: [{ type: "inline", cards: [CustomerChurn] }],
              conflict_detection: true },
    monitors: { freshness: { check_interval: "12h", stale_threshold: "90d" },
                consistency: { cross_card_validation: true } }
}

Blueprints (v1.1) bundle a DIO program — agents, cards, context assemblies, governance rules, personas, locales, parameters — into a single distributable .nbp. Useful for publishing vertical DIO solutions (telecom churn, financial fraud, HIPAA patient prediction).

Agent adapters (v1.1) let DIO orchestrate external agents — Claude Code, Bedrock Agents, OpenClaw — as first-class citizens in the agent registry, with cost tracking and RACI accountability.

Operationalising DIO (v1.2)

• Plugins — inject cross-cutting concerns (cost attribution, compliance audit, response caching, SIEM logging) across every DIO agent without touching agent code. before_tool / after_tool hooks are especially useful for logging every action taken by specialist agents. Use scope: "global" for the whole DIO stack, or name a specific specialist agent.
• Session service — DIO phase state (BA brief → Modeling ER → DS predictions → MLOps deployment) persists across runs via backend: "postgresql" with event-sourced delta records. Enables session rewind for debugging long DIO orchestrations.
• eval_test / eval_set — replaces the DataSims ablation block for structured DIO testing. trajectory_mode: "exact" verifies the specialist agents fire in the right order; criteria: ["safety", "hallucination"] gates DIO deployments in CI/CD via JUnit XML output.
• Artifact store — versioned outputs from specialist agents (DataTest reports, MLOps deployment manifests, Governance classification decisions) with automatic metadata and 7-year SOX/PDPA retention on S3.
• a2a_config — expose the dio agent as an A2A v0.3 server so external frameworks (LangChain, ADK, CrewAI) can submit tasks. Or discover remote A2A agents and have DIO orchestrate them alongside native specialists.

Research-Driven DIO (v1.3)

Pair a research agent with DIO specialists for autonomous optimization of data pipelines, feature sets, or model configurations. The research agent iterates a mutable artifact (e.g., ./feature_pipeline.py) toward a metric extracted by a metric_extractor; DataTest and MLOps validate each iteration; DIO orchestrates the workflow.

program FeatureDiscovery {
    mission: "Find minimal feature set for 90% churn AUC",
    success_criterion: "ChurnAUC",
    constraints: ["Only modify ./feature_pipeline.py", "Never modify ./eval_dataset.csv",
                  "Maximum 50 features", "No PII fields"],
    process: ["Baseline", "Hypothesize feature change", "Train + evaluate", "Keep or revert"],
    autonomy: "full"
}
metric_extractor ChurnAUC { direction: "higher_is_better", method: "json_path",
                            path: "$.metrics.test_auc", tolerance: 0.005 }
session_service ChurnExperiments { backend: "postgres", ttl: "90d" }

research agent FeatureFinder {
    provider: "openai", model: "gpt-4o", budget: B,
    program: FeatureDiscovery, metric: ChurnAUC, experiment_log: ChurnExperiments,
    mutable_artifacts:   ["./feature_pipeline.py"],
    immutable_artifacts: ["./eval_dataset.csv", "./train.py"],   // SHA-256 enforced
    hypothesis_required: true, target_metric: 0.92,
    iteration_budget: { time_per_run: "10m", max_runs: 200 }
}
// DIO specialists still run validation and deployment after each research iteration

Compounding DIO Knowledge with Wikis (v1.4)

A wiki agent sitting alongside DIO captures every experiment, model decision, and post-mortem into a compounding markdown knowledge base with knowledge-graph visualisation. DataTest reports, Causal RCA outputs, and Data-BA BRDs can be wiki_ingested; the bridge converts v1.1 knowledge_cards to wiki concept pages and back. Run wiki_graph to see the relationships between concepts, entities, and specialist-agent decisions.

wiki DIOKnowledge {
    topic: "dio_research",
    raw_path: "./wiki/raw", wiki_path: "./wiki/pages",
    page_types: ["source", "entity", "concept", "synthesis"],
    contradiction_detection: true, auto_extract_entities: true
}
wiki agent DIOCurator {
    provider: "anthropic", model: "claude-opus-4", budget: B,
    wikis: [DIOKnowledge],
    operations: ["ingest", "query", "lint", "graph"],
    graph_config: { deterministic_pass: true, semantic_pass: true,
                    community_detection: "louvain", output_format: "html" },
    knowledge_cards: [CustomerChurn],      // v1.1 card bridge
    governance: [WikiPDPA]                 // v1.1 governance fires at ingest
}

Stacking Summary for DIO

A modern production DIO program typically layers:

1. v0.9.x — the 14 specialist agents + dio agent + infrastructure_profile + Agent.MD + RACI.
2. v1.0 — goal_integrity, circuit_breaker, human_gate on sensitive phases; securitysentinel agent monitoring the whole stack; costguardian agent for FinOps; gateway + model_router in front of DIO.
3. v1.1 — knowledge_cards per domain concept/policy/decision; context_assembly per specialist agent per phase; governance_rules for PDPA/GDPR/SOX; a blueprint to package the whole thing.
4. v1.2 — a plugin stack (logging + cost + cache + audit) applied globally; session_service with PostgreSQL for phase state; eval_set gating CI/CD; artifact_store on S3.
5. v1.3 — research agents for autonomous feature/model optimization, feeding results back to DataTest and MLOps.
6. v1.4 — a wiki capturing every outcome, with the knowledge graph making cross-project learning visible.

All six layers compose; the 14 specialist agents and dio agent syntax never changes.

---

Reference Links

• The Intelligent Data Organization (eBook)
• Neam Language Reference
• DataSims
• Neam Compiler (nightly)
• NeamSkills Library