data-processor

Herodotus AI Skill: Data Processor (HDP) (v1)

Purpose

Use this skill to implement verifiable computation pipelines with HDP and enforce soundness constraints over historical on-chain data.

When to use

• Build Cairo modules that consume proof-backed chain data.
• Run dry-run -> fetch-proofs -> sound-run.
• Enforce correctness over candidate data sourced off-chain.

Source-of-truth

• https://docs.herodotus.cloud/data-processor/introduction
• https://docs.herodotus.cloud/data-processor/architecture
• https://docs.herodotus.cloud/data-processor/pipeline
• https://docs.herodotus.cloud/data-processor/capabilities
• https://docs.herodotus.cloud/data-processor/verification
• https://docs.herodotus.cloud/data-processor/state-management
• https://docs.herodotus.cloud/data-processor/state-server
• https://docs.herodotus.cloud/data-processor/output
• https://docs.herodotus.cloud/data-processor/eth-call
• https://docs.herodotus.cloud/data-processor/design-patterns
• https://docs.herodotus.cloud/data-processor/examples
• https://docs.herodotus.cloud/data-processor/reference-cli
• https://docs.herodotus.cloud/data-processor/reference-configuration
• https://docs.herodotus.cloud/data-processor/reference-types
• https://docs.herodotus.cloud/data-processor/debugging

Architecture pattern

Treat HDP as the soundness layer, not the query/discovery layer:

indexer/discovery -> candidate set -> HDP constrained validation -> verifiable output -> app decision

Constraint patterns for query-like products

• Completeness constraint: ensure expected bounded set is fully represented.
• No-duplication constraint: deterministic ordering + uniqueness checks.
• Membership constraint: recompute keys/fields from proof-backed values.
• Anchoring constraint: compare derived commitment to trusted root/counter/checkpoint.
• Fail-closed policy: reject on any mismatch or missing proof.

Implementation workflow

1. Build candidate dataset off-chain with provenance metadata.
2. Feed candidates + policy parameters into module inputs.
3. Run hdp dry-run to discover touched keys.
4. Run hdp fetch-proofs to assemble proof inputs.
5. Run hdp sound-run for deterministic verified execution.
6. Use task_hash, output_root, and mmr_metas as final outputs.

Anti-hallucination guardrails

• Do not claim HDP is a general indexer or SQL engine.
• Do not claim local stage success guarantees settlement.
• Do not invent HDP APIs/flags outside documented surfaces.
• Do not mix HDP framework semantics with Data Processor API semantics.

Self-contained reference example

# 1) Generate candidate set (off-chain)
./indexer_job --from <start> --to <end> --out candidates.json

# 2) Discover keys
hdp dry-run -m module.compiled_contract_class.json --print_output

# 3) Fetch proofs
hdp fetch-proofs --inputs dry_run_output.json --output proofs.json

# 4) Deterministic verified run
hdp sound-run -m module.compiled_contract_class.json --proofs proofs.json --print_output

Output checklist

• Constraints documented and testable
• Candidate provenance retained
• Proof input and dry-run output matched
• Deterministic result validated across reruns