/ingest

Wiki Ingest — raw/ to wiki/ Pipeline

Process unprocessed source material from raw/ into structured wiki pages.

Iron Law

Read the ENTIRE raw file before cooking any page from it.

Skimming produces shallow summaries. Reading completely produces precise, well-connected pages.

Process

1. Check the stash

Call stash to see what's in raw/ and what needs processing.

• unprocessed — new files, never turned into wiki pages
• changed — files modified since last processing, may need page updates
• processed — already converted, skip unless re-ingesting

yap("START INGEST") to log the operation.

2. For each unprocessed or changed file

2a. Read the file completely. No skimming. Extract:

• Key claims and definitions
• Named entities (people, papers, tools, orgs)
• Methods and concepts introduced
• Comparisons made (explicit or implicit)
• The source's own confidence signals ("we hypothesize" vs "we demonstrate")

2b. Decompose into page candidates. One source often produces multiple pages:

Example: A research paper introducing "Constitutional AI"

Page candidates:
1. entity: "Constitutional AI Paper" (the paper itself)
2. concept: "Constitutional AI" (the method)
3. concept: "AI Self-Critique" (a sub-technique introduced)
4. comparison: "Constitutional AI vs RLHF" (explicit comparison in the paper)

2c. Check for overlaps. For each candidate, call lookup with key terms.

3. Create-vs-Update Decision

For each candidate where lookup found an existing page:

• No overlap → CREATE new page
• New info replaces old entirely → CREATE + supersedes
• Minor addition → UPDATE existing (cook same slug)
• Major new angle → CREATE + extends
• New info contradicts → CREATE + contradicts (NEVER overwrite)

See commands/references/create-vs-update.md for full rules.

4. Choose page type

Question	Type
Named thing (person, paper, org, tool)?	entity
Abstract idea, method, principle?	concept
Explicitly comparing 2+ things?	comparison
Condensing this raw source into overview?	summary

See commands/references/page-types.md for full heuristic.

5. Build relationships

For every connection, choose the relationship type:

| Evidence strengthening a claim? | supports | | Evidence challenging a claim? | contradicts (must be reciprocal) | | Building on top of a concept? | extends | | Replacing entirely? | supersedes | | Practical application of theory? | implements | | Referencing as source? | cites | | Condensed version? | summarizes |

See commands/references/relationship-types.md for decision tree.

Write the reason. Must explain WHY to someone who hasn't read either page.

6. Assign confidence

Source Type	Base Level
Peer-reviewed paper, verified claims	high
Reputable blog, official docs	medium
Unverified, personal post, LLM-generated	low

Downgrade if: source >2yr old in fast field, you extrapolated beyond claims, source expresses uncertainty.

See commands/references/confidence-rules.md for full rules.

7. Cook pages

Call cook for each page with all fields populated. Include the raw file path in sources.

Page creation order matters:

1. Entities first (they're referenced by other types)
2. Concepts next
3. Comparisons (reference the concepts they compare)
4. Summaries last (reference everything above)

8. Finalize

1. vibes to check graph health after all pages are written
2. Fix any issues found (broken links, one-way contradictions)
3. tidy to regenerate index
4. yap("END INGEST — created N pages from M raw files")