ingest

Ingest workflow

Process a raw source into the wiki via a 5-phase pipeline. Use $ARGUMENTS as the source file path if provided (e.g., /ingest raw/source-document.md).

Prerequisites — Auto-bootstrap

If the wiki infrastructure is missing, create it before proceeding. This makes /ingest the only command a new user needs.

1. Check for wiki/ directory structure. If any of these are missing, create them:

   • raw/, wiki/, wiki/concepts/, wiki/techniques/, wiki/entities/, wiki/synthesis/
   • .llm-wiki/lenses/ (lens metadata directory)

2. Check for wiki/index.md. If missing, create with empty category scaffolding:

   ---
   title: Wiki Index
   type: index
   created: YYYY-MM-DD
   updated: YYYY-MM-DD
   ---
   
   # Wiki Index
   
   ## Entities
   
   ## Concepts
   
   ### Foundational
   
   ### Derived
   
   ## Techniques
   
   ## Synthesis & Comparisons
   

3. Check for wiki/log.md. If missing, create with initial entry:

   ---
   title: Wiki Log
   type: log
   created: YYYY-MM-DD
   updated: YYYY-MM-DD
   ---
   
   # Wiki Log
   
   ## [YYYY-MM-DD] init | Wiki bootstrapped via first ingest
   
   Created directory structure and initial files.
   

4. Skip wiki/overview.md creation here. Phase 5 will create or update it with real content from the source — no empty placeholder needed.

If anything was created, briefly note it to the user (e.g., "Wiki infrastructure created — proceeding with ingest.") but do not pause for confirmation.

If the wiki already exists, this section is a no-op.

Phase 1: Read & Comprehend

• Verify the source file exists and is readable. If not found, list files in raw/ to help the user find the right path. If raw/ is empty, suggest the user add a source document first.
• Read the full source. For large files (>100KB or >1000 lines), read in logical chunks (sections, chapters) and accumulate context.
• Image awareness: After reading the text, scan for image references (![alt](path), ![[filename]], or HTML <img> tags):
  • If images are found, view each one using the Read tool (which handles image files natively).
  • Note what each image conveys: diagram structure, key labels, relationships, data patterns.
  • Prioritize images that convey structure (flowcharts, concept maps, process diagrams) over decorative images.
  • Record the file paths of substantive images (skip decorative ones). In Phase 4, each relevant wiki page should both describe the visual information in prose AND embed the original image via ![description](path) (see CLAUDE.md "Image handling" for path format).
• Build a mental model: what entities, concepts, and techniques does this source introduce?
• Read wiki/index.md to identify overlaps with existing wiki content.

Phase 2: Discuss with User

• Present 3-7 key takeaways, ordered by significance.
• Flag connections to existing wiki content:
  • "This reinforces X" — strengthens an existing page
  • "This contradicts Y" — note the conflict and which source to trust
  • "This introduces a new angle on Z" — extends existing coverage
• Lens detection: Read .llm-wiki/lenses/ for active lens files (status: active). Then:
  • If active lenses exist and one or more fit the source: Use the AskUserQuestion tool to ask which lens to apply. Options: the best-fit lens with "(Recommended)", any other matching lenses, and "No lens (base domain)". Include a brief description on each option explaining why it fits (or doesn't).
  • If no lens fits and the source is outside the base domain: Use the AskUserQuestion tool to propose a new lens. Options: "Create a new lens" (describe the proposed domain), "No lens (base domain)".
  • If the source fits the base domain: Proceed without mentioning lenses. No friction on the happy path.
  • If a new lens is requested: Draft the lens file (three sections: What to Notice, How to Categorize, Quality Standards), write it to .llm-wiki/lenses/<name>.md with status: draft, and present it to the user for review. Use the AskUserQuestion tool to confirm the draft. Options: "Approve lens", "Revise lens". Pause until approved. Then update status to active.
• Direction confirmation: After presenting takeaways (and resolving any lens choice), use the AskUserQuestion tool to confirm direction. Options: "Proceed as presented (Recommended)", "Adjust emphasis or skip topics". Ask one question at a time — do not batch the lens question and direction question into a single AskUserQuestion call.
• Do not proceed to Phase 3 until the user confirms direction.

Phase 3: Plan the Transform

• List pages to create, each with proposed type and title. If a lens is active, include lens-specific page types and note the lens in the plan (e.g., "Create wiki/datasets/imagenet.md (type: dataset, lens: machine-learning)").
• List existing pages to update, with what's changing.
• Propose cross-reference additions.
• Present the plan in text, then use the AskUserQuestion tool to get approval. Options: "Approve plan (Recommended)", "Adjust plan". If the user chooses to adjust, incorporate their feedback and re-present the updated plan with another AskUserQuestion call.
• Do not proceed to Phase 4 until the user approves the plan.

Phase 4: Write & Integrate

• Create new pages in the correct subdirectory for their type:
  • entity → wiki/entities/
  • concept → wiki/concepts/
  • technique → wiki/techniques/
  • comparison, synthesis, query-answer → wiki/synthesis/
  • Lens-specific types → subdirectory defined in the lens file (create directory if needed)
• Follow the quality standards for each type (see CLAUDE.md "Page quality standards"):
  • entity — background, key contributions, works, See also
  • concept — definition, principles, implications, relationships, See also
  • technique — what/when/how, example, pitfalls, See also
  • Lens-specific types — follow required sections defined in the active lens's "Quality Standards"
• If a lens is active:
  • Add lens: <lens-name> to the frontmatter of every page created during this ingest.
  • Apply the lens's quality standards (required sections/fields) in addition to base standards.
  • Follow the lens's "What to Notice" priorities when deciding what to extract and emphasize.
  • Use domain-prefixed filenames when the page name might collide with another lens's page (e.g., ml-optimization.md).
• Update existing pages — add new information, mark contradictions, strengthen cross-references.
• Add ## See also sections to all new and updated pages.
• Embed source images where they add value (see CLAUDE.md "Image handling"). For each substantive image identified in Phase 1: describe the visual content in prose (so the page stands alone without images), then embed the original with ![description](../../raw/path/to/image.png). Use the correct relative path from the page's subdirectory — wiki content pages are two directory levels below the project root. Skip decorative images. For external URL images, preserve the original URL as-is.
• Apply cross-reference conventions: first-mention linking in body text, bidirectional linking, entity pages as hubs. Link format: [[kebab-case-filename|Display Text]] — the target is always the filename stem you chose when creating the page. Example: if you created wiki/entities/anton-chekhov.md, other pages link with [[anton-chekhov|Anton Chekhov]]. Never use the page title or spaces in the link target.

Phase 5: File & Report

• Update wiki/index.md: add new entries in the correct hierarchical category (Entities, Concepts > Foundational/Derived, Techniques, Synthesis & Comparisons).
• Create or update wiki/overview.md: If this is the first ingest, create the overview from scratch based on this source. Otherwise, revise the existing overview to reflect what this source adds. Keep it short (1-2 screens).
• Append to wiki/log.md (include lens name if one was active):

  ## [YYYY-MM-DD] ingest | Source Title
  or
  ## [YYYY-MM-DD] ingest (lens-name) | Source Title
  
  Source: `raw/path/to/source.md`
  Lens: lens-name (or "base" if no lens)
  
  **New pages:** (list with one-line descriptions)
  **Updated pages:** (list with what changed)
  **Cross-references added:** (count or brief list)
  

• Self-check. Verification happens in two layers:
  • Layer 1 (automatic): The post-write hook (scripts/wiki-check.py) validates each page as you write it — frontmatter, See also, type/directory match, wikilink resolution. If you see [wiki-check] WARNING in hook output, fix the issue immediately before continuing.
  • Layer 2 (from memory): After all pages are written, verify cross-page coherence from your context window (do NOT re-read files from disk):
    • Cross-references are bidirectional (if page A links page B, B's See also includes A).
    • Every new page appears in wiki/index.md.
    • No duplicate index entries.
    • If a lens was active: all new pages have lens field in frontmatter.
    • If substantive images were identified in Phase 1: at least one wiki page embeds an image reference (![...](...)) with a valid relative path to raw/.
  • Fix any Layer 2 issues via Edit. Note all corrections (from both layers) in the report (e.g., "Self-check: 2 issues auto-corrected").
• Report to user: summary of all changes with page links.

Guidelines

• One source at a time unless the user asks for batch processing.
• Prefer updating existing pages over creating new ones when content overlaps.
• Flag contradictions with existing wiki content explicitly — note which source to trust and why.
• Distinguish concept from technique using these tests:
  • Concept test: "Can you understand this without doing anything?" → concept. Concepts are frameworks, principles, taxonomies, mental models.
  • Technique test: "Does this have steps you can follow?" → technique. Techniques are methods, processes, workflows with applicability conditions.
  • When both: If something is both a framework AND a method, create a concept page for the framework and a technique page for the method, cross-linked.
• Every page must have proper YAML frontmatter and a ## See also section.
• For large sources (>100KB): read in sections, process incrementally, but produce the full set of pages in one pass.
• For multi-file sources: ingest as a group, coordinating entity/concept/technique pages across the set.