chunk-markdown

Chunk Markdown

Split a Markdown document into semantically meaningful chunks based on heading hierarchy and content classification. Produces numbered files and a manifest index.

When to use

• Markdown file is too long for convenient reading or processing
• Need to organize content by chapter, section, appendix, or index
• Preparing document for downstream processing by section type
• Creating granular pieces for RAG or semantic search indexing

Inputs to gather

• Markdown file path (required)
• Heading depth for split: --depth (optional; defaults to 2, i.e., split at ##)
• Minimum tokens per chunk: --min-tokens (optional; defaults to 200)
• Maximum tokens per chunk: --max-tokens (optional; defaults to 8000)

Procedure

1. Parse the Markdown file, identifying all headings and their hierarchy levels.
2. Split into chunks at the specified heading depth. Do not split headings deeper than the threshold unless necessary to respect token bounds.
3. Recognize and classify special sections using regex patterns on heading text:
   • Front matter, preamble (usually before first main heading)
   • Table of contents / index (headings containing "contents", "index", "toc")
   • Glossary (headings containing "glossary")
   • Appendix (headings containing "appendix", "appendices")
   • Bibliography / references (headings containing "bibliography", "references")
   • Otherwise classify as "chapter" or "section"
4. Estimate token count per chunk using word count multiplied by 1.3 (rough estimate). Do not require external dependencies beyond stdlib or tiktoken if available.
5. Merge chunks smaller than --min-tokens with adjacent sections.
6. Split chunks larger than --max-tokens at lower heading levels if possible.
7. Create output directory chunks/ next to the input Markdown.
8. Write each chunk to chunks/NN-slug.md, where NN is zero-padded sequence and slug is derived from the heading (kebab-case, sanitized).
9. Create chunks/index.toon manifest with one entry per chunk, fields: slug, title, kind (e.g., "chapter", "appendix", "glossary"), tokens, source_lines (start line and end line in original file).

Output / side effects

• chunks/ directory created next to input Markdown
• Numbered Markdown files for each chunk
• chunks/index.toon manifest file with metadata per chunk
• Console output showing chunk count and classification summary

Safety / constraints

• Do not modify the original Markdown file
• Preserve all content; no text should be dropped
• Avoid breaking code blocks or tables across chunks if possible