define-taxonomy

Define Taxonomy

Multi-level classification scheme. Categories at the top, tags or sub-categories beneath.

When to use

• "I want a hierarchy, not a flat list."
• "Categories, tags, AND sub-categories."
• "Organize this corpus the way Wikipedia does — top-level, then subsections."

Design choices to confirm first

• Depth: 2 levels (category → sub) or 3 (category → tag → sub)?
• Exclusivity: is a doc in exactly one category (tree)? Or can it live under multiple categories with tags as a free set (facet)?
• Target counts: how many top-level? How many per level? (Typical: 8-15 top, 3-8 per branch.)

Procedure

1. Top level via suggest-categories: produce N top-level categories with a stratified sample.
2. Sub-classify each branch:
   • For each top-level category, filter the corpus to documents confidently assigned to it (run a provisional categorize-corpus pass).
   • Recursively call suggest-categories on each subset with appropriate k.
   • Small branches (<20 docs) stop — don't sub-divide.
3. Tags (if multi-facet): separately derive a flat tag vocabulary from high-signal noun phrases / NER output (ner-extraction + keyphrase extraction via KeyBERT or YAKE). Tags cross-cut categories.
4. Coherence check: for each branch, ask a cheap LLM: "Do these sub-category labels make sense as children of ? Suggest merges/splits." One call per parent.
5. Export:

   taxonomy:
     - category: Infrastructure
       definition: ...
       sub_categories:
         - name: Containers
           examples: [...]
         - name: CI/CD
     - category: AI & ML
       ...
   tags:  # facets, orthogonal to tree
     - language/python
     - language/go
     - status/wip
     - stage/production
   

6. Apply: feed back through categorize-corpus (category) + tag-assignment pass.

Pitfalls

• Trees look tidy but most real corpora are facet-shaped. Don't force a tree when the user's data is multi-dimensional (e.g. repos have language × domain × stage — three facets, not a single tree).
• Over-depth: 3 levels × 10 branches × 5 subs = 500 leaf nodes. Nothing useful sits in most of them. Stop when branches are thin.
• Don't invent sub-categories to fill out symmetry. A category with one natural sub is fine — just leave it flat under that branch.

Cost control

Same as suggest-categories — this is a tree of suggest-categories calls, each on a filtered subset. Embedding work dominates; LLM labeling is per-cluster, not per-doc.