mongodb-population

MongoDB Population Skill

Installation: If not already installed, add the required packages:

• pnpm add @efesto-cloud/population (for Populate type and normalizePopulate helper)
• pnpm add @efesto-cloud/mongodb-population (for BasePopulator and QueryBuilder classes)

Adds MongoDB population support — typed eager-loading of related documents via aggregation $lookup — to an existing entity. The entity, its DTO, document type, mapper, and repository are assumed to already exist. This skill only patches them where needed and writes the population infrastructure.

Scope: Shape type, QueryBuilder, Populator, plus targeted patches to entity, DTO, document, mapper, and repository interface/implementation.

Does not: create entities or repositories from scratch, write use cases, or manage DI container wiring.

---

Phase 0 — Clarify Intent

If the user has not specified which entity to populate and/or which fields should be populated, use AskUserQuestion to ask:

1. Which entity needs population?
2. Which fields should be populatable, and for each:
   • What is the source collection/entity?
   • Is it a single value (1:1) or an array (1:many)?
   • Does the related entity itself have a populator already? (nested population)

Do not proceed until you have at least the entity name and one field to populate.

---

Phase 1 — Discover the Project Structure

Before touching any file, orient yourself:

1. Find the collection enum — typically src/db/CollectionNameEnum.ts or similar. You'll need the collection name constant for $lookup.
2. Check for existing populators — browse src/repo/shape/, src/repo/populate/, src/repo/query/. If any exist, read one to match the exact import style.
3. Read the target entity — src/entity/FooEntity.ts
4. Read the target DTO — src/dto/IFoo.ts
5. Read the target document — src/db/Documents/FooDocument.ts
6. Read the target mapper — src/mapper/FooMapper.ts
7. Read the repository interface — src/repo/IFooRepo.ts
8. Read the repository implementation — src/repo/impl/FooRepoImpl.ts

If src/repo/shape/ or src/repo/populate/ directories do not yet exist, create them.

---

Phase 2 — Patch Satellite Files

Patch each file only where something is actually missing. Do not rewrite files wholesale.

2a. Entity

For each populated field bar on entity Foo:

• Props type must include an optional field for the populated value:
  • 1:1 → bar: Bar | null (initialized to null in create())
  • 1:many via foreign key on Bar → bars: Bar[] (initialized to [] in create())
• create() static method — if the populated field has a meaningful default, accept it as an optional param. Typically bar is not passed to create() (it starts null/empty and is filled by the mapper after aggregation).
• toDTO() — if the DTO has an optional bar? field, map it: bar: this.props.bar?.toDTO() ?? null.
• Getter — add get bar(): Bar | null (or Bar[]) if missing.
• No populateBar() mutation method needed — the mapper sets the field directly after aggregation.

2b. DTO

For each populated field bar on IFoo:

• Add bar?: IBar | null (optional — it may or may not be present depending on query).
• For 1:many: bars?: IBar[].
• If the DTO lives inside a namespace, add the field to the correct variant.
• If a separate index.ts re-exports the DTO, no change needed there unless you added a new sub-type.

2c. Document

For each populated field bar on FooDocument:

• Add bar?: BarDocument | null (always optional — absent on raw stored documents, present only after $lookup).
• For 1:many: bars?: BarDocument[].
• The FK reference field (bar_id: ObjectId | null) should already be present; do not add a second FK.

2d. Mapper

For each populated field bar, update FooMapper.from():

// After building the base entity:
if (doc.bar) {
    entity.props.bar = BarMapper.from(doc.bar);
}
// or for arrays:
if (doc.bars) {
    entity.props.bars = doc.bars.map(BarMapper.from);
}

The to() direction (entity → document) should not include populated fields — they are loaded, not saved, through this path.

---

Phase 3 — Write Population Core Files

Read the reference files before writing:

• references/query-builder-example.ts — QueryBuilder with populateWith()
• references/populator-example.ts — flat Populator (no nesting)
• references/populator-nested-example.ts — nested Populator delegating to sub-populator
• For Shape type examples, see the population skill's references/shape-example.ts

3a. Shape — src/repo/shape/FooShape.ts

// Leaf fields use `true`; fields whose related entity is also populatable use that entity's Shape type.
import type { BarShape } from './BarShape.js'; // only if Bar also has a populator

export type FooShape = {
    bar: true;           // 1:1, leaf — Bar has no further population
    items: true;         // 1:many, leaf
    baz: BazShape;       // 1:1, nested — Baz itself has populatable fields
};

3b. QueryBuilder — src/repo/query/FooQueryBuilder.ts

import { normalizePopulate, type Populate } from '@efesto-cloud/population';
import { QueryBuilder } from '@efesto-cloud/mongodb-population';
import FooDocument from '~/db/Documents/FooDocument.js';
import FooPopulator from '../populate/FooPopulator.js';
import type { FooShape } from '../shape/FooShape.js';

export default class FooQueryBuilder extends QueryBuilder<FooDocument> {
    populateWith(fields: Populate<FooShape> = {}): this {
        const normalized = normalizePopulate(fields, FooPopulator.SHAPE);
        const pipeline = FooPopulator.buildPipeline(normalized);
        this.push_populate_pipeline(pipeline);
        return this;
    }
}

3c. Populator — src/repo/populate/FooPopulator.ts

For each field:

• 1:1 relationship (Bar lives in its own collection, Foo stores bar_id): use lookup + unwind.
• 1:many relationship (Bar stores foo_id as FK, or Foo stores an array of IDs): use lookup only, no unwind.
• Nested population (Bar itself has a populator): pass a sub-pipeline to the lookup. See references/populator-nested-example.ts.

import { BasePopulator } from '@efesto-cloud/mongodb-population';
import type { NormalizedPopulate } from '@efesto-cloud/population';
import CollectionNameEnum from '~/db/CollectionNameEnum.js';
import type TCollectionName from '~/db/TCollectionName.js';
import type { FooShape } from '../shape/FooShape.js';

export default class FooPopulator extends BasePopulator<FooShape, TCollectionName> {
    static readonly SHAPE: FooShape = {
        bar: true,
        items: true,
    };

    private bar(): void {
        if (!this.markPopulated('bar')) return;
        this.addStages(
            this.lookup({
                from: CollectionNameEnum.bar,   // collection name constant
                localField: 'bar_id',           // FK on Foo document
                foreignField: '_id',
                as: 'bar',
            }),
            this.unwind('bar'),                 // 1:1 — flatten array to single object
        );
    }

    private items(): void {
        if (!this.markPopulated('items')) return;
        this.addStages(
            this.lookup({
                from: CollectionNameEnum.item,
                localField: '_id',              // Foo's own _id
                foreignField: 'foo_id',         // FK on Item documents
                as: 'items',
            }),
            // No unwind — keeps the array
        );
    }

    populate(spec: NormalizedPopulate<FooShape>): this {
        if (spec.bar) this.bar();
        if (spec.items) this.items();
        return this;
    }

    static buildPipeline(spec: NormalizedPopulate<FooShape>): import('mongodb').Document[] {
        return new FooPopulator().populate(spec).build();
    }
}

---

Phase 4 — Patch the Repository

4a. Repository Interface — src/repo/IFooRepo.ts

Add the Options namespace with a populate field, and add options? param to every query method (save/saveMany/delete do not need it):

import type { Populate } from '@efesto-cloud/population';
import type { FooShape } from './shape/FooShape.js';

interface IFooRepo {
    search(query: IFooRepo.Search, options?: IFooRepo.Options): Promise<Foo[]>;
    get(id: ObjectId, options?: IFooRepo.Options): Promise<Maybe<Foo>>;
    findByIds(ids: ObjectId[], options?: IFooRepo.Options): Promise<Foo[]>;
    // ... other query methods
    save(entity: Foo): Promise<void>;
}

namespace IFooRepo {
    export type Options = {
        populate?: Populate<FooShape>;
    };
}

export default IFooRepo;

4b. Repository Implementation — src/repo/impl/FooRepoImpl.ts

Switch each query method to use FooQueryBuilder with .populateWith(options?.populate):

async get(id: ObjectId, options?: IFooRepo.Options): Promise<Maybe<Foo>> {
    const pipeline = new FooQueryBuilder()
        .match({ _id: id } as Filter<FooDocument>)
        .populateWith(options?.populate)
        .limit(1)
        .build();

    const results = await this.coll.aggregate<FooDocument>(
        pipeline, { session: this.db.session }
    ).toArray();

    if (results.length === 0) return Maybe.none();
    return Maybe.maybe(FooMapper.from(results[0]!));
}

Methods that already use aggregate() just need .populateWith(options?.populate) inserted into the builder chain. Methods that use findOne() or find() should be converted to aggregate() with the QueryBuilder.

---

Special Cases

Polymorphic entity (discriminated union)

If Foo has a type discriminator and different variants have different populatable fields:

• The Shape can include all fields across variants: { fontFile: true; rasterFile: true; vectorFile: true; }.
• In the populator, each private method populates only the relevant field — because $lookup on a non-existent FK just returns an empty array, which is then dropped by unwind or ignored.
• Alternatively, if the variant shapes are completely disjoint, create separate Shape types with a union.

Nested population (the related entity also has a populator)

When Bar itself has a BarPopulator, you can pass a sub-pipeline into the $lookup:

private bar(nestedSpec: NormalizedPopulate<BarShape>): void {
    if (!this.markPopulated('bar')) return;
    const nestedPipeline = BarPopulator.buildPipeline(nestedSpec);
    this.addStages(
        this.lookup({
            from: CollectionNameEnum.bar,
            localField: 'bar_id',
            foreignField: '_id',
            as: 'bar',
            pipeline: nestedPipeline,    // <-- sub-population
        }),
        this.unwind('bar'),
    );
}

The Shape field must then be typed as BarShape (not true), and the populate() method receives spec.bar as a NormalizedPopulate<BarShape>.

$lookup with $in (Foo stores an array of IDs)

When Foo.bar_ids is an array of ObjectIds pointing to Bar documents:

this.lookup({
    from: CollectionNameEnum.bar,
    localField: 'bar_ids',   // array field on Foo
    foreignField: '_id',
    as: 'bars',
})
// No unwind — result is an array matching the IDs

Optional relationship (FK can be null)

For bar_id: ObjectId | null, the $lookup returns an empty array when FK is null. unwind() always uses preserveNullAndEmptyArrays: true, so nullable FKs are handled safely with the standard call:

this.addStages(
    this.lookup({ from: CollectionNameEnum.bar, localField: 'bar_id', foreignField: '_id', as: 'bar' }),
    this.unwind('bar'),
);

Then in the mapper: entity.props.bar = doc.bar ? BarMapper.from(doc.bar) : null.

---

Phase 5 — Typecheck

Run the typecheck command for the core package then fix any errors before considering the task done.

---

Reference Files

• references/query-builder-example.ts — Full QueryBuilder
• references/populator-example.ts — Flat populator (leaf fields only)
• references/populator-nested-example.ts — Populator with nested sub-population
• Shape type examples: see population skill's references/shape-example.ts