agent-memory

agent-memory

A persistent memory layer for Claude Code (and any MCP-compatible agent) that captures every user prompt and tool_call from your coding sessions, exposes them for recall via an MCP server + FastAPI, and feeds them back into a local fine-tune pipeline that produces project-specific tool-calling LoRAs. Built locally, runs locally — Postgres + pgvector, FastAPI on 127.0.0.1:3377, GGUFs in LM Studio. The hooks are the recorder; the mem_* tables are the journal; the fine-tune pipeline is what turns that journal into a model that actually knows your codebases.

System overview

Claude session ──► hooks (UserPromptSubmit, PreToolUse, PostToolUse, SessionStart/End)
                          │
                          ▼
                   FastAPI ingest (port 3377, Bearer-token auth)
                          │
                          ▼
                   Postgres (mem_user_prompts, mem_tool_calls, mem_sessions,
                             mem_projects, mem_observations, mem_lessons)
                          │
            ┌─────────────┴──────────────┐
            ▼                            ▼
   MCP recall surface          fine-tune pipeline
   (search, timeline,          (build_v3_dataset.py)
    get_observations,                  │
    save_memory, lessons)              ▼
                              LoRA train → merge → GGUF
                                       │
                                       ▼
                                  LM Studio

The write path (hooks → FastAPI → queue → Postgres) is fire-and-forget and never blocks Claude. The read path (MCP tools) uses Reciprocal Rank Fusion across pgvector and Postgres FTS. The training path reads directly from mem_tool_calls joined to mem_user_prompts and emits Qwen-format chat datasets.

Repository structure

Path	Purpose
app/	FastAPI service (lifecycle, routes, middleware, queue worker, redaction, auth)
app/routes/	REST endpoints by domain: health, observations, sessions, admin, lessons, prompts, tool-calls
mcp_server.py	Stdio MCP server. Own DB pool + embedding model, zero FastAPI dependency
hooks/	Claude Code Node.js hooks (UserPromptSubmit, Pre/PostToolUse, SessionStart/End, ensure-services)
scripts/migrations/	Versioned SQL migrations (001-initial-schema through 013-project-consolidation)
scripts/fine_tune/	Training pipeline: dataset builders, validator, smoke tests, GGUF verify, the wizard
scripts/backfill/	Backfill tool_calls + prompts from Claude JSONL session logs
scripts/ (root)	install_backup_schedule.sh, backup.sh, run_migrations.py, install-agent-memory-*.sh
models/	Base/LoRA/merged/GGUF artifacts. Symlinked to Dropbox cold storage. Gitignored
data/	Postgres backups + processed datasets (processed/qwen25_tools/v2/, processed/qwen3_tools/v3/). Gitignored
tests/	pytest API integration tests + tests/fine_tune/ validator/dataset tests + real-world A/B harnesses
docs/	fine_tune/ (V3_PLAN, V2_DATA_PIPELINE_PLAN, FAILURE_MODES, WIZARD, PIPELINE_RUNBOOK), training_runs/, backups.md, PRIMER.md
hooks/hooks.json	Reference hook registration; copy into ~/.claude/settings.json
install.js	Legacy Node installer (Docker + native PG, MCP register). Still works for fresh installs

Database schema (overview)

All tables are mem_*-prefixed to avoid collisions in a shared Postgres.

Table	Purpose
mem_tool_calls	Every tool_call captured from Claude sessions — input, output preview, success, errors. The training fuel
mem_user_prompts	User prompts that drove the tool_calls. Linked from mem_tool_calls.prev_user_prompt_id (migration 012)
mem_sessions	Claude session identifiers + start/end time + final status
mem_projects	Project identity keyed on git root + remote + branch (post migration 013 consolidation)
mem_observations	Explicit memory observations (semantic notes the agent or user saved). 768-dim pgvector embeddings
mem_observation_queue	Async processing queue for the worker (FOR UPDATE SKIP LOCKED)
mem_lessons	Proactive rules triggered before risky tool calls (Edit/Write/Bash/NotebookEdit)
mem_schema_migrations	Migration history. One row per applied file in scripts/migrations/

Full schema lives in scripts/migrations/*.sql. The current head is 013-project-consolidation.sql. See docs/PRIMER.md for column-level details and docs/fine_tune/V2_DATA_PIPELINE_PLAN.md for the prompt↔tool_call linkage design introduced by migration 012.

Hooks — how data gets in

Five Node.js hooks live in hooks/. They are designed fire-and-forget (~30-40ms p99) and exit 0 on every error path so a misconfigured or down agent-memory never blocks Claude.