research-engine

Research Engine

Use this skill to run the research-engine workflow in Codex. The Claude Code slash commands in commands/ are the canonical source for the full pipeline; this skill adapts the same contract to Codex tools.

Outputs

Create or update a session under:

research/YYYY-MM-DD-<slug>/
├── README.md
├── sources.json
├── intent.json
├── session.md          # follow-ups only
├── transcript.md       # YouTube only, when available
└── related/*.md        # when related sources exist

Use Korean for synthesized findings and answers unless the user asks for another language. Keep direct quotes in the original language.

New Research Workflow

1. Parse the request into target, optional --yes, optional --fresh, and optional --slug <name>.
2. Classify the target with scripts/classify_url.sh.
3. Preview the target:
   • YouTube: verify yt-dlp; fetch metadata and transcript with scripts/yt_fetch.sh. For the lightweight preview, captions are preferred; if captions are absent, captions attempts Groq Whisper fallback from GROQ_API_KEY (env or ~/.config/research-engine/) and otherwise returns partial status. For visual/demo/tutorial focus or missing transcript, run scripts/yt_fetch.sh frames <url> <cache>/frames and Read the JPEG paths in frames.json. Full analysis (the youtube-adapter, or its direct equivalent on Codex) is AV-first: download media once with yt_fetch.sh media, always run frames + transcribe (Whisper primary transcript) from the local file, and use captions --captions-only into a separate captions/ directory only to cross-check proper nouns/numbers/terms against the Whisper transcript.
   • arXiv: fetch title, abstract, and body. Prefer https://arxiv.org/html/<id>; fall back to PDF.
   • GitHub: use gh repo view when authenticated; otherwise fetch public repo metadata and README from the web.
   • Blog/docs/community URL: fetch the page as markdown or readable text.
   • Topic: perform web search and keep a source pool broad enough for cross-checking.
4. Resolve intent. With --yes, infer {purpose, focus, audience_level, notes} and mark intent_mode: "assumed". Without --yes, ask 1-3 concise questions before continuing.
5. Create research/YYYY-MM-DD-<slug>/, save intent.json, and gather evidence.
6. Use the adapter contract in lib/adapter_contract.md as the internal evidence shape. When a task is broad enough and the user permits parallel agents, dispatch focused subagents matching agents/*-adapter.md; otherwise do the same evidence gathering directly with Codex tools.
7. Synthesize README.md with the section contract in lib/report_sections.md. Every factual claim in 핵심 포인트, 상세 분석, and 인용 / 원문 must have a precise [n] citation.
8. Write sources.json with one-indexed sources, original input, input type, intent, and creation timestamp.
9. If Notion credentials are configured, run scripts/push_to_notion.sh <report_dir>, store output_notion_url in sources.json, and add the Notion line to README.md. If not configured, skip silently.
10. Auto-ingest this run into the LLM wiki (parity with commands/research.md Step 7.6). Fold the just-finished session into the durable wiki immediately, so it never lives only as a raw research/<slug>/ artifact. Runs automatically; skip silently if WIKI_AUTO_INGEST=0 or no vault resolves; never fail the research run on a wiki error (artifacts are already persisted).
    • Resolve + bootstrap (note: target the resolved vault, not local wiki/):

      [ "${WIKI_AUTO_INGEST:-1}" = "0" ] && echo "wiki auto-ingest disabled" && exit 0
      node lib/wiki/vault_resolve.mjs --explain      # inspect "ok"
      VAULT="$(node lib/wiki/vault_resolve.mjs)"      # "ok": false → skip silently (no vault)
      mkdir -p "${VAULT}"/{concepts,entities,synthesis,ephemeral,_drafts,_todos,_index}
      [ -f "${VAULT}/AGENTS.md" ] || cp lib/wiki/AGENTS.template.md "${VAULT}/AGENTS.md"
      [ -f "${VAULT}/index.md" ] || printf '# Wiki Index\n' > "${VAULT}/index.md"
      

    • Ingest this run's <slug> via the "Wiki Workflow" ingest contract below, but write to the resolved ${VAULT}: read research/<slug>/README.md + sources.json + ${VAULT}/index.md + ${VAULT}/AGENTS.md → one ${VAULT}/_index/plan-<slug>.json (constitution-conformant pagePlan; links only to catalog-real slugs) → apply once:

      node lib/wiki/apply.mjs --vault "${VAULT}" --plan "${VAULT}/_index/plan-<slug>.json" --date <today>
      

      The log.md exact-match dedup guard makes re-running /research for the same slug a no-op. Capture created/merged counts for the final response.
    • Mirror the verbatim report (parity with commands/research.md Step 7.6.3): unless WIKI_MIRROR_REPORT=0 or the vault did not resolve, copy the full README verbatim into ${VAULT}/reports/ with a Korean human-readable filename:

      [ "${WIKI_MIRROR_REPORT:-1}" = "0" ] || node lib/wiki/report_mirror.mjs --vault "${VAULT}" --research-dir "research/<slug>" --date <today>
      

      Body unchanged; frontmatter augmented with tags: [ai-generated, research-report], report_slug, source. Idempotent per report_slug; the reports/ folder is isolated from the concept/entity index + lint. Never fail the run on a mirror error. Capture the mirrored file path for the final response.
11. Final response: report path plus a two-line TL;DR. If the wiki step ran, append one line: 📚 LLM Wiki: {created}개 생성 / {merged}개 병합 → ${VAULT} (or wiki: skipped when no vault). If the verbatim mirror ran, append one more line: 📄 Report (verbatim): ${VAULT}/reports/<file>.

Follow-Up Workflow

1. Resolve the session from --slug <name> or scripts/find_latest_session.sh research.
2. Read the session README.md, sources.json, intent.json, and existing session.md.
3. Decide whether the question is answerable from existing sources or needs 1-2 new focused fetches.
4. Answer in Korean with existing [n] citations; append new sources to sources.json only when newly fetched evidence is used.
5. Append the exchange to session.md using the format in commands/research-followup.md.
6. Push to Notion if configured and report the session.md path.

Quality Bar

• Prefer primary sources over summaries whenever available.
• Do not pad related sources; include only sources with a specific relationship to the target.
• Remove unsupported claims instead of leaving uncited statements.
• For academic inputs, separate method, experiments, and author-stated limitations as described in lib/report_sections.md.
• Record partial failures in the report instead of aborting the whole run.
• Before claiming completion, verify that README.md, sources.json, and intent.json exist and citations in README.md correspond to entries in sources.json.

Wiki Workflow (Codex 패리티)

commands/wiki.md가 정본. Codex에서 동일 계약으로:

1. wiki/ 없으면 생성 + lib/wiki/AGENTS.template.md → wiki/AGENTS.md 복사 + 빈 index.md.
2. ingest: research/<slug>(raw 불변) + wiki/index.md 카탈로그 읽기 → 헌법대로 pagePlan JSON(tldr/perspective/links, links는 카탈로그 실재 slug만) 생성 → node lib/wiki/apply.mjs --vault wiki --plan <tmp> --date <today> 1회.
3. lint: node lib/wiki/lint.mjs --vault wiki.
4. query: grep/카탈로그 후보 → 위키 페이지에서 인용 합성(읽기 전용). --file 환류·lint --fix 자동수정은 MVP out-of-scope(후속).
5. publish: scripts/wiki_publish.sh. 한글 합성, raw 절대 수정 금지, 무출처 주장 금지, slug ASCII.