oss-audit

OSS Audit

Audit the target repository before making broad changes. Prefer evidence over guesses and produce a report that another contributor can act on without re-discovery.

Context: $ARGUMENTS

Input Contract

• Input: repository root or subdirectory, plus any user constraints such as "no new deps", "keep Python 3.10", or "CI only".
• Required discovery: languages, package managers, entry points, current verification commands, docs coverage, licensing and citation state, reproducibility surface, and operational surface.
• Default: if no path is provided, audit the current working directory.

Output Contract

Create or update OSS_AUDIT.md in the repository root. The report must contain:

1. Repository summary: stack, entry points, package/build system, current automation, licensing and reproducibility status, and major risks.
2. Findings grouped by:
   • Correctness
   • Maintainability
   • Testability
   • Security
   • Performance
   • Observability
   • Documentation
3. For every finding: priority (P0, P1, or P2), evidence, why it matters, and the smallest credible fix.
4. A required file-level change list with one row per path to add or modify.
5. A short "do first / do later" summary.

The file-level change list is mandatory. Use this shape:

| Priority | Path | Action | Reason |
|----------|------|--------|--------|
| P0 | tests/test_cli.py | add | No automated protection for main user flow |
| P1 | .github/workflows/ci.yml | add | No blocking CI on pull requests |
| P1 | CITATION.cff | add | Paper-code repo has no clear citation path |
| P2 | SECURITY.md | add | Missing disclosure and support policy |

Also return a short chat summary with the highest-priority blockers.

Non-goals

• Do not perform broad refactors during the audit.
• Do not rewrite README or add tooling configs yet unless the user explicitly asks.
• Do not invent runtime behavior that you cannot verify from code, docs, or commands.

Workflow

Step 1: Map the repository

• Identify the main language(s), package manager(s), and executable entry points.
• Find existing automation: tests, linters, type checks, formatters, CI, docs, release files.
• Read the highest-signal files first: README*, CONTRIBUTING.md, LICENSE*, CITATION.cff, package manifests, lockfiles, Makefile, build scripts, notebooks, experiment configs, and top-level source entry points.

Step 2: Probe the current verification surface

• Try to locate existing commands for install, run, lint, and test.
• For paper or benchmark code, also locate the smallest reproduce, evaluate, or smoke-run command that does not require private assets.
• If commands are documented, prefer those exact commands.
• If commands are missing or broken, record that as evidence instead of guessing silently.

Step 3: Inspect the seven audit categories

For each category, look for concrete repo evidence:

• Correctness: missing validation, crash-prone scripts, hidden assumptions, stateful side effects, notebook or experiment flows that only work in one execution order.
• Maintainability: tangled structure, duplicated logic, missing boundaries, inconsistent naming/config, no clear mapping from public claims to runnable code paths.
• Testability: no test harness, poor seams, reliance on live services, no deterministic fixtures, no tiny reproducibility sample for research code.
• Security: secrets risk, unsafe shelling out, unchecked input, missing dependency/update policy, private dataset/model paths or embedded credentials.
• Performance: obviously expensive hot paths, repeated I/O, unbounded loops, wasteful startup work.
• Observability: no logs, no error context, no exit codes, no troubleshooting guidance.
• Documentation: weak setup/run docs, missing contribution expectations, missing support/security info, missing LICENSE, citation guidance, or reproduction instructions.

Step 4: Prioritize

Use this rubric:

• P0: blocks safe use, safe change, or safe review right now.
• P1: materially improves contributor velocity or repo reliability this iteration.
• P2: useful hardening work that can wait until the basics are stable.

Prefer changes that improve multiple categories at once.

Step 5: Write the report

Keep the report executable. Every recommendation should imply a concrete next file or command.

Anti-patterns

• Do not modify code, configs, or docs during the audit unless the user explicitly changed scope.
• Do not report findings without concrete evidence: file path, command output, or missing artifact.
• Do not produce an unprioritized laundry list; every finding must be P0, P1, or P2.
• Do not invent runtime behavior that cannot be verified from code, docs, or commands.
• Do not skip categories because the repo appears simple; record "no findings" explicitly when clean.

Self-check

Before declaring this stage complete, verify:

• OSS_AUDIT.md exists in the repo root.
• It contains a repository summary, all seven audit categories, a file-level change table, and a "do first / do later" section.
• Every finding includes priority, evidence, why it matters, and the smallest credible fix.
• Any failed command, unclear stack detail, or private-data blocker is recorded explicitly.

Failure Handling

• If the repository is too large to audit fully, state the scope you audited and why.
• If a command fails, capture the failing command and observed error in OSS_AUDIT.md, then continue with static inspection.
• If the stack is unclear, list the ambiguity under assumptions and avoid speculative recommendations.
• If the repo depends on private data, checkpoints, or credentials, record that as a release blocker and recommend the smallest public-safe fixture or dry-run path.

Done Criteria

• OSS_AUDIT.md exists and contains the repository summary, all seven audit sections, a prioritized file-level change table, and a "do first / do later" summary.
• Every finding in OSS_AUDIT.md includes P0, P1, or P2, evidence, impact, and the smallest credible fix.
• Any release-readiness blocker related to licensing, citation, reproducibility, private assets, or failed commands is recorded explicitly.