Harness Supply Chain Audit

Harness Supply Chain Audit

6-factor dependency risk evaluation adapted from Trail of Bits security skill patterns. Surfaces dependency risk flags for human review — not automated verdicts.

When to Use

• Before a major release to assess dependency risk
• After adding new dependencies
• During security audits or compliance reviews
• When on_milestone trigger fires (part of release gate)
• NOT as a replacement for npm audit — this complements it with risk signals beyond CVEs
• NOT for license compliance (separate concern)

Iron Law

Present findings as flags for human review, never as verdicts. A dependency flagged as "high risk" may be entirely appropriate for a project. The skill surfaces signals; humans decide whether to act.

---

Process

Phase 1: INVENTORY — Build Dependency List

1. Resolve project root. Use the path argument or default to the current directory.

2. Detect lockfile. Check for the following in order:

   • package-lock.json (npm)
   • pnpm-lock.yaml (pnpm)
   • yarn.lock (yarn)
   • If none found: report "No lockfile detected. Run npm install first." and stop.

3. Parse direct dependencies from package.json:

   • Read dependencies and devDependencies
   • Build a list: { name, version, isDev }

4. Parse transitive depth from lockfile:

   • For package-lock.json: read packages keys to extract the dependency tree. Nesting depth of node_modules/ segments in keys indicates transitive depth.
   • For pnpm-lock.yaml: read importers section for direct dependencies (keyed by workspace path, e.g., . for root). Each importer lists dependencies and devDependencies with version specifiers. Read packages section for resolved versions — keys are package identifiers (e.g., /@scope/[email protected]) with resolution (tarball URL + integrity hash) and dependencies sub-map for transitives.
   • For yarn.lock: parse block-format entries. Each block header is "pkg@version-range": followed by indented fields: version (resolved), resolved (tarball URL), integrity (hash), and dependencies sub-block listing transitive deps as "name" "version-range" pairs.
   • Assign each package a depth (0 = direct, 1 = first-level transitive, etc.)
   • Flag packages with depth > 5 for transitive risk evaluation

5. Build inventory table:

   INVENTORY: <project-name>
   Direct dependencies: N
   Dev dependencies: N
   Total packages (including transitives): N
   Deep transitive packages (depth > 5): N
   

6. Proceed to EVALUATE.

---

Phase 2: EVALUATE — Score Dependencies on 6 Factors

For each direct dependency (and any transitive with depth > 5), score on 6 factors:

Network access required: npm registry (https://registry.npmjs.org/<pkg>) and GitHub API (https://api.github.com/repos/<owner>/<repo>).

• If npm registry returns 404: mark as "unresolvable", flag for manual review, skip remaining factors
• If GitHub API rate limits hit: score maintenance-status as "unknown", continue with other factors
• If no GitHub repo link in package metadata: skip maintenance-status factor, note in report

Factor 1: Maintainer Concentration

• Fetch: GET https://registry.npmjs.org/<pkg>
• Check: maintainers array length
• Score:
  • High risk: 1 maintainer (bus factor = 1)
  • Medium risk: 2-3 maintainers
  • Low risk: 4+ maintainers

Factor 2: Maintenance Status

• Source: npm time field (last publish date) + GitHub API commit activity
• npm: GET https://registry.npmjs.org/<pkg> → time.modified
• GitHub: GET https://api.github.com/repos/<owner>/<repo>/commits?per_page=1 → latest commit date
• Score:
  • High risk: Last publish > 12 months ago AND no GitHub commits in 6 months
  • Medium risk: Last publish > 12 months ago OR no commits in 6 months (not both)
  • Low risk: Active in both dimensions

Factor 3: Popularity Signal

• Fetch: GET https://api.npmjs.org/downloads/point/last-week/<pkg>
• Score:
  • High risk: < 1,000 weekly downloads
  • Medium risk: 1,000–10,000 weekly downloads
  • Low risk: > 10,000 weekly downloads
  • Note: Low popularity is a signal, not a verdict — internal/niche packages are expected to be low

Factor 4: Install Scripts

• Read: node_modules/<pkg>/package.json (or lockfile-resolved path) → scripts field
• Check for: preinstall, postinstall, install, preuninstall, postuninstall
• Score:
  • High risk: Any install script present
  • Low risk: No install scripts
  • Note: Some install scripts are legitimate (native addon compilation). Flag for review.

Factor 5: Known CVEs

• Run: npm audit --json or pnpm audit --json
• Parse: map findings to their package name
• Score:
  • Critical: Any high/critical severity CVE
  • Medium risk: Moderate severity CVE
  • Low risk: No CVEs or low severity only

Factor 6: Transitive Risk

• Source: Lockfile depth analysis from INVENTORY phase
• Score:
  • High risk: Depth > 5 AND subtree size > 20 transitive packages
  • Medium risk: Depth > 5 OR subtree size > 20
  • Low risk: Depth ≤ 5 and subtree size ≤ 20

Risk Scoring

Combine factor scores into an overall risk level:

Overall Risk	Condition
Critical	Factor 5 is Critical (any high/critical CVE)
High	2+ factors scored High, OR Factor 1 is High + Factor 2 is High
Medium	1 factor scored High, OR 3+ factors scored Medium
Low	All factors Low or at most 1 Medium

---

Phase 3: REPORT — Generate Risk Summary

1. Produce risk summary table sorted by overall risk (Critical first):

   Supply Chain Audit: <project-name>
   Date: <ISO date>
   Packages evaluated: N direct + M deep transitives
   
   ┌─────────────────────┬──────────┬────────────┬─────────────┬────────────┬──────┬─────────────┐
   │ Package             │ Version  │ Maintainers│ Last Publish│ Downloads  │ CVEs │ Overall Risk│
   ├─────────────────────┼──────────┼────────────┼─────────────┼────────────┼──────┼─────────────┤
   │ example-pkg         │ 1.2.3    │ 1 (HIGH)   │ 18mo (HIGH) │ 500 (MED)  │ none │ HIGH        │
   │ another-pkg         │ 2.0.0    │ 12         │ 2mo         │ 50k        │ 1 mod│ MEDIUM      │
   └─────────────────────┴──────────┴────────────┴─────────────┴────────────┴──────┴─────────────┘
   

2. Detail section for Critical and High risk packages:

   HIGH RISK: [email protected]
   ├── Maintainer concentration: 1 maintainer (bus factor = 1)
   ├── Maintenance status: Last publish 18 months ago, no commits in 12 months
   ├── Popularity: 500 weekly downloads
   ├── Install scripts: none
   ├── Known CVEs: none
   └── Transitive risk: depth 2, subtree 4 packages
   Recommendation: Consider replacing with a well-maintained alternative,
   or pin the version and monitor for abandonment.
   

3. Install script warnings (any package with install scripts):

   INSTALL SCRIPTS DETECTED:
   - [email protected]: postinstall — native addon compilation (likely legitimate)
   - [email protected]: postinstall — review script contents before trusting
   

4. Summary line:

   RESULT: 1 Critical, 2 High, 3 Medium, N Low — Review flagged items before release
   

5. Output: Print report to stdout. If --output <file> was passed, also write to that file.

---

Gates

• Stop if no lockfile. Do not evaluate without a lockfile — results will be unreliable.
• Present as flags, not verdicts. Never state "this package is unsafe." State "this package has signals that warrant review."
• Do not block on API failures. If npm registry or GitHub API is unavailable, note which factors were skipped and continue with available data.

Harness Integration

• harness validate — Run after creating the skill files to verify they are properly placed.
• Triggers: on_milestone fires this skill as part of the milestone completion checklist.
• Depends on: harness-security-scan — run after mechanical scanning to complete the security picture.
• Output: Stdout report, optionally written to file via --output. No state files written.

Evidence Requirements

When reporting findings, cite the source for each factor:

• Maintainer data: registry.npmjs.org/<pkg> → maintainers field
• Publish date: registry.npmjs.org/<pkg> → time.modified
• Downloads: api.npmjs.org/downloads/point/last-week/<pkg>
• Install scripts: node_modules/<pkg>/package.json → scripts
• CVEs: npm audit --json output
• Depth: lockfile analysis

Do not assert risk scores without citing the specific data point that generated the score.

Success Criteria

• Running /harness:supply-chain-audit on a project with dependencies outputs a risk table with all 6 factors scored
• A dependency with a sole maintainer and no commits in 12 months scores "high risk"
• A dependency with a postinstall script is flagged in the install scripts section
• API failures produce "unknown" scores with a note, not errors that stop the audit
• All findings are framed as flags for human review, not automated verdicts

Escalation

• If a critical CVE is found: Surface immediately — do not bury it in the table. Recommend blocking the dependency update or requiring an immediate patch before merge.
• If all maintainers are unresponsive: Flag the package as abandoned and recommend finding an alternative. Include download counts to help the user assess how widely adopted the package is.
• If an install script has unknown behavior: Do not guess. State that the script requires manual review and link to the script source.
• If the npm or GitHub API is unavailable: Note which factors were skipped with "unknown" scores. Do not fail the audit — partial results are better than none.
• If the user asks for a verdict ("is this safe?"): Decline to give a binary answer. Supply chain risk is probabilistic. Present the risk signals and let the human decide.

Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

Rationalization	Why It Is Wrong
"This package has high risk signals but it is widely used, so it must be safe"	The Iron Law: present findings as flags for human review, never as verdicts. Popularity does not eliminate bus-factor risk or maintenance abandonment.
"The npm API returned an error for this package, so I will skip it and move on"	API failures produce "unknown" scores with a note, not skips. Partial results with noted gaps are always better than incomplete audits.
"The install script is probably just native addon compilation, so I do not need to flag it"	Every install script must be flagged in the report. "Probably legitimate" is exactly the assumption that supply chain attacks exploit.

Examples

Supply Chain Audit: my-project
Date: 2026-03-31
Packages evaluated: 24 direct + 3 deep transitives (depth > 5)

CRITICAL (1):
  [email protected] — CVE-2021-23337 (high severity, unpatched)

HIGH (2):
  [email protected] — sole maintainer, last publish 22 months ago
  [email protected] — sole maintainer, postinstall script detected

MEDIUM (3):
  [email protected] — 800 weekly downloads (low popularity signal)
  ...

LOW (18): no significant risk signals

INSTALL SCRIPTS:
  [email protected] — postinstall (native compilation, likely legitimate)
  [email protected] — postinstall (REVIEW: contents unknown)

RESULT: 1 Critical, 2 High, 3 Medium, 18 Low
Next steps: Update lodash to patch CVE. Review sketchy-helper postinstall script.
Consider alternatives to abandoned-util.

Example Output

Supply Chain Audit: my-project
Date: 2026-03-31
Packages evaluated: 24 direct + 3 deep transitives (depth > 5)

CRITICAL (1):
  [email protected] — CVE-2021-23337 (high severity, unpatched)

HIGH (2):
  [email protected] — sole maintainer, last publish 22 months ago
  [email protected] — sole maintainer, postinstall script detected

MEDIUM (3):
  [email protected] — 800 weekly downloads (low popularity signal)
  ...

LOW (18): no significant risk signals

INSTALL SCRIPTS:
  [email protected] — postinstall (native compilation, likely legitimate)
  [email protected] — postinstall (REVIEW: contents unknown)

RESULT: 1 Critical, 2 High, 3 Medium, 18 Low
Next steps: Update lodash to patch CVE. Review sketchy-helper postinstall script.
Consider alternatives to abandoned-util.