skill-audit

Skill Audit — Deterministic Marketplace Linter

A pure-Python (stdlib only) linter that walks every SKILL.md, script, asset, and reference under the marketplace and applies regex / structure-based rules. No LLM, no external dependencies, no network — safe to run in CI or as a pre-commit hook.

This skill is the mechanical half of the audit story. Its companion multi-llm-review skill (planned) will handle the semantic half (e.g. detecting that Gulati et al. 2020 is a CytoTRACE v1 citation, not v2 — which a linter cannot know).

When to use

• Before committing changes to any SKILL.md / script / reference.
• In CI — fail PRs that introduce new P0 / P1 issues.
• When onboarding a new plugin or skill — baseline it against the marketplace convention before merging.
• After a multi-LLM review run — verify that the (a)-class deployment-grounded fixes have actually landed.
• Periodically (monthly?) — run with --threshold P2 to surface staleness and drift.

How to use

The script lives at scripts/audit.py in this skill. Invoke directly from the marketplace root:

# Full marketplace audit
python3 meta-skills/skills/skill-audit/scripts/audit.py

# Single plugin
python3 meta-skills/skills/skill-audit/scripts/audit.py \
  --scope plugin --path bfabric-lims

# Single skill
python3 meta-skills/skills/skill-audit/scripts/audit.py \
  --scope skill --path single-cell-spatial-general/skills/seurat-analysis

# One rule only
python3 meta-skills/skills/skill-audit/scripts/audit.py \
  --rule no_plaintext_credentials

# JSON output for CI
python3 meta-skills/skills/skill-audit/scripts/audit.py --json

# Only fail on P0 (P1+P2 advisory)
python3 meta-skills/skills/skill-audit/scripts/audit.py --threshold P0

# List all rules
python3 meta-skills/skills/skill-audit/scripts/audit.py --list-rules

Exit codes

Code	Meaning
0	No findings at or above threshold
1	Findings at or above threshold (default threshold: P1)
2	Invocation error

Severity policy

• P0 — deployment-grounded: security leak, broken example, malformed structure. Should block release.
• P1 — usability / future-fragility: personal paths, broken cross-refs, brittle pins. Should block merge to main.
• P2 — advisory: timestamps, version footers, style. Surfaced but not blocking.

The default --threshold P1 means CI fails on P0+P1, P2 is informational only.

Rule catalog

See references/rule_catalog.md for the full catalog with rationale and provenance per rule. Quick reference:

Severity	Rule ID	What it catches
P0	no_plaintext_credentials	mysql -p<pwd>, password = "...", API keys, Bearer tokens (placeholder-filtered)
P0	frontmatter_required_fields	Missing name / description, mismatched name vs directory, stray (user) / {{...}} artefacts
P0	broken_references	references/X.md mentions that don't resolve (within-skill, sibling-skill, or ../skill/references/ cross-skill)
P0	unsafe_gstore_writes	rm / mv / cp targeting /srv/gstore/ (bypasses gtools safety mediation)
P0	credential_url_mismatch	http:// vs https:// disagreement for fgcz-bfabric.uzh.ch across skills
P1	no_personal_paths	/home/<username>/, Analyses_<Name>/, ps_<name> conda envs
P1	broken_relative_skill_paths	../<skill>/SKILL.md cross-plugin refs (won't resolve under marketplace install)
P1	nodelist_hardcode	#SBATCH --nodelist=fgcz-r-NNN (use --constraint=L40S instead)
P1	inconsistent_api_within_plugin	Seurat v5 slot= vs layer= mix, qs:: vs qs2:: mix in one plugin
P2	version_drift	Skill body **Version**: X.Y.Z footer vs plugin.json version mismatch
P2	stale_timestamp	Last Updated: YYYY-MM-DD older than 365 days

CI integration (suggested)

Add to .gitlab-ci.yml:

audit:
  stage: test
  image: python:3.11-slim
  script:
    - python3 meta-skills/skills/skill-audit/scripts/audit.py --threshold P1
  allow_failure: false

For pre-commit hook (.git/hooks/pre-commit):

#!/bin/bash
exec python3 meta-skills/skills/skill-audit/scripts/audit.py --threshold P0

Default exclusions

The linter default-excludes meta-skills whose own documentation contains rule examples that look like violations:

• meta-skills/skill-audit (this skill — rule catalog documents mysql -p<pwd>, /home/<user>/, etc.)
• meta-skills/multi-llm-review (persona prompts in solo_mode.md describe what to flag)

To audit them anyway (with the self-referential noise), pass --include-self.

What this skill explicitly does NOT catch

The linter is regex-based. It will miss:

• Citation accuracy (CytoTRACE v1 vs v2 paper mix-up — semantic).
• Scientific correctness (wrong threshold, wrong feature-key join — needs domain reasoning).
• API drift beyond the few cases hard-coded (slot=/layer=, qs::/qs2::).
• Working-but-suboptimal code (e.g. anti_join("Name") that deletes the KEEP row — a semantic bug).
• Logic errors in scripts.
• Cross-plugin coordination (e.g. cellranger-fgcz reference paths matching genome-reference-build outputs).

These belong to the multi-llm-review skill (planned). Treat skill-audit as the cheap, fast first pass; multi-llm-review as the expensive, thorough second pass.

Known false-positive shapes

The linter prefers low FP over noisy coverage. Cases where you may need to ignore a finding:

• Placeholder hostnames in prose (fgcz-r-NNN, fgcz-c-XXX): filtered by the rule.
• Tool versions in skill body (Cell Ranger 9.0.0+, Repository: ... | Version: 2.5.5 |): the version_drift rule requires a footer-style **Version**: X.Y.Z so most tool-version mentions are skipped.
• Personal paths in code comments documenting the problem itself: e.g. ## register_custom_analysis.py lives in Ronald's home (/home/rdomi/) — the rule still fires, but the finding is actually valid signal (the path is still real and breaks).

When a true FP is found, the right fix is to refine the rule's regex (in scripts/audit.py), not to add per-file ignores. Per-file ignore comments are not yet supported in this prototype.

See also

• meta-skills/fgcz-context — institutional context that defines the conventions this linter enforces (e.g. gStore POSIX-read-only, gtools mediation, gi_* envs read-only, --constraint=L40S over --nodelist).
• meta-skills/multi-llm-review (planned) — semantic counterpart for cases regex cannot reach.
• references/rule_catalog.md — per-rule rationale, regex pattern, provenance, example.