ai-literacy-assessment

AI Literacy Assessment

Assess a team's AI collaboration literacy level by combining observable evidence from the repository with clarifying questions, then produce a timestamped assessment document and a README badge.

The Assessment Process

Phase 1: Observable Evidence

Scan the repository for signals that indicate which framework level the team is operating at.

Habitat document discovery comes first. Before scanning for any of the level indicators below, apply the methodology in references/habitat-discovery.md to find HARNESS.md, AGENTS.md, and CLAUDE.md — including their alternative paths and embedded forms. A habitat document found at a non-conventional path counts as present for the Level 3 indicators that reference it; "not at the default path" is not the same as "doesn't exist". Every absence claim must come from a fully-completed search across known alternatives, with the discovery report citing what was matched and where.

Each signal below maps to a specific level:

Level 0-1 indicators (awareness + prompting):

• Does the repo exist and contain code? (baseline)
• Are there any AI-related configuration files at all?

Level 2 indicators (verification):

• CI workflows that run tests (*.yml in .github/workflows/)
• Test coverage enforcement (coverage thresholds in CI or build files)
• Vulnerability scanning (govulncheck, OWASP, Docker Scout)
• Markdownlint or other linting in CI
• Mutation testing configuration
• Small, TDD-paced diffs visible in commit history (Human Pace signal)
• Depletion signals recognised — developer can name observable markers of degraded judgment (Depletable Collaborator signal)

Level 3 indicators (habitat engineering):

• CLAUDE.md or equivalent context engineering file (apply references/habitat-discovery.md — alternative paths and embedded forms count as present)
• HARNESS.md with declared constraints (same — alternative paths count)
• AGENTS.md compound learning memory (same)
• Parallel-tool config evidence: .cursor/rules/, .github/copilot-instructions.md, .windsurf/rules/, or custom AI tooling locations expressing harness control through whichever AI surface the team uses. Apply references/tool-config-evidence.md for the methodology. A project with rich parallel-tool configs is at L3 context engineering even without HARNESS.md/CLAUDE.md, but tool-config evidence does NOT signal architectural constraints or compound learning by itself.
• MODEL_ROUTING.md model-tier guidance
• .claude/skills/ project-local skills
• .claude/agents/ custom agent definitions
• .claude/commands/ custom commands
• Hooks configuration (hooks.json)
• REFLECTION_LOG.md with entries
• .markdownlint.json or equivalent config
• Spec-scoped changes constraint in HARNESS.md (Human Pace signal)
• Change cadence drift GC rule active (Human Pace signal)
• Session boundaries designed into workflow — time-based, not task-based (Depletable Collaborator signal)
• Recovery cadence visible in work log — gaps between sessions respected (Depletable Collaborator signal)
• Depletion Check practised — 90-minute time-based self-assessment (Depletable Collaborator signal)

Level 4 indicators (specification architecture):

• specs/ directory with specification files
• Implementation plans (plan.md, plan-*.md)
• Agent pipeline with orchestrator
• Plan approval gate in orchestrator
• Loop guardrails (MAX_REVIEW_CYCLES)
• Spec-to-PR mapping — each spec produces one PR (Human Pace signal)
• Team-level sustainability — spec decomposition accounts for human energy, not just modularity (Depletable Collaborator signal)

Level 5 indicators (sovereign engineering):

• Platform-level tooling (reusable plugins)
• Cross-team harness templates
• OpenTelemetry configuration
• Organisational governance documentation
• Multiple agent teams or cloud async agents
• Change cadence metrics reviewed as team health signal (Human Pace signal)
• Sustainable pace as platform metric alongside cost and quality (Depletable Collaborator signal)
• Agent orchestration policies account for human verification capacity (Depletable Collaborator signal)

Phase 2: Clarifying Questions

After scanning, ask questions to fill gaps that observable evidence cannot answer. Focus on:

• Workflow habits: "Do you write specs before code, or after?"
• Verification practices: "Do you verify AI output systematically or trust it if it looks right?"
• Cost awareness: "Do you know what your AI tools cost per month?"
• Team practices: "Does your team have shared AI conventions, or does each developer work differently?"
• Learning: "Do you capture what you learn from AI sessions?"

Ask 3-5 questions maximum. Each question should disambiguate between adjacent levels.

Phase 3: Assessment Document

Produce a timestamped Markdown document at assessments/YYYY-MM-DD-assessment.md with:

1. Header: team name, date, assessor, assessed level
2. Observable evidence: what was found in the repo (with file paths)
3. Clarifying responses: summary of answers
4. Level assessment: primary level with rationale
5. Discipline maturity: context engineering, architectural constraints, guardrail design — each rated
6. Strengths: what the team does well at their current level
7. Gaps: what's missing for the next level
8. Recommendations: 3-5 specific actions to progress
9. Immediate adjustments applied: what was fixed during this assessment
10. Workflow operation changes: accepted changes to how artifacts are used
11. Reflection: what the assessment itself revealed
12. Next assessment date: suggested quarterly re-assessment

Phase 4: Immediate Habitat Adjustments

After documenting the assessment, identify adjustments that can be made immediately — without changing any application code or requiring team discussion. These are habitat hygiene fixes:

Stale counts: If HARNESS.md Status section shows outdated counts, update them. If README badges show old numbers, update them.

Missing entries: If AGENTS.md GOTCHAS is empty but the assessment revealed gotchas, add them. If REFLECTION_LOG.md has no entries from this assessment, add one.

Drift detection: If HARNESS.md declares constraints that no longer match reality (tools removed, workflows renamed), update the declarations.

Mechanism map staleness: If the README mechanism map is missing components that the scan found (new agents, commands, hooks, skills), update it.

Present each adjustment to the user and apply it immediately. Record what was adjusted in the assessment document.

Phase 5: Workflow Operation Recommendations

Based on the gaps identified, recommend specific changes to how existing workflows and artifacts are operated (not built — the infrastructure exists, it just needs to be used differently):

Operating rhythm: Recommend cadences for harness audits, reflection reviews, mutation score checks, and cost monitoring. Suggest adding these to a calendar or checklist.

Habit formation: Identify which framework habits (from Part VII) are not yet automatic and suggest specific practice exercises.

Artifact activation: Identify artifacts that exist but are not actively used (e.g., AGENTS.md that isn't read at session start, MODEL_ROUTING.md that isn't consulted when dispatching agents) and recommend how to activate them.

Promotion opportunities: Identify unverified HARNESS.md constraints that could be promoted to agent or deterministic with available tooling.

Present each recommendation to the user. For accepted recommendations, apply the change (update CLAUDE.md with new cadences, promote HARNESS.md constraints, add operating notes to AGENTS.md). Record accepted and rejected recommendations in the assessment document.

Phase 5b: Improvement Plan

After workflow recommendations, invoke the literacy-improvements skill with the assessed level and the gaps from section 7 of the assessment document. The skill handles target level selection, plan generation, and interactive execution.

Phase 5 and Phase 5b are complementary:

• Phase 5 = operate better at your current level
• Phase 5b = build toward the next level

The skill records its outcomes (accepted, skipped, deferred) in the assessment document.

Phase 6: Assessment Reflection

Capture a reflection on the assessment itself:

• What did the scan reveal that was surprising?
• Where did observable evidence diverge from the team's self-perception?
• What should future assessments pay attention to?

Append this to REFLECTION_LOG.md as a structured entry.

Phase 7: README Badge

Add or update a badge in the project's README showing the assessed level:

[![AI Literacy](https://img.shields.io/badge/AI_Literacy-Level_N-COLOUR?style=flat-square)](assessments/YYYY-MM-DD-assessment.md)

Colour coding:

Level	Colour	Hex
L0	Grey	808080
L1	Light blue	87CEEB
L2	Blue	4682B4
L3	Teal	20B2AA
L4	Green	2E8B57
L5	Gold	DAA520

Link target: the assessment document, so anyone who clicks the badge sees the full assessment with evidence and rationale.

Scoring Heuristic

The assessed level is the highest level where the team has substantial evidence across all three disciplines. A team with L3 context engineering but L1 verification is assessed at L1 — the weakest discipline is the ceiling.

Level	Minimum evidence required
L0	Repo exists, team is aware of AI tools
L1	Some AI tool usage, basic prompting
L2	Automated tests in CI, systematic verification of AI output
L3	CLAUDE.md + at least 3 harness constraints enforced + custom agents or skills
L4	Specifications before code + agent pipeline with safety gates
L5	Platform-level governance + cross-team standards + observability

Content-shape sophistication adjustments

Surface counts (script count, hook count, agent count, command count) are insufficient on their own. Apply the content-shape methodology in references/sophistication-markers.md before assigning a level. That reference defines simple-vs-sophisticated markers per artefact type and the level adjustments they justify.

The principle: a project with one sophisticated state-based orchestration script is not at the same maturity as one with ten simple bash hooks. Sophistication markers raise the floor on the discipline they evidence (orchestration sophistication → guardrail design; state-based hook sophistication → architectural constraints). The weakest-discipline-is-the-ceiling rule still applies — a single sophisticated artefact does not raise the overall level unless the other disciplines also have evidence at that level.

Every sophistication marker the assessor applies must be cited explicitly in the assessment document — what was found and where — so the level determination is auditable. No silent shifts.

The adjustments are introduced conservatively at this release — prefer surfacing markers without changing previously-assigned levels unless the evidence is unambiguous. As the framework accumulates assessments using the markers, the adjustments will tune.