From qa-data-notebooks
Validate Jupyter notebooks via the `pytest --nbval` plugin - re-execute cells and compare outputs to stored results. Cover the strict path (output match required), `--nbval-lax` (failure-only), `--sanitize-with` for dynamic outputs, and per-cell controls (`#NBVAL_SKIP`, `#NBVAL_IGNORE_OUTPUT`, `#NBVAL_RAISES_EXCEPTION`).
How this skill is triggered — by the user, by Claude, or both
Slash command
/qa-data-notebooks:nbval-testsThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
nbval is a pytest plugin that validates Jupyter notebooks by
nbval is a pytest plugin that validates Jupyter notebooks by re-executing cells and comparing outputs against stored results, "ensuring that the notebook is behaving as expected and that changes to underlying source code haven't affected the results" per the nbval docs.
pip install nbval pytest
Per the nbval docs.
pytest --nbval my_notebook.ipynb
Re-executes every cell; fails if any output differs from stored.
pytest --nbval-lax my_notebook.ipynb
"Collects notebooks and runs them, failing if there is an error" -
skips output comparison unless cells bear the #NBVAL_CHECK_OUTPUT
marker per the nbval docs. Use as the default for tutorials where
output is incidental and execution is what matters.
Add comments at cell start:
| Marker | Effect |
|---|---|
# NBVAL_SKIP | Cell not executed during testing |
# NBVAL_IGNORE_OUTPUT | Cell runs; output diff ignored |
# NBVAL_CHECK_OUTPUT | Force output checking (lax mode) |
# NBVAL_RAISES_EXCEPTION | Validate that the cell raises |
Cell tags (in notebook metadata, lowercase-with-dashes:
nbval-skip, nbval-ignore-output, etc.) are equivalent and
recommended for non-Python kernels.
For timestamps, UUIDs, memory addresses:
pytest --nbval my_notebook.ipynb --sanitize-with sanitize.cfg
sanitize.cfg:
[regex1]
regex: \d{1,2}/\d{1,2}/\d{2,4}
replace: DATE-STAMP
[regex2]
regex: 0x[0-9a-fA-F]+
replace: MEMORY-ADDR
[regex3]
regex: \d+\.\d+(?:e-?\d+)?
replace: NUMBER
Tune carefully - over-sanitizing makes nbval miss real regressions.
# Whole notebooks/ directory
pytest --nbval notebooks/
# Filter by name
pytest --nbval notebooks/ -k "tutorial"
# Single notebook + verbose
pytest --nbval --verbose notebooks/intro.ipynb
# GitHub Actions
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install
run: |
pip install -r requirements.txt
pip install nbval pytest
- name: Run notebook tests (lax)
run: pytest --nbval-lax notebooks/ --sanitize-with sanitize.cfg
For tutorial repos, lax mode + sanitize is usually right.
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Use strict mode for tutorial notebooks | Every random seed change fails CI | Use --nbval-lax (Step 3) |
Skip cells liberally with # NBVAL_SKIP | Coverage shrinks; notebook becomes untested | Use # NBVAL_IGNORE_OUTPUT instead - still verifies execution |
| Sanitize all numeric output | Real regressions hidden | Targeted regexes (Step 5) |
| Run nbval against notebooks that mutate disk/state | Tests become flaky | Use ephemeral working dirs; monkeypatch.chdir(tmp_path) |
No requirements.txt pinning | "Works on author's machine"; CI fails on minor lib bumps | Pin notebook deps separately from prod deps |
.ipynb file - git diffs
for output cells are noisy. Use nbstripout only when sanitize
npx claudepluginhub testland/qa --plugin qa-data-notebooksProvides UI/UX resources: 50+ styles, color palettes, font pairings, guidelines, charts for web/mobile across React, Next.js, Vue, Svelte, Tailwind, React Native, Flutter. Aids planning, building, reviewing interfaces.
Searches MemPalace before answering questions about past work, people, projects, or prior decisions. Returns verbatim stored content instead of guessing from model memory.