From add-scrut-cli-tests
Set up scrut snapshot-based CLI integration testing for a CLI project. Use when the user says "add scrut tests", "set up scrut", "add CLI integration tests", "add snapshot tests for the CLI", "wire up scrut", or wants end-to-end snapshot tests for a CLI binary. Detects the project language (Go, Swift, Rust, Zig, Python, Ruby, Shell), creates starter test files under tests/scrut/, adds Makefile targets, and integrates with CI. Pairs with the write-scrut-tests skill for authoring conventions.
How this skill is triggered — by the user, by Claude, or both
Slash command
/add-scrut-cli-tests:add-scrut-cli-testsThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Set up [scrut](https://github.com/facebookincubator/scrut) snapshot-based CLI integration testing for a CLI project with Makefile targets and CI workflow integration.
Set up scrut snapshot-based CLI integration testing for a CLI project with Makefile targets and CI workflow integration.
.github/workflows/ci.yml workflow (or equivalent CI workflow file) is recommended for CI integrationIdentify the project language by checking for manifest files:
| Marker(s) | Language |
|---|---|
go.mod | Go |
Package.swift | Swift |
Cargo.toml | Rust |
build.zig | Zig |
pyproject.toml, setup.py | Python |
Gemfile, *.gemspec | Ruby |
| Executable scripts (no manifest) | Shell |
If no manifest is found and executable shell scripts exist in the root, bin/, or scripts/, treat the project as a shell script CLI.
If the project type cannot be determined, ask the user what language the project uses and where the binary or executable is located.
Check for an existing tests/scrut/ directory; if present, warn and ask whether to add to it or abort.
Collect the following, inferring from existing files where possible:
build target for the output binary name (look for -o bin/NAME or -o NAME), or derive from the last segment of the module path in go.modPackage.swift for executable target names, or check the Makefile build targetCargo.toml for [[bin]] entries or the name field under [package]build.zig for b.addExecutable(.{ .name = "..." }) and use the .name value, or use the directory namepyproject.toml for [project.scripts] entriesexecutables or look in bin/ or exe/bin/NAME or a build output directory (e.g., $(CURDIR)/bin/NAME for Go, $(CURDIR)/zig-out/bin/NAME for Zig)bin/NAME, ./NAME)_BIN (e.g., bopca becomes BOPCA_BIN, my-tool becomes MY_TOOL_BIN)If the user already provided some or all of these in their initial request, do not re-ask.
Create the tests/scrut/ directory:
mkdir -p tests/scrut
Generate two starter test files using the templates from the references:
tests/scrut/help.md: uses the template from the Help Test Template section below. Replace TOOL with the binary name, TOOL_BIN with the environment variable name, and run the binary with --help to capture the actual help output for the initial snapshot.
tests/scrut/version.md: uses the template from the Version Test Template section below. Replace TOOL with the binary name, TOOL_BIN with the environment variable name.
To populate the initial help output snapshot:
TOOL_BIN="BINARY_PATH" scrut create --title "Root help" '"${TOOL_BIN}" --help'
If scrut is not installed locally, write the test files with placeholder output and instruct the user to run make test-scrut-update after installing scrut to populate the snapshots.
If no Makefile exists:
build and test-scrut targets appropriate to the languageIf a Makefile exists, check for any of these targets: test-scrut, test-scrut-update, test-all. If any exist, warn and ask before overwriting.
Add the Makefile targets from the Makefile Targets section below:
TOOL_BIN with the environment variable nameBINARY_PATH with the path to the built binary or executable scriptTESTS_DIR with tests/scrut/build dependency from test-scrut and test-scrut-update targetsAppend test-scrut test-scrut-update test-all to the .PHONY declaration (or create one if it does not exist).
If the Makefile has an all target, consider adding test-all to it or replacing test with test-all as appropriate. Ask the user if unsure.
Look for the CI workflow file:
ls .github/workflows/ci.yml .github/workflows/ci.yaml
If found, add a test-scrut job using the template from the CI Job Template section below:
scrut-setup-cmd to the command that builds the project binary (e.g., make build). For interpreted languages (shell scripts, Python, Ruby), omit this input.scrut-env to the environment variable mapping the binary path (e.g., TOOL_BIN=./bin/my-tool). Use the same TOOL_BIN variable name and BINARY_PATH value from the Makefile targets.runs-on: macos-latest as an input in the reusable workflow's with: block (do not add a top-level runs-on field on the job).needs), set up appropriate dependencies for the new jobIf no CI workflow exists, skip this step and note that the user should add CI workflow configuration manually.
Build the binary (if applicable) and run the scrut tests to verify the setup:
make test-scrut
If scrut is not installed locally, inform the user to install from the facebookincubator/scrut GitHub releases. Determine the latest release tag (run gh release list --repo facebookincubator/scrut --limit 1 --json tagName --jq '.[0].tagName'):
mkdir -p ~/.local/bin
gh release download SCRUT_VERSION --repo facebookincubator/scrut --pattern 'scrut-SCRUT_VERSION-SCRUT_PLATFORM.tar.gz' --dir /tmp
tar -xzf /tmp/scrut-SCRUT_VERSION-SCRUT_PLATFORM.tar.gz -C /tmp
cp /tmp/scrut-SCRUT_PLATFORM/scrut ~/.local/bin/
Replace SCRUT_VERSION with the pinned release tag (e.g., v0.4.3) and SCRUT_PLATFORM with the appropriate identifier (e.g., macos-aarch64, linux-x86_64).
Scrut does not currently publish checksums or signatures with its releases, so integrity verification is not yet possible. If checksums become available in the future, add a verification step after the download, mirroring the guidance in the CI Job Template section below.
Ensure ~/.local/bin is on PATH. If it is not already, add it to your shell profile (e.g., export PATH="$HOME/.local/bin:$PATH" in ~/.zshrc or ~/.bashrc).
If tests fail because the expected output does not match, update the snapshots:
make test-scrut-update
Then re-run to confirm:
make test-scrut
If a markdownlint config exists (.markdownlint-cli2.jsonc, .markdownlint.jsonc, .markdownlint.yaml, or .markdownlint.json), add "MD014": false to disable the "dollar signs used before commands" rule. Scrut test files use $ command notation that triggers this rule.
If no markdownlint config exists, skip this step.
Print a summary of what was created and modified:
tests/scrut/help.md, tests/scrut/version.md)Makefile, .github/workflows/ci.yml)make test-scrut: run scrut testsmake test-scrut-update: update test snapshotsmake test-all: run both unit tests and scrut testsMakefile exists and the project is a compiled language, warn that a build step is needed and offer to create a minimal Makefile, or suggest the relevant scaffolding skill (e.g., scaffold-go-cli for Go projects)Makefile exists and the project is an interpreted language, offer to create a minimal Makefile with just the scrut test targetstests/scrut/ already exists, ask before adding or overwriting filesscrut is not installed locally, write test files with placeholder output and explain how to install scrut and update snapshotsbuild target and the project requires one, ask the user which target builds the binarymake test-scrut fails, check the output and attempt to fix; if the issue is stale snapshots, run make test-scrut-updateStarter test file template for verifying --help output.
Write tests/scrut/help.md with the following content:
# Help output
Tests for TOOL help commands.
## Root help
```scrut
$ "${TOOL_BIN}" --help
HELP_OUTPUT
```
## Short help flag
```scrut
$ "${TOOL_BIN}" -h
HELP_OUTPUT
```
| Placeholder | Description | Example |
|---|---|---|
TOOL | Human-readable tool name | bopca |
TOOL_BIN | Environment variable name for the binary path | BOPCA_BIN |
HELP_OUTPUT | Actual output from running the binary with --help | (captured during setup) |
--help and -h should produce identical output. If they differ, adjust accordingly.HELP_OUTPUT should be the exact output captured from running the binary. Use scrut create or scrut update to populate this.* (glob+) placeholder for the output, then run make test-scrut-update to replace it with the real output.subcommand-help.md).Starter test file template for verifying version command output.
Write tests/scrut/version.md with the following content:
# Version output
Tests for TOOL version commands.
## Version command
```scrut
$ "${TOOL_BIN}" version
TOOL v* (glob)
commit: * (glob)
built: * (glob)
```
## Version flag
```scrut
$ "${TOOL_BIN}" --version
TOOL v* (glob)
commit: * (glob)
built: * (glob)
```
| Placeholder | Description | Example |
|---|---|---|
TOOL | Human-readable tool name (as printed by the binary) | bopca |
TOOL_BIN | Environment variable name for the binary path | BOPCA_BIN |
(glob) patterns for these lines.--version and not a version subcommand (or vice versa), remove the test case that does not apply.Scrut test targets to add to the project Makefile.
## Run scrut CLI tests
test-scrut: build
@echo "Running scrut CLI tests..."
@if ! command -v scrut >/dev/null 2>&1; then \
echo "scrut not installed. Install from https://github.com/facebookincubator/scrut"; \
exit 1; \
fi
TOOL_BIN="BINARY_PATH" scrut test TESTS_DIR
## Update scrut test expectations
test-scrut-update: build
TOOL_BIN="BINARY_PATH" scrut update --replace --assume-yes TESTS_DIR
## Run all tests (unit + scrut)
test-all: test test-scrut
| Placeholder | Description | Example |
|---|---|---|
TOOL_BIN | Environment variable name for the binary path | BOPCA_BIN |
BINARY_PATH | Path to the built binary | $(CURDIR)/bin/bopca |
TESTS_DIR | Directory containing scrut test files | tests/scrut/ |
test-scrut depends on build so the binary is compiled before tests run. For interpreted languages (shell scripts, Python, Ruby) where no build step is needed, remove the : build dependency from test-scrut and test-scrut-update. The binary path points directly to the executable script.command -v scrut) gives a clear error message if scrut is not installed.test-scrut-update uses --replace to overwrite the original files and --assume-yes to skip confirmation prompts.test-all chains both unit tests (test) and scrut tests (test-scrut)..PHONY declaration.GitHub Actions job template for running scrut CLI tests. Uses the cboone/gh-actions reusable workflow, which handles scrut installation (using an internal SHA-256 checksum manifest), checkout, and test execution internally. This verified checksum source is available only via the reusable workflow; scrut's published GitHub releases do not currently provide checksums for manual local installation.
test-scrut:
uses: cboone/gh-actions/.github/workflows/run-scrut-tests.yml@91f9abd25d4f82354c0f950dfc8b6d7525b0f5b5 # v3.0.0
with:
scrut-setup-cmd: "SETUP_CMD"
scrut-env: "TOOL_BIN=BINARY_PATH"
| Placeholder | Description | Example |
|---|---|---|
SETUP_CMD | Command to build the project binary before running scrut tests (optional, omit for interpreted languages) | make build, cargo build --release, zig build |
TOOL_BIN | Environment variable name for the binary path | BOPCA_BIN |
BINARY_PATH | Path to the built binary or executable script | ./bin/bopca, ./target/release/bopca |
scrut-setup-cmd runs before scrut tests to build the project binary. For interpreted languages (shell scripts, Python, Ruby) where no build step is needed, omit this input entirely.scrut-env accepts newline-delimited KEY=VALUE pairs. Relative paths (starting with ./) are automatically resolved to absolute paths by the reusable workflow.runs-on: ubuntu-latest. For projects that require macOS (e.g., macOS framework dependencies), add runs-on: macos-latest.scrut-test-dir is tests/. Override with scrut-test-dir: "tests/scrut/" if your test files are in a subdirectory.Reference for writing scrut test files. See the scrut documentation for the complete specification.
Scrut test files are standard Markdown files (.md) containing fenced code blocks with the scrut language identifier. Each file represents a group of related test cases. All test cases within a single file share the same shell process, so variables, aliases, and exports persist across blocks.
Lines starting with $ are shell commands. Lines starting with > are command continuations. All other lines within the code block are expected output.
The expected output must match exactly, line by line:
$ echo "hello world"
hello world
Use * (any characters) and ? (one character) with the (glob) suffix:
$ my-tool version
my-tool v* (glob)
commit: * (glob)
built: * (glob)
Use regular expressions with the (regex) suffix:
$ my-tool version
my-tool v\d+\.\d+\.\d+ (regex)
Match output containing non-printable characters (ANSI escapes, tabs) with the (escaped) suffix:
$ printf "foo\tbar"
foo\tbar (escaped)
Quantifiers control how many output lines a single expectation line can match:
| Quantifier | Meaning | Example |
|---|---|---|
? | Zero or one line | * (glob?) |
* | Zero or more lines | * (glob*) |
+ | One or more lines | * (glob+) |
Common pattern for matching variable-length output:
$ my-tool --help
Usage:
my-tool [command]
* (glob+)
By default, scrut expects exit code 0. Specify non-zero exit codes with bracket notation as the last line of expected output:
$ my-tool --bad-flag 2>&1 | head -1
Error: unknown flag "--bad-flag"
[1]
When a command should produce no output, leave the code block empty after the command:
$ my-tool --quiet version
By default, scrut validates stdout only. To validate stderr, use the output_stream attribute:
$ my-tool --bad-flag
Error: unknown flag "--bad-flag"
[1]
To validate combined stdout and stderr, either use the attribute or redirect in the command:
$ my-tool bad-command 2>&1 | head -1
Error: unknown command "bad-command" for "my-tool"
Attributes can be set in curly braces after the language tag:
| Attribute | Description | Example |
|---|---|---|
timeout | Max execution time | {timeout: 10s} |
fail_fast | Stop document on failure | {fail_fast: true} |
output_stream | Stream to validate | {output_stream: stderr} |
environment | Extra environment variables | {environment: {"KEY": "val"}} |
YAML frontmatter controls document-level settings:
---
prepend:
- "../shared/setup.md"
defaults:
timeout: 10s
total_timeout: 30s
---
Scrut provides these variables in every test execution:
| Variable | Description |
|---|---|
$TESTDIR | Directory containing the test file |
$TESTFILE | Name of the current test file |
$TMPDIR | Fresh temporary directory per test file |
NO_COLOR=1 to suppress color codes in output.head, tail, or grep to test specific lines.$(mktemp -d) for operations that create files.| sort.jq extraction over snapshotting raw text for structured data.help.md, version.md, error-handling.md).make test-scrut-update to regenerate expected output after intentional changes.cboone/gh-actions SHAs before scaffoldingThe cboone/gh-actions reusable-workflow refs in this skill's templates are SHA-pinned with a # vX.Y.Z comment that was current when the template was authored. New releases of cboone/gh-actions rot those SHAs. Before emitting a workflow into a user's repo, refresh both the SHA and the comment to current latest:
TAG="$(gh release view --repo cboone/gh-actions --json tagName --jq '.tagName')"
SHA="$(gh api "repos/cboone/gh-actions/commits/${TAG}" --jq '.sha')"
echo "${SHA} # ${TAG}"
Replace each cboone/gh-actions/.../<workflow>.yml@<old-sha> # <old-tag> in the emitted workflow with the new SHA and tag. Dependabot in the user's repo keeps them in sync afterwards.
Provides a checklist for code reviews covering functionality, security, performance, maintainability, tests, and quality. Use for pull requests, audits, team standards, and developer training.
npx claudepluginhub cboone/agent-harness-plugins --plugin add-scrut-cli-tests