llm-guidelines

LLM Guidelines Skill

An Agent Skill that applies the community guidelines for empirical software engineering studies involving LLMs to a paper draft and its supplementary material. Agent Skills is an open standard originally developed by Anthropic. The format is now read by Cursor, GitHub Copilot, OpenAI Codex, Gemini CLI, Claude Code, and JetBrains Junie, among others; see the client list.

The guideline content is published at https://llm-guidelines.org and authored in the companion paper repository https://github.com/se-uhd/llm-guidelines-paper. The Markdown files in this repository are generated from those LaTeX sources; see https://github.com/se-uhd/llm-guidelines-website for the conversion pipeline. Do not edit the generated files in this repository directly.

Versioning

Skill releases use the same CalVer (YYYY.MM) tag as the paper and the website. The current version is recorded in VERSION and embedded in .claude-plugin/plugin.json and skills/llm-guidelines/SKILL.md.

Install as a Claude Code plugin

/plugin marketplace add se-uhd/llm-guidelines-skill
/plugin install llm-guidelines

Invoke from Claude Code with a LaTeX source:

/llm-guidelines:llm-guidelines path/to/paper.tex

Or with a PDF:

/llm-guidelines:llm-guidelines path/to/paper.pdf

Supplementary-material paths are optional, and you can pass more than one when artifacts live in separate locations (e.g., code on GitHub, data on Zenodo):

/llm-guidelines:llm-guidelines path/to/paper.tex path/to/code-repo path/to/dataset

To pick up a new release after a YYYY.MM tag is pushed, refresh the marketplace catalog and update the installed plugin:

/plugin marketplace update llm-guidelines
/plugin update llm-guidelines
/reload-plugins

Install as a user skill (no plugin)

git clone https://github.com/se-uhd/llm-guidelines-skill.git
mkdir -p ~/.claude/skills
cp -r llm-guidelines-skill/plugins/llm-guidelines/skills/llm-guidelines ~/.claude/skills/

Invoke from Claude Code with a LaTeX source:

/llm-guidelines path/to/paper.tex

Or with a PDF:

/llm-guidelines path/to/paper.pdf

Supplementary-material paths are optional, and you can pass more than one when artifacts live in separate locations (e.g., code on GitHub, data on Zenodo):

/llm-guidelines path/to/paper.tex path/to/code-repo path/to/dataset

Use in other Agent Skills clients

The directory skills/llm-guidelines/ is a self-contained Agent Skill. Copy it into the skills location of the client you use; each client's docs are linked from the Agent Skills client list.

What the skill does

Given a paper (LaTeX source or PDF) and any supplementary material directories, the skill:

1. Identifies the study type(s) using the taxonomy in study-types/.
2. Confirms the work is in scope using scope.md.
3. Runs a per-guideline assessment using the eight files under plugins/llm-guidelines/skills/llm-guidelines/guidelines/.
4. Writes a Markdown report to llm-guidelines-report.md in the current working directory and prints it to the console.

The report frames findings as opportunities to clarify or strengthen reporting. It is intended as a self-check for authors. Reviewers may use it for inspiration but should not treat it as a rejection rubric.

Repository layout

.claude-plugin/
  marketplace.json     marketplace entry (single-plugin)
plugins/
  llm-guidelines/
    .claude-plugin/
      plugin.json      plugin manifest
    skills/
      llm-guidelines/
        SKILL.md       skill entry point and workflow
        scope.md       in-scope / out-of-scope definition
        guidelines/    eight standalone guideline files
        study-types/   seven study-type files plus parent groupings
        checklist.md   consolidated reporting checklist
VERSION                CalVer version, mirrors paper and website

Maintainer notes

The bundled Markdown is generated by the build pipeline in the website repository (see its CLAUDE.md "Skill Pipeline" section for the full procedure, including version bumps, validation, and lessons from initial setup).

When changing .claude-plugin/marketplace.json or plugins/llm-guidelines/.claude-plugin/plugin.json, validate before committing:

claude plugin validate plugins/llm-guidelines
claude plugin validate .claude-plugin/marketplace.json

If a user reports Failed to install: This plugin uses a source type your Claude Code version does not support, do not assume the problem is the source field. Claude Code emits that same message for any unrecognized key in either manifest. Run claude plugin validate first — the validator surfaces the real error.

License