airbds-assessment

AIRBDS AI-Readiness Dataset Scoring Metric

A versioned, machine-readable scoring metric for evaluating the AI-readiness of bioscience datasets. Developed by the AI-Ready Bioscience Datasets (AIRBDS) Working Group of the AIBIO-UK network, funded by BBSRC.

---

Overview

The AIRBDS metric provides a structured checklist that researchers, data curators, and repository managers can use to assess how suitable a bioscience dataset is for use in AI/ML workflows. Each of the 28 questions is answered Yes/No and contributes a weighted score. Datasets are then assigned one of four grades:

Grade	Description
🔴 Caution	May have serious issues — fails one or more Critical criteria
🟤 Bronze	Passes most Critical questions (≥ 7/8)
⚪ Silver	Passes all Critical + ≥ 50% of Important questions
🟡 Gold	Passes all Critical and Important + ≥ 50% of Optional questions

Questions are grouped into four scopes and three weight tiers:

Scope	Questions	Weight tiers covered
Infrastructure	ACM-1 – ACM-10	Critical, Important, Optional
Metadata	ACM-11 – ACM-17	Critical, Important, Optional
Content	ACM-18 – ACM-23	Critical, Important, Optional
Ethics	ACM-24 – ACM-28	Critical, Important, Optional

Weight points: Critical = 80 pts · Important = 5 pts · Optional = 2 pts

Ethics note: Questions ACM-24 to ACM-28 apply to datasets containing human or animal subject data. For datasets with no such data, these questions default to "Yes" (not applicable) and should be noted in the review file.

---

Repository Structure

airbds-metric/
├── metric/
│   ├── airbds_metric_v0.3.yaml   # Canonical metric — all 28 questions (YAML)
│   ├── airbds_metric_v0.3.csv    # Canonical metric — all 28 questions (CSV)
│   ├── scoring_schema.yaml       # Grade thresholds, weight definitions (YAML)
│   ├── scoring_schema.csv        # Grade thresholds, weight definitions (CSV)
│   ├── review_template.yaml      # Blank review template (YAML)
│   └── review_template.csv       # Blank review template (CSV / spreadsheet)
├── reviews/                      # Completed dataset review files
├── skills/                       # AI agent skills for performing assessments and installation instructions
├── docs/
│   ├── tutorial-csv.md           # Beginner tutorial — Excel / Google Sheets
│   └── tutorial-yaml.md          # Intermediate tutorial — text editor / CLI
├── AIRBDS Core Metric scoring v0.3 - _initials_-_#_ TEMPLATE.xlsx
│                                 # Original working-group spreadsheet (archived)
├── CHANGELOG.md
├── CITATION.cff
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE.md
└── README.md

---

Formats & Tutorials

The metric is available in two formats. Both cover the same 28 questions and produce the same grade — choose whichever suits your workflow.

→ Interactive tutorial site — step-by-step guide that helps you choose the right format and walks you through the full review process.

Format	Best for	Template	Tutorial
CSV	Beginners · Excel or Google Sheets · no coding required	review_template.csv	Beginner CSV Tutorial
YAML	Intermediate · text editor / command line	review_template.yaml	Intermediate YAML Tutorial

---

Quick Start (YAML)

For the spreadsheet workflow, see the Beginner CSV Tutorial.

1. Copy the template:

   cp metric/review_template.yaml reviews/<accession>_<initials>_1.yaml
   

2. Answer all 28 questions ("Yes" or "No") in your copy.
3. Submit a PR — see CONTRIBUTING.md.
4. Automated processing handles the rest: score calculation, grade assignment, CSV conversion, and file renaming — see Automated Review Processing below.

Full reviewer instructions are in metric/scoring_schema.yaml.

---

Metric Questions at a Glance

Infrastructure (ACM-1 – ACM-10)