Stats
Actions
Tags
From agent-almanac
Sets up systematic data labeling workflows using Label Studio with quality controls, inter-annotator agreement, and active learning for supervised ML projects.
How this skill is triggered — by the user, by Claude, or both
Slash command
/agent-almanac:label-training-dataThis skill is limited to the following tools:
The summary Claude sees in its skill listing — used to decide when to auto-load this skill
> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.
See Extended Examples for complete configuration files and templates.
Systematically label data for supervised ML with quality controls and efficient workflows.
Set up Label Studio as the labeling platform.
# Install Label Studio
pip install label-studio
# Or use Docker for production
docker pull heartexlabs/label-studio:latest
# Create project directory
mkdir -p labeling-project/{data,exports,config}
cd labeling-project
# Initialize Label Studio
label-studio init my_project
# Start Label Studio server
label-studio start my_project --port 8080
Access at http://localhost:8080 (default credentials: create on first visit).
For production deployment with Docker:
# docker-compose.yml
version: '3.8'
services:
label-studio:
image: heartexlabs/label-studio:latest
ports:
- "8080:8080"
# ... (see EXAMPLES.md for complete implementation)
docker-compose up -d
Expected: Label Studio running and accessible, PostgreSQL database initialized for production use.
On failure: If port 8080 already in use, change port in config, if Docker fails check Docker daemon is running, ensure sufficient disk space for data volumes, check firewall allows port 8080.
Create labeling configuration for your task type.
# labeling-project/config/labeling_config.py
"""
Label Studio configuration templates for common tasks.
"""
# Text Classification (single label)
TEXT_CLASSIFICATION = """
<View>
# ... (see EXAMPLES.md for complete implementation)
Expected: Labeling interface configured with appropriate controls for task type, data imported successfully, interface accessible to annotators.
On failure: Validate XML config with Label Studio's config validator, check data file format (JSON or CSV), ensure image/audio URLs are accessible if using external storage, verify API key has correct permissions.
Format data for import and prioritize examples for labeling.
# labeling-project/prepare_data.py
import pandas as pd
import json
import random
from typing import List, Dict
from sklearn.cluster import KMeans
import numpy as np
# ... (see EXAMPLES.md for complete implementation)
Expected: Data formatted correctly for Label Studio import, sampling strategy prioritizes informative examples, tasks include metadata for tracking.
On failure: Verify JSON format with jq or Python json.load(), check that URLs are accessible if using remote images, ensure no special characters break JSON encoding, validate column names match config.
Set up processes to measure and improve annotation quality.
# labeling-project/quality_control.py
import pandas as pd
import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix
from typing import Dict, List, Tuple
import logging
logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)
Expected: Inter-annotator agreement measured (Cohen's Kappa > 0.6 is moderate, >0.8 is good), difficult tasks identified for review, annotator performance tracked.
On failure: If Kappa very low (<0.4), review labeling guidelines for clarity, retrain annotators, simplify label schema, check for ambiguous examples, consider using expert annotators for gold standard.
Export labels and prepare for ML training.
# labeling-project/export_labels.py
import requests
import pandas as pd
import json
from typing import List, Dict
import logging
logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)
Expected: Annotations exported in training-ready format, label distribution balanced or documented, data quality validated before training.
On failure: Verify API key permissions, check export format compatibility with your ML framework, handle missing annotations gracefully, validate JSON structure matches expected format.
Automate labeling workflow with active learning integration.
# labeling-project/active_learning_pipeline.py
import schedule
import time
import logging
from datetime import datetime
from prepare_data import DataSampler, prepare_label_studio_format
from export_labels import LabelStudioExporter, convert_to_training_format
import pandas as pd
# ... (see EXAMPLES.md for complete implementation)
Expected: Active learning selects informative examples automatically, labeling batches prepared weekly, model retrained when sufficient new labels available.
On failure: If uncertainty sampling doesn't improve model, try diversity sampling, if annotators can't keep up reduce batch size, monitor labeling queue length, implement backpressure if queue grows too large.
version-ml-data - Version control for labeled datasetstrack-ml-experiments - Track model performance as labels addednpx claudepluginhub pjt222/agent-almanacUse this skill when the user asks to "design a human evaluation", "human eval process", "annotation guidelines", "how to set up human review of AI outputs", "how to get humans to evaluate AI quality", "build a labeling process", "create annotation criteria", or wants to set up a structured process for humans to evaluate AI output quality.
Turns model work into production ML systems with data contracts, repeatable training, quality gates, deployable artifacts, and monitoring. Useful for ranking, search, recommendations, classifiers, forecasting, embeddings, LLMs, anomaly detection, and batch analytics.
Uploads images, labels, organizes datasets, creates Roboflow projects (detection/segmentation/keypoint/classification), manages tags, splits, versions, and searches with RoboQL.