Monitors deployed model performance, detects data drift, and manages model health using the DataRobot Python SDK. Use for tracking prediction accuracy, feature drift, and prediction anomalies.
How this skill is triggered — by the user, by Claude, or both
Slash command
/datarobot-agent-skills:datarobot-model-monitoringThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
This skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.
This skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.
Most common use case: Check deployment health and data drift
deployment.get_service_stats(...) to review prediction volume/latencydeployment.get_feature_drift(...) / deployment.get_target_drift(...)get_service_stats_over_time(...) and drift periods to assess trendsExample: "Check the health of deployment abc123 and report any data drift issues"
Use this skill when you need to:
User request: "Check the health of deployment abc123 and report any data drift issues."
Agent workflow:
User request: "Set up alerts for deployment xyz789 to notify when feature drift exceeds 0.2."
Agent workflow:
This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:
pip install datarobot
Use these DataRobot SDK and MLOps API methods for monitoring:
Deployment Monitoring:
deployment.get_service_stats(...) - Get service statistics (latency, volume, etc.)deployment.get_feature_drift(...) - Get feature drift metrics (returns FeatureDrift objects)deployment.get_target_drift(...) - Get target drift metrics (returns TargetDrift)deployment.get_prediction_results(...) - Retrieve recorded prediction results (if enabled)Model Performance:
model.get_metrics() - Get model performance metricsmodel.get_roc_curve() - Get ROC curve for comparisonNote: Some monitoring features may require DataRobot MLOps API. See the Common Patterns section below for examples.
import datarobot as dr
import os
# Initialize client
client = dr.Client(
token=os.getenv("DATAROBOT_API_TOKEN"),
endpoint=os.getenv("DATAROBOT_ENDPOINT")
)
# Get deployment
deployment = dr.Deployment.get("abc123")
# Get service stats (requires MLOps monitoring to be enabled)
stats = deployment.get_service_stats()
print(f"Prediction count: {stats.prediction_count}")
print(f"Mean response time (ms): {stats.mean_response_time}")
# Get recorded prediction results (if available / enabled)
try:
recent = deployment.get_prediction_results(limit=10)
print(f"Recent prediction results: {len(recent)}")
except Exception as e:
print(f"Prediction results not available: {e}")
import datarobot as dr
# Get deployment
deployment = dr.Deployment.get("abc123")
# Get feature drift (requires MLOps monitoring)
try:
drifts = deployment.get_feature_drift()
high = [d for d in drifts if (d.drift_score or 0) > 0.2]
print(f"Features with drift_score > 0.2: {len(high)}")
for d in high[:10]:
print(f"{d.name}: {d.drift_score}")
except Exception as e:
print(f"Feature drift requires MLOps monitoring: {e}")
Recommended thresholds:
Adjust thresholds based on your domain and use case sensitivity.
Common errors and solutions:
pip install datarobot
import datarobot as dr
import os
client = dr.Client(
token=os.getenv("DATAROBOT_API_TOKEN"),
endpoint=os.getenv("DATAROBOT_ENDPOINT", "https://app.datarobot.com")
)
Note: Some monitoring features require DataRobot MLOps API access. Check your DataRobot plan for MLOps availability.
npx claudepluginhub datarobot-oss/datarobot-agent-skills --plugin datarobot-agent-skillsDeploys DataRobot models to production, manages deployments, configures prediction environments, and handles model swaps or A/B testing.
Detects data drift and concept drift in production ML models using Evidently AI, PSI, KS tests, and custom metrics. Sets up automated alerts and reports to catch model degradation before it impacts business metrics.
Evaluate model performance — check for accuracy drops, data drift, and error patterns. Use when asked about "model accuracy dropped", "evaluate the model", "check for drift", or "model performance".