oodle-drop-rules

Oodle Drop Rules — Cost Control

This skill teaches the agent to drop or sample high-volume metrics safely: estimate the impact first, prefer sampling over dropping, and never silently delete a metric a dashboard depends on.

Prerequisites

brew install oodle-ai/oodle/oodle
oodle configure

Confirm the drop-rules endpoint works:

oodle drop-rules list -o json | jq 'length'

Command Execution Order

Before running any oodle command:

1. Check whether the target metric prefix and matchers are already in context.
2. If not, run oodle metrics list --match <prefix> -o json | jq 'length' to estimate impact.
3. Confirm no critical dashboards or monitors depend on the affected series:
   • oodle dashboards list -o json | jq '.[] | select(.panels[]?.query | contains("<metric>"))'
   • oodle monitors list -o json | jq '.[] | select(.query | contains("<metric>"))'
4. Prefer action: sample (e.g. sampleRate: 0.1) on the first attempt; switch to action: drop only after the sampled data confirms low value.
5. Run oodle drop-rules create -f rule.json.

Quick Reference

Task	Command
List rules	oodle drop-rules list -o json
Get rule	oodle drop-rules get <id> -o json
Create rule	oodle drop-rules create -f rule.json
Update rule	oodle drop-rules update <id> -f rule.json
Delete rule	oodle drop-rules delete <id> --force

Common Operations

Rule schema

{
  "name": "drop-noisy-debug-metrics",
  "matchers": [
    {"name": "level", "value": "debug"},
    {"name": "env",   "value": "staging"}
  ],
  "action": "drop",
  "sampleRate": null
}

Field	Meaning
matchers	All matchers must match for the rule to fire (logical AND)
action	drop (discard) or sample (keep a fraction)
sampleRate	Required when action="sample"; float in (0,1]. null for drop

Estimating impact before creating a rule

# ✅ CORRECT — count series that will be affected
oodle metrics list --match "debug_" -o json | jq 'length'

# ✅ CORRECT — confirm no dashboard panel queries the metric
oodle dashboards list -o json | jq '.[] | select(.panels[]?.query | contains("debug_traffic_total"))'

# ✅ CORRECT — confirm no monitor depends on it
oodle monitors list -o json | jq '.[] | select(.query | contains("debug_traffic_total"))'

# ❌ WRONG — create the rule first, find out from on-call later
oodle drop-rules create -f rule.json

Creating a sample rule (safer first step)

{
  "name": "sample-debug-metrics-staging",
  "matchers": [
    {"name": "__name__", "value": "debug_traffic_total"},
    {"name": "env",      "value": "staging"}
  ],
  "action": "sample",
  "sampleRate": 0.1
}

# ✅ CORRECT — keep 10% of the series; observe for a week before dropping
oodle drop-rules create -f rule.json

Promoting a sample rule to drop

# ✅ CORRECT — get → switch action → update
oodle drop-rules get dr_123 -o json > rule.json
jq '.action = "drop" | .sampleRate = null' rule.json > rule.new.json
oodle drop-rules update dr_123 -f rule.new.json

# ❌ WRONG — partial payload nulls matchers
oodle drop-rules update dr_123 -f <(echo '{"action":"drop"}')

Deleting (re-enabling ingestion)

# ✅ CORRECT
oodle drop-rules get dr_123 -o json > /dev/null
oodle drop-rules delete dr_123 --force

# ❌ WRONG — speculative delete by name match
oodle drop-rules delete "$(oodle drop-rules list | grep debug | awk '{print $1}')" --force

Best Practices

Estimate impact with oodle metrics list --match <prefix> -o json | jq 'length' before creating a rule

A drop rule that matches more series than expected can hide real signal.

# ✅ CORRECT
oodle metrics list --match "kube_pod_" -o json | jq 'length'
# 1742 series — confirm with the team that all 1742 are safe to drop before creating a rule

# ❌ WRONG — create rule based on a guess; later discover a critical metric was matched
oodle drop-rules create -f rule.json

Prefer action: sample with sampleRate: 0.1 on the first iteration

Sampling preserves enough signal to confirm the metric truly is low-value before fully dropping it.

# ✅ CORRECT — week 1: sample at 10%, observe dashboards
"action": "sample", "sampleRate": 0.1
# week 2: if no dashboards or monitors regressed, switch to drop
"action": "drop", "sampleRate": null

# ❌ WRONG — drop on first attempt; can break a dashboard nobody remembered
"action": "drop"

Always include both env and service (or __name__) in matchers

Broad matchers like {level: debug} alone can match production metrics that happen to share a label.

# ✅ CORRECT — scoped to one env
"matchers": [{"name":"level","value":"debug"},{"name":"env","value":"staging"}]

# ❌ WRONG — also drops debug metrics in prod
"matchers": [{"name":"level","value":"debug"}]

Always get before update to preserve fields

Update is a full-document replace.

# ✅ CORRECT
oodle drop-rules get dr_123 -o json > rule.json
jq '.sampleRate = 0.05' rule.json > rule.new.json
oodle drop-rules update dr_123 -f rule.new.json

# ❌ WRONG — drops matchers and action
oodle drop-rules update dr_123 -f <(echo '{"sampleRate":0.05}')

Name rules with <action>-<metric-or-domain>-<scope>

Predictable names make it easy to find and revert a rule when a dashboard breaks.

# ✅ CORRECT
"name": "drop-debug-metrics-staging"
"name": "sample-kube-pod-info-prod"

# ❌ WRONG
"name": "rule1"

Failure Handling

Error	Cause	Fix
401 Unauthorized	Invalid or missing API key	Run oodle configure or set OODLE_API_KEY
404 Not Found	Drop rule ID does not exist	Verify with oodle drop-rules list -o json
connection refused	Wrong OODLE_DEPLOYMENT URL	Check OODLE_DEPLOYMENT env var
sampleRate required	action: sample without sampleRate	Add sampleRate between 0 and 1 (e.g. 0.1)
Dashboard panel suddenly empty	Drop rule matched a metric the panel queries	Run oodle drop-rules list -o json to find the rule; delete it (oodle drop-rules delete <id> --force) or narrow its matchers
Monitor went into no data	Drop rule matched the monitor's metric	Same fix as above; alternatively switch the rule from drop to sample
Cost did not decrease	Matchers don't actually match the high-volume series	Re-run oodle metrics list --match ... and compare label sets to the rule's matchers
429 Too Many Requests	Bulk drop-rule sync	Add --retries 3, throttle to <10 creates per second

References

• Oodle CLI repo
• Oodle docs