agentic-kpi-tracking

Agentic KPI Tracking Skill

Guide measurement and tracking of agentic coding KPIs to assess ZTE readiness.

When to Use

• Measuring agentic workflow effectiveness
• Tracking progress toward ZTE
• Analyzing success patterns
• Identifying improvement areas

Core KPIs

Summary Metrics

Metric	Calculation	Target
Current Streak	Consecutive successes (Attempts <= 2)	Higher is better
Longest Streak	Best consecutive success run	Track improvement
Average Presence	Mean attempts across all runs	Target: 1
Total Plan Size	Sum of all plan sizes	Track scaling
Total Diff Size	Sum of all changes (added + removed)	Track throughput

Per-Run Metrics

Metric	Source	Meaning
Attempts	Count of plan/patch runs	1 = perfect, higher = retries
Plan Size	Lines in plan file	Task complexity
Diff Size	Lines added + removed	Change magnitude
Files Changed	Number of files modified	Change scope

Calculation Methods

Attempts Count

Only count workflow restarts:

attempts_incrementing = ["adw_plan_iso", "adw_patch_iso"]
attempts = count(workflow in all_adws if workflow in attempts_incrementing)

Build/test/review don't increment - only full replans.

Streak Calculation

current_streak = 0
for run in reversed(runs):
    if run.attempts <= 2:
        current_streak += 1
    else:
        break

Diff Statistics

git diff origin/main --shortstat
# Output: X files changed, Y insertions(+), Z deletions(-)

KPI File Format

Store in app_docs/agentic_kpis.md or equivalent:

# Agentic KPIs

## Summary

| Metric | Value |
| --- | --- |
| Current Streak | 5 |
| Longest Streak | 12 |
| Average Presence | 1.3 |
| Total Plan Size | 450 lines |
| Total Diff Size | 2,340 lines |

## Detail

| Date | ADW ID | Issue | Class | Attempts | Plan Size | Diff +/- | Files |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2024-01-15 | abc123 | #45 | /bug | 1 | 35 | +45/-12 | 3 |
| 2024-01-14 | def456 | #44 | /feature | 2 | 85 | +120/-30 | 8 |

Tracking Workflow

Step 1: Gather Current Run Data

From state or git:

• ADW ID
• Issue number
• Issue classification
• Plan file path
• All workflows run (for attempts)

Step 2: Calculate Metrics

attempts = count_attempts(all_adws)
plan_size = wc_lines(plan_file)
diff_stats = parse_git_diff()

Step 3: Update Detail Table

Add new row with current run data.

Step 4: Recalculate Summary

Update all summary metrics based on full detail table.

Step 5: Analyze Trends

• Is streak increasing?
• Is average presence decreasing?
• Are plan sizes growing (handling bigger tasks)?

ZTE Readiness Indicators

Based on KPIs, assess ZTE readiness:

Indicator	Threshold	Status
Current Streak	>= 5	Ready to try ZTE
Average Presence	<= 1.5	Good efficiency
Recent Failures	0 in last 10	High confidence
Plan Size Trend	Increasing	Scaling up

Key Memory References

• @agentic-kpis.md - KPI definitions from Lesson 002
• @zte-progression.md - How KPIs relate to ZTE levels
• @zte-confidence-building.md - Using KPIs for confidence

Output Format

Provide KPI update:

## KPI Update

**Run:** {adw_id}
**Issue:** #{issue_number} ({issue_class})

### This Run
- Attempts: 1
- Plan Size: 45 lines
- Diff: +67/-23 (4 files)

### Updated Summary
- Current Streak: 6 (was 5)
- Longest Streak: 12 (unchanged)
- Average Presence: 1.28 (improved)

### Analysis
[Trend observations and recommendations]

Anti-Patterns

• Gaming metrics (easy tasks only)
• Ignoring failures (not counting retries)
• Not tracking consistently
• Celebrating streaks over actual delivery

Version History

• v1.0.0 (2025-12-26): Initial release

---

Last Updated

Date: 2025-12-26 Model: claude-opus-4-5-20251101