plugin-introspector

Plugin Introspector — White-box Monitoring & Self-Improvement Meta-Plugin

A meta-plugin that monitors Claude Code workflow plugin execution at white-box level, collects execution data, and generates data-driven improvement proposals.

Role

A monitoring and analysis meta-plugin that provides end-to-end visibility into Claude Code workflow plugin execution. Collects tool traces, API interactions, token usage, and execution patterns via hooks. Provides analysis agents that identify bottlenecks, evaluate quality, detect anomalies, and generate concrete improvement proposals.

Terminology

• Plugin: An installable package (directory with .claude-plugin/plugin.json and skills/)
• Skill: An invocable unit within a plugin (directory with SKILL.md)
• Agent: A Task subagent definition (.md file in agents/)
• Proposal / Improvement: A concrete, ROI-scored change suggestion generated by improve

Core Principles

1. Data-driven improvement: Never guess — always base decisions on collected execution data
2. Minimal overhead: Hook scripts must complete in <50ms, use append-only JSONL
3. OTel compatibility: Trace data follows OpenTelemetry GenAI Semantic Conventions
4. Tiered collection: Tier 0 (pure hooks) → Tier 1 (+ OTel Collector with native telemetry)
5. Meta-rules compliance: All generated improvements must pass meta-rules validation
6. Zero external dependencies: bash + jq only — no Python, no Node.js (Tier 1 adds optional otelcol-contrib)

---

Pre-Execution: Session Resolution

Before any command, resolve the session directory:

1. Run: ls -t ~/.claude/plugin-introspector/sessions/ | head -1
   → Gives the most recent session ID
2. If --session {id} was provided: use that ID instead
3. Set: SESSION_DIR = ~/.claude/plugin-introspector/sessions/{resolved-id}
4. Verify: stats.json exists in SESSION_DIR
5. If not found: display "No session data. Run a session with hooks enabled first."

Full details: resources/orchestration-protocol.md

Pre-Execution: Target Plugin Resolution

When --target {plugin-name} is specified:

1. Search for plugin at: {working_dir}/plugins/{plugin-name}/
2. Fallback: check marketplace.json for source path
3. Catalog plugin components → store in SESSION_DIR/target_plugin.json
4. If not found: display "Plugin '{name}' not found" with available list

When --target is omitted:

• Infer plugin from traces: scan input_summary fields for plugin-specific paths
• If Skill tool invoked: identify from skill name
• If no plugin identified: treat as general session analysis

---

Data Storage

~/.claude/plugin-introspector/
├── sessions/{session-id}/
│   ├── meta.json              # Session metadata (git info, collection_tier)
│   ├── tool_traces.jsonl      # Pre/post tool traces with input_summary
│   ├── api_traces.jsonl       # API request/response metrics
│   ├── otel_traces.jsonl      # OTel spans (hook-generated or merged native)
│   ├── stats.json             # Aggregated session statistics
│   ├── evaluation.json        # Quality evaluation results (from evaluate)
│   ├── security_events.jsonl  # DLP violations, command risk events (from security hooks)
│   └── target_plugin.json     # Discovered target plugin info (from --target)
├── session_history.jsonl      # Cross-session summary records
├── evaluation_history.jsonl   # Cross-session evaluation scores
├── alerts.jsonl               # Anomaly detection alerts
├── improvement_log.jsonl      # Applied improvement history
├── aggregates.json            # Pre-computed cross-session aggregates (from Stop hook)
├── security_baseline.json     # Security baseline for anomaly comparison (optional)
├── telemetry.jsonl            # Opt-in anonymous telemetry (PI_TELEMETRY=1)
├── plugin-profiles/{plugin}/  # Per-plugin workflow profiles and baselines
├── otel-export/               # Tier 1: OTel Collector File Exporter output
└── otel-collector/            # Tier 1: Collector binary and config

---

Commands

Quick Reference

Command	Agent	Options
status	—	--session {id}
dashboard	—	--session {id}, --full
flow	—	--session {id}
profile	plugin-profiler	--target {plugin} (required)
analyze	workflow-analyzer	--session {id}, --target {plugin}
tokens	token-optimizer	--session {id}
api	api-tracker	--session {id}
context	context-auditor	--session {id}, --target {plugin}
evaluate	quality-evaluator	--session {id}, --target {plugin}
alerts	anomaly-detector	--session {id}
optimize *	auto-optimizer	--target {plugin}, --component {file}
improve	improvement-generator	--session {id}, --target {plugin} (required)
report	5 agents (see below)	--session {id}, --target {plugin}
apply	—	(reads proposals from last improve/optimize)
trace	—	--tool {name}, --errors, --slow, --last {N}
web	—	--otel, --json, --csv
quick-scan	quick-scanner	--target {plugin} (required)
security-scan	—	--target {plugin} (required)
security-audit	security-auditor	--session {id}
security-dashboard	—	--session {id}
compliance-report *	security-reporter	--period {30d|7d|date~date}
rotate-data	—	--dry-run, env: PI_RETENTION_DAYS, PI_RETENTION_LINES
otel-setup *	—	install, start, stop, status, env
otel-security-map *	—	--watch, --stats

* available, experimental

---

Agent Invocation Pattern

Details: resources/orchestration-protocol.md

All analysis agents are invoked via the Task tool using the prompt template defined in orchestration-protocol.md:

1. Read the agent definition: agents/{agent-name}.md
2. Read the relevant JSONL data files from SESSION_DIR (apply truncation per Data Size Management)
3. Construct prompt from Task Prompt Template (see orchestration-protocol.md)
4. Invoke Task with subagent_type: "general-purpose", model per agent's Model Assignment
5. Parse agent's JSON output
6. Display results / write to SESSION_DIR

For multi-agent commands (report, improve, optimize):

1. Read all data files once and reuse across agents
2. For 'report': run independent agents in parallel (workflow-analyzer, token-optimizer, api-tracker, context-auditor), then quality-evaluator
3. For 'improve': run analysis agents + load cross-session data (evaluation_history, improvement_log, plugin profile), aggregate all, pass to improvement-generator with improvement-pipeline.md procedure
4. For 'optimize': read evaluation.json + evaluation_history + improvement_log + target component, pass to auto-optimizer

---

status — Current Session Overview

1. Resolve session directory
2. Read stats.json
3. Display: session ID, duration, tool calls, token estimate, error count, top tools

---

dashboard — htop-style Real-time Dashboard

1. Resolve session directory
2. Run Bash: scripts/dashboard.sh {session-id} [--full]
3. Renders token progress bar, per-tool breakdown, error rate

---

flow — Execution Flow Tree (OTel)

1. Resolve session directory
2. Read otel_traces.jsonl
3. Build parent-child span tree using parent_span_id
4. Display indented tree with timing and token info

---

profile — Generate Plugin Profile

1. Resolve target plugin (required: --target)
2. Read target plugin's main SKILL.md content
3. Read target plugin's component catalog (from target_plugin.json)
4. Read agent: agents/plugin-profiler.md
5. Invoke Task (haiku): pass SKILL.md content + component catalog
6. Output: workflow type, phases, detection patterns, key files
7. Write result to ~/.claude/plugin-introspector/plugin-profiles/{plugin}/profile.json
8. Create empty phase-baselines.json and learned-patterns.jsonl if not present

---

analyze — Deep Workflow Analysis

1. Resolve session + target plugin
2. Read tool_traces.jsonl + otel_traces.jsonl
3. Read agent: agents/workflow-analyzer.md
4. Invoke Task (sonnet): pass agent definition + data
5. Output: patterns, bottlenecks, efficiency ratio, recommendations

---

tokens / api / context — Single-Agent Analysis Commands

All follow the same pattern: resolve session → read data → invoke agent → display output.

Command	Agent	Data Files	Extra
tokens	token-optimizer (sonnet)	tool_traces.jsonl + api_traces.jsonl + stats.json	—
api	api-tracker (sonnet)	api_traces.jsonl	—
context	context-auditor (sonnet)	api_traces.jsonl + tool_traces.jsonl	--target: read SKILL.md + agents for static cost

---

evaluate — Quality Evaluation (LLM-as-Judge)

1. Resolve session + target plugin (optional: --target scopes evaluation)
2. Read tool_traces.jsonl + otel_traces.jsonl + stats.json
3. Read agent: agents/quality-evaluator.md
4. Invoke Task (sonnet): pass agent definition + data
5. Output: 4-dimension scores + quantified improvement signals
6. Write result to SESSION_DIR/evaluation.json
7. Append to ~/.claude/plugin-introspector/evaluation_history.jsonl:
   • From quality-evaluator output: weighted_score, scores
   • From stats.json: key_metrics (tool_calls, total_tokens_est, errors, error_rate, duration_ms)
   • From orchestrator phase detection (via plugin profile + trace analysis): phase_breakdown (optional, only for phased workflows)
   • From quality-evaluator improvement_signals: top_waste_sources (top 3 waste entries)
   • From improvement_log: improvements_active (IDs where status == "applied")
8. Update phase baselines (if target_plugin has profile with workflow.type == "phased"):
   • Read plugin-profiles/{plugin}/phase-baselines.json
   • Update running mean/stddev for each phase using current phase_breakdown
   • Increment sessions_count, set maturity.baselines_available = true if count ≥ 5
9. Closed Loop Check: Read ~/.claude/plugin-introspector/improvement_log.jsonl
   • Find entries where status == "applied" and post_score == null
   • If --target specified: only check entries matching target_plugin
   • For each pending entry: compare current weighted_score with pre_score
   • Update post_score and set status to validated or regressed
   • If regressed (score dropped >0.5): suggest rollback via improvement-apply-protocol.md

---

alerts — Anomaly Detection

1. Read ~/.claude/plugin-introspector/alerts.jsonl → display recent alerts
2. If deeper analysis requested:
   • Resolve session
   • Read tool_traces.jsonl + session_history.jsonl + alerts.jsonl
   • Read agent: agents/anomaly-detector.md
   • Invoke Task (haiku): pass agent definition + data (including alerts.jsonl for deduplication)
3. Output: alert list with severity, suggested actions

---

optimize — Auto-Optimize (APE Loop) (available, experimental)

1. Resolve target plugin (required: --target)
2. Resolve target component (--component or auto-select lowest-scoring)
3. Read evaluation.json (run evaluate first if missing)
4. Read evaluation_history.jsonl + improvement_log.jsonl (for contrastive analysis + historical learning)
5. Read agent: agents/auto-optimizer.md
6. Read target component file content
7. Invoke Task (opus): pass agent definition + evaluation + history + component content
8. Output: optimized version with diff + predicted score improvement
9. Display diff for user review — do NOT auto-apply

CAUTION: Review all diffs before applying. See resources/improvement-apply-protocol.md.

---

improve — Generate Improvement Proposals

1. Resolve session + target plugin (required: --target)
2. Run analysis pipeline (or reuse if already run in this session):
   • Parallel: workflow-analyzer, token-optimizer, context-auditor
   • Parallel: anomaly-detector, quality-evaluator
3. Load cross-session data:
   • evaluation_history.jsonl (last 20 records)
   • improvement_log.jsonl (all for target plugin)
   • Plugin profile: profile.json, phase-baselines.json, learned-patterns.jsonl
4. Read target plugin component files (apply file selection per orchestration-protocol.md)
5. Read agent: agents/improvement-generator.md
6. Read procedure: resources/improvement-pipeline.md
7. Invoke Task (opus): pass agent definition + pipeline procedure + all analysis results + cross-session data + selected plugin files
8. Output: ROI-scored proposals with quantified evidence, counterfactuals, diffs, meta-rules validation
9. Display proposals for user review — do NOT auto-apply

---

report — Full Analysis Report

1. Resolve session + target plugin
2. Read all JSONL data files once
3. Run independent agents in parallel (4 concurrent Task calls):
   • workflow-analyzer → analysis
   • token-optimizer → tokens
   • api-tracker → api
   • context-auditor → context
4. Run quality-evaluator sequentially (can use results from step 3 if needed)
5. Aggregate into unified report with sections:
   • Executive Summary, Workflow Analysis, Token Efficiency, API Performance, Context Audit, Quality Score, Recommendations

---

apply — Apply Improvement Proposals

1. Read the most recent improve or optimize output (proposals)
2. For each proposal, follow improvement-apply-protocol.md:
   • Display proposal summary + diff for user review
   • Wait for explicit user confirmation
   • Backup → Edit → Verify → Log to improvement_log.jsonl
3. After all approved proposals applied:
   • Display summary of changes
   • Suggest: "Run evaluate --target {plugin} after next session to validate improvements"

CAUTION: Never auto-apply. Each proposal requires explicit user approval.

---

quick-scan — Quick Plugin Diagnosis (1-minute)

1. Resolve target plugin (required: --target)
2. Analyze plugin structure: plugin.json, SKILL.md, agents, scripts, resources
3. Run security-scan.sh for security score
4. Read agent: agents/quick-scanner.md
5. Invoke Task (haiku): lightweight analysis
6. Output: formatted box diagram with structure summary, security score, recommendations
7. Suggest follow-up commands for detailed analysis

Use case: First-time plugin evaluation, PR review, quick health check

Details: agents/quick-scanner.md

---

rotate-data — Data Retention Management

1. Run Bash: scripts/rotate-data.sh
2. Deletes session directories older than PI_RETENTION_DAYS (default: 30)
3. Trims JSONL files to PI_RETENTION_LINES (default: 1000)
4. Displays summary of deleted sessions and freed space

Environment variables:

• PI_RETENTION_DAYS=30: Days to keep session directories
• PI_RETENTION_LINES=1000: Lines to keep in JSONL files
• PI_DRY_RUN=1: Preview what would be deleted without actually deleting
• PI_AUTO_ROTATE=1: Auto-run at session end (optional)

Example:

# Dry run to see what would be deleted
PI_DRY_RUN=1 /plugin-introspector rotate-data

# Keep only 7 days of data
PI_RETENTION_DAYS=7 /plugin-introspector rotate-data

---

security-scan — Plugin Static Security Analysis

1. Resolve target plugin (required: --target)
2. Run Bash: scripts/security-scan.sh {plugin-path}
3. Scans hook scripts for dangerous patterns (data exfiltration, reverse shells, credential theft)
4. Scans SKILL.md + resources for prompt injection patterns
5. Scans agent definitions for risky tool permission combinations
6. Output: JSON report with findings, severity levels, risk score
7. Logs findings to alerts.jsonl if any found

Environment variables:

• PI_ENABLE_SECURITY=1: Enables runtime command risk logging
• PI_ENABLE_DLP=1: Enables DLP scanning
• PI_SECURITY_BLOCK=1: Enables CRITICAL command blocking (use with caution)

Note: security-check.sh intentionally omits || true in plugin.json so that exit 2 can block CRITICAL commands when PI_SECURITY_BLOCK=1. All other hook scripts use || true per meta-rules.

Details: resources/security-patterns.md

---

security-audit — Session Security Audit

1. Resolve session
2. Read tool_traces.jsonl + otel_traces.jsonl + stats.json + security_events.jsonl + alerts.jsonl
3. Read agent: agents/security-auditor.md
4. Invoke Task (haiku): pass agent definition + all data (including stats.json)
5. Output: risk level, file access audit, command audit, suspicious sequences, DLP summary
6. Recommendations for security improvements

---

security-dashboard — Security Risk Visualization

1. Resolve session
2. Run Bash: scripts/security-dashboard.sh {session-id}
3. Renders security-focused dashboard with:
   • Overall risk score bar
   • DLP violations, sensitive reads/writes, risky commands
   • Recent security events
   • Security configuration status (DLP, security check, blocking)

---

compliance-report — Compliance Report Generation (available, experimental)

1. Read session_history.jsonl for specified period
2. Read alerts.jsonl + security_events.jsonl + improvement_log.jsonl (aggregated)
3. Optionally run security-scan for each active plugin
4. Read agent: agents/security-reporter.md
5. Invoke Task (sonnet): pass agent definition + period data
6. Output: SOC 2 / ISO 27001 compliance report with executive summary, security events, plugin audit, recommendations, and compliance mapping

---

otel-setup — OTel Collector Setup (Tier 1) (available, experimental)

1. Run Bash: scripts/setup-otel-collector.sh {subcommand}
   • install: Download OTel Collector Contrib binary (~100MB+)
   • start: Start collector with otel-config.yaml (OTLP receiver → file exporter)
   • stop: Stop collector
   • status: Show collector status, export data info, environment variables
   • env: Print shell environment variables to set
2. Display output to user

After setup, the collection tier automatically upgrades from Tier 0 to Tier 1. Hook scripts detect the running collector and skip redundant OTel span generation. Native OTel spans (with accurate token counts) are merged at session end.

See: Collection Tiers in orchestration-protocol.md

---

otel-security-map — OTel Security Event Mapper (available, experimental)

1. Run Bash: scripts/otel-security-mapper.sh [session-id|--watch|--stats]
2. Converts native Claude Code OTel events to PI security_events.jsonl format
3. Enables deeper security analysis using OTel's tool_parameters (bash_command, full_command)
4. Output modes:
   • <session-id>: Process specific session's OTel data
   • --watch: Real-time mapping mode
   • --stats: Show security statistics from OTel data

Use case: Enhanced security analysis with native OTel data (Tier 1+)

---

trace — Raw Trace Viewer

1. Resolve session
2. Read tool_traces.jsonl with filters: --tool {name}, --errors, --slow, --last {N}
3. Display formatted records

---

web — Export for External Tools

1. Resolve session
2. Export format: --otel, --json, --csv
3. Write export file, display path

---

Environment Variables

All environment variables are opt-in. Default behavior requires no configuration.

Variable	Default	Description
PI_ENABLE_SECURITY	0	Enable runtime command risk logging
PI_ENABLE_DLP	0	Enable DLP (sensitive data detection)
PI_SECURITY_BLOCK	0	Block CRITICAL commands (use with caution)
PI_RETENTION_DAYS	30	Days to keep session data
PI_RETENTION_LINES	1000	Lines to keep in JSONL files
PI_DRY_RUN	0	Preview data rotation without deleting
PI_AUTO_ROTATE	0	Auto-rotate data at session end
PI_SHOW_REMINDER	1	Show PI commands reminder at session start
PI_TELEMETRY	0	Enable opt-in anonymous telemetry (local only)
PI_BASELINE_MAX_AGE	30	Days of history for security baseline

Telemetry Policy (PI_TELEMETRY):

• Data stored locally in telemetry.jsonl — never transmitted externally
• Collects: command counts only (no content, paths, or personal data)
• Use scripts/telemetry.sh status to view collected data

---

Resources (On-demand)

Document	Purpose
orchestration-protocol.md	Agent invocation, data passing, pipeline composition
improvement-apply-protocol.md	Safe improvement application with rollback
improvement-pipeline.md	Quantified improvement generation procedure
data-schema.md	JSONL record format specifications
security-patterns.md	Security threat patterns, DLP signatures, compliance mapping
ci-cd-integration.md	GitHub Actions, GitLab CI, Jenkins pipeline examples
skill-activation-guide.md	Optional Forced Eval Hook installation for 84% skill activation
otel-collector-guide.md	OTel Collector setup with ClickStack/HyperDX
docker-compose.otel.yml	Docker Compose for OTel stacks
otel-collector-config.yaml	Standalone OTel Collector configuration

Agents

Agent	Purpose	Model
workflow-analyzer	Execution pattern analysis, bottleneck detection	sonnet
token-optimizer	Token waste identification, optimization suggestions	sonnet
context-auditor	Context window usage audit	sonnet
api-tracker	API interaction monitoring and analysis	sonnet
quality-evaluator	LLM-as-Judge multi-dimension evaluation	sonnet
anomaly-detector	Real-time anomaly detection via Z-score/MA	haiku
plugin-profiler	Target plugin workflow profiling	haiku
improvement-generator	Concrete improvement proposal generation	opus
auto-optimizer	APE-based automatic prompt optimization	opus
security-auditor	Session security audit and risk classification	haiku
security-reporter	Compliance report generation (SOC 2, ISO 27001)	sonnet
quick-scanner	Fast plugin structure and security diagnosis	haiku

Skills (Knowledge Base)

Skill	Purpose
meta-rules	Agent/skill writing rules and anti-bloat constraints
analysis-patterns	Reusable analysis patterns and heuristics
cost-tracking	Model-specific pricing and cost calculation

---

Closed Loop: Improve → Apply → Validate

The full improvement cycle connects improve, evaluate, and the apply protocol:

┌─────────┐     ┌───────────┐     ┌─────────┐     ┌──────────┐
│ improve  │────▶│ user      │────▶│  apply   │────▶│ log to   │
│ (propose)│     │ review    │     │ (backup, │     │ improve- │
│          │     │ (approve?)│     │  edit,   │     │ ment_log │
│          │     │           │     │  verify) │     │ .jsonl   │
└─────────┘     └───────────┘     └─────────┘     └────┬─────┘
                                                       │
                                                       ▼
┌─────────┐     ┌───────────┐     ┌─────────────────────────┐
│ rollback │◀───│ regressed │◀───│ evaluate (next session)  │
│ (if bad) │    │ score?    │    │ → checks pending         │
│          │    │           │    │   post_scores in log     │
└─────────┘    └───────────┘    └─────────────────────────┘

Lifecycle

1. improve generates proposals with diffs and meta-rules validation
2. User reviews each proposal (never auto-apply)
3. Apply follows improvement-apply-protocol.md:
   • Backup → Edit → Verify → Log to improvement_log.jsonl with post_score: null
4. Next evaluate run (step 9 — Closed Loop Check) detects pending entries and fills post_score
5. If validated (post_score >= pre_score): mark status: "validated"
6. If regressed (post_score < pre_score - 0.5): suggest rollback, mark status: "regressed"

Rollback Trigger

When evaluate detects regression:

1. Display: "Improvement {proposal_id} caused score regression: {pre_score} → {post_score}"
2. Offer rollback: "Restore from backup? (yes/no)"
3. If yes: follow rollback procedure in improvement-apply-protocol.md
4. Log rollback to improvement_log.jsonl

---

Self-Referential Loop Prevention

1. Hook scripts check CLAUDE_TOOL_NAME + structured Skill name via jq — skip if Skill invocation targets plugin-introspector
2. Analysis agents exclude introspector-related tool calls from analysis
3. When running improve --target plugin-introspector: extra caution, meta-rules strictly enforced