llm-testing-expert

LLM Testing and Evaluation

Description

Provide LLM test strategy design, test dataset construction, automated evaluation, and security red teaming recommendations to ensure models meet production requirements for functionality, performance, robustness, and safety.

When to Use

• User requests "design LLM testing strategy" or "how to evaluate model quality"
• User needs to build test datasets, design test cases, or perform regression testing
• User seeks prompt optimization, engineering, and version management recommendations
• User needs to evaluate model performance metrics (accuracy, hallucination rate, safety, latency, cost)
• User asks how to conduct A/B testing or comparative evaluation (multi-model/multi-version)
• User needs automated testing workflows or CI/CD integration
• User requests red team testing design (jailbreak, prompt injection, adversarial attacks)
• User asks about specialized testing strategies for RAG systems or Agent applications

When NOT to Use

• User only needs traditional software testing (unit tests, integration tests) without LLM-specific concerns
• User's problem is about model training or fine-tuning techniques, not testing/evaluation
• User needs data annotation or data cleaning, not test strategy design
• User only asks how to use an LLM framework or tool, not testing strategy
• User's problem is pure Prompt Engineering (how to write better prompts), not how to test prompt effectiveness

Input

{
  testTarget: {
    type: string                  // Test object type (base-model/fine-tuned-model/rag-system/agent/prompt-template)
    modelInfo?: {
      name: string                // Model name (e.g., gpt-4, claude-3.5-sonnet)
      version?: string            // Version
      deployment?: string         // Deployment method (API/self-hosted)
    }
    applicationContext?: string   // Application scenario (e.g., customer service, code generation, document summarization)
  }
  testObjectives: {
    functional?: boolean          // Functional correctness testing
    performance?: boolean         // Performance testing (latency, throughput, cost)
    robustness?: boolean          // Robustness testing (adversarial samples, edge cases)
    safety?: boolean              // Safety testing (jailbreak, injection, PII leakage)
    userExperience?: boolean      // User experience testing
  }
  constraints: {
    budget?: string               // Testing budget (API call cost/manual annotation cost)
    timeline?: string             // Testing timeline
    existingTestAssets?: string[] // Existing test assets (test sets, annotated data)
    complianceRequirements?: string // Compliance requirements (GDPR, industry regulations)
  }
  riskAreas?: string[]            // Known risk areas (e.g., hallucination, bias, privacy leakage)
  existingMetrics?: string        // Existing evaluation metrics and baselines
}

Output

{
  testStrategy: {
    scope: string                 // Test scope definition
    approach: string              // Test approach (black-box/white-box/gray-box)
    testLevels: {
      unit?: string               // Unit testing (single prompt/single tool call)
      integration?: string        // Integration testing (multi-turn dialogue/tool chain)
      system?: string             // System testing (end-to-end scenarios)
      acceptance?: string         // Acceptance testing (user scenario coverage)
    }
  }
  testPlan: {
    functional: {
      testCases: {
        id: string
        scenario: string          // Test scenario
        input: string             // Input sample
        expectedOutput: string    // Expected output (can use fuzzy rules)
        passCriteria: string      // Pass criteria
      }[]
      coverage: string[]          // Coverage dimensions (instruction following/format output/reasoning ability, etc.)
    }
    performance: {
      metrics: {
        name: string              // Metric name (latency/throughput/token-cost)
        target: string            // Target value (e.g., p95 < 2s)
        measurement: string       // Measurement method
      }[]
      loadProfile: string         // Load model (concurrent users, request pattern)
    }
    robustness: {
      adversarialCases: string[]  // Adversarial sample design
      edgeCases: string[]         // Edge cases
      stressScenarios: string[]   // Stress scenarios (extra-long input, extreme parameters)
    }
    safety: {
      redTeamScenarios: {
        type: string              // Attack type (jailbreak/injection/data-extraction)
        technique: string         // Attack technique
        expectedDefense: string   // Expected defense measures
      }[]
      harmfulContentCategories: string[] // Harmful content categories (violence/discrimination/privacy)
    }
  }
  testDataset: {
    sources: string[]             // Data sources (public benchmarks/domain data/synthetic data)
    composition: {
      positive: number            // Positive case ratio
      negative: number            // Negative/adversarial case ratio
      edge: number                // Edge case ratio
    }
    sampleSize: string            // Sample size (calculated by statistical significance)
    labelingStrategy: string      // Labeling strategy (manual/automated/hybrid)
  }
  evaluationMethod: {
    automated: {
      metrics: string[]           // Automated metrics (BLEU/ROUGE/exact-match/regex)
      tools: string[]             // Evaluation tools
    }
    humanEval: {
      criteria: string[]          // Human evaluation criteria
      raterGuidelines: string     // Annotator guidelines
      interRaterAgreement: string // Consistency requirement (e.g., Kappa > 0.7)
    }
    llmAsJudge?: {
      judgeModel: string          // Judge model
      rubric: string              // Scoring rules
      calibration: string         // Calibration method (align with human annotation)
    }
  }
  regressionPlan: {
    triggerConditions: string[]   // Conditions triggering regression (model update/prompt change)
    baselineVersion: string       // Baseline version
    comparisonMetrics: string[]   // Comparison metrics
    reportFormat: string          // Report format
  }
  cicdIntegration?: string        // CI/CD integration solution
  specializedTests?: {
    rag?: {
      retrievalQuality: string    // Retrieval quality testing (recall/ranking)
      citationAccuracy: string    // Citation accuracy testing
      faithfulness: string        // Faithfulness testing (based only on retrieved content)
    }
    agent?: {
      toolCallCorrectness: string // Tool call parameter correctness
      planningRationality: string // Planning rationality
      errorRecovery: string       // Error recovery capability
    }
  }
}

Execution Workflow

Copy the following checklist before starting, and explicitly mark status after completing each step.

Step 1: Test Objective and Risk Identification

• Clarify test object type (base model/fine-tuned model/RAG/Agent/prompt template)
• Confirm core test objectives (functional/performance/robustness/safety/user experience)
• Identify known risk areas (hallucination, bias, privacy leakage, prompt injection)
• Understand application scenario and constraints (budget, timeline, compliance requirements)

Feedback Loop: If test objectives are unclear or conflicting (e.g., comprehensive coverage with extremely low cost), MUST align priorities with user.

Step 2: Test Strategy Formulation

• Choose test approach (black-box/white-box/gray-box)
• Define test levels (unit/integration/system/acceptance), adopt test pyramid model
• Determine test coverage (functional dimensions, performance metrics, security scenarios)
• Select evaluation methods (automated/manual/LLM-as-a-Judge/hybrid)

Test Pyramid Principle:

• Bottom (unit tests): Most numerous, lowest cost, fastest execution (e.g., single prompt function test)
• Middle (integration tests): Moderate quantity, test component interaction (e.g., multi-turn dialogue, tool chain)
• Top (system tests): Few critical scenarios, end-to-end validation (e.g., complete user journey)

Feedback Loop: If user budget or timeline is extremely limited, prioritize designing high-priority smoke tests rather than pursuing comprehensive coverage.

Step 3: Test Dataset Construction

• Determine data sources (public benchmarks like MMLU/HumanEval, domain data, anonymized production logs, synthetic data)
• Design case distribution: Positive cases (60-70%) + Negative/adversarial cases (20-30%) + Edge cases (10%)
• Define ID, scenario, input, expected output, pass criteria for each case
• Determine sample size (based on statistical significance, typically requires 100+ samples)

Case Design Principles:

• Positive cases: Cover core functionality and common scenarios
• Negative cases: Test rejection capability, error handling, adversarial samples
• Edge cases: Extra-long input, extreme parameters, multilingual mix, special characters

Feedback Loop: If existing test assets are insufficient, prioritize sampling real cases from production logs rather than fully synthetic data.

Step 4: Evaluation Method Design

• Automated evaluation: Select appropriate metrics (BLEU/ROUGE/exact-match/regex/structured output validation)
• Manual evaluation: Develop evaluation criteria, annotator guidelines, ensure consistency (Kappa > 0.7)
• LLM-as-a-Judge: Select judge model, design scoring rules, calibrate with human annotation

Evaluation Method Selection:

• Simple tasks (e.g., classification, extraction): Automated evaluation (exact-match/F1)
• Complex tasks (e.g., summarization, creation): LLM-as-a-Judge + human sampling validation
• Safety testing: Manual evaluation (detect harmful content, jailbreak success rate)

Feedback Loop: If automated metrics are inconsistent with manual evaluation, MUST recalibrate or adjust metric weights.

Step 5: Specialized Test Design (if applicable)

RAG System Testing

• Retrieval quality: Test recall (Recall@K), ranking quality (MRR/NDCG)
• Citation accuracy: Verify citation sources are correct, check for hallucinated citations
• Faithfulness: Check if answer is based only on retrieved content, no fabrication

Agent System Testing

• Tool call correctness: Verify tool selection and parameter passing are correct
• Planning rationality: Assess if step planning is efficient and logic is coherent
• Error recovery: Test handling capability when encountering tool failures or exceptions

Feedback Loop: If RAG/Agent testing reveals systematic issues (e.g., poor retrieval quality, high tool call failure rate), should return to system design level for optimization, not just adjust testing.

Step 6: Versioning and Regression Testing

• Record model version, prompt version, test dataset version
• Define regression trigger conditions (model update/prompt change/test set expansion)
• Establish baseline and compare new version performance
• Generate traceable test reports (including pass rate, performance comparison, failure case analysis)

Regression Testing Principles:

• Every change must pass core test set (Golden Set)
• New features must add corresponding test cases
• Performance degradation exceeding threshold (e.g., accuracy drop >5%) should block release

Feedback Loop: If regression testing finds performance degradation, MUST analyze root cause (model issue/prompt issue/test set issue), not just rollback directly.

Step 7: Security Red Team Testing (if applicable)

• Design jailbreak scenarios: Test bypassing safety guardrails
• Design injection attacks (Prompt Injection): Test malicious instruction injection
• Design data extraction attacks: Test PII leakage, training data memorization
• Design harmful content generation: Test violence, discrimination, misinformation, etc.

Red Team Testing Methods:

• Manual testing: Security experts design adversarial samples
• Automated attacks: Use tools (e.g., Garak, PromptInject) to generate attack samples
• Crowdsourced red teaming: Invite external parties to attempt attacks

Feedback Loop: If red team testing finds serious vulnerabilities, MUST prioritize fixing and retesting, not cover up or ignore.

Failure Handling

Insufficient Test Coverage

• Symptom: User-provided test objectives are too broad, cannot comprehensively cover with limited resources
• Action: Design tiered test plan based on risk priority (high-risk scenarios first), clarify which scenarios are not covered

Wrong Evaluation Metrics

• Symptom: User uses inappropriate metrics (e.g., using BLEU to evaluate dialogue quality)
• Action: Explain metric limitations, recommend more suitable metric combinations (e.g., dialogue quality uses LLM-as-a-Judge + manual evaluation)

Automated and Manual Evaluation Inconsistency

• Symptom: Automated metrics show performance improvement, but manual evaluation finds quality degradation
• Action: Analyze inconsistency reasons (metric design issue/annotation bias/model overfitting), recalibrate evaluation system

Test Data Quality Issues

• Symptom: Test set contains large amounts of noise, duplicates, or unrealistic samples
• Action: Establish test set quality check process (deduplication/anomaly detection/representativeness verification), prioritize using anonymized real production data

Red Team Testing Finds Serious Vulnerabilities

• Symptom: Model can be easily jailbroken or injected
• Action: Immediately pause deployment, fix security issues (strengthen input filtering/output review/model fine-tuning), re-conduct red team testing

See Also

• deep-learning-expert — training and fine-tuning the models under test.
• python-expert — building evaluation harnesses, fixtures, and pytest-based regression suites.
• software-architect — designing observable, A/B-testable LLM application infrastructure.