custom-metric-builder

custom-metric-builder

Custom metric for adk eval — score on whatever matters for your domain (medical accuracy, code correctness, brand voice, citation quality).

When to use

• Default tool-match / similarity scores miss what you actually care about
• Domain expertise needed (e.g., medical correctness, legal accuracy)
• LLM-as-judge for subjective qualities (helpfulness, tone)
• Rubric scoring with multiple dimensions

Template — programmatic metric

from google.adk.evaluation import Metric, EvalCaseResult

class CitationCountMetric(Metric):
    """Count [source: URL] citations in the agent's final response."""

    name = "citation_count"

    def evaluate(self, case_result: EvalCaseResult) -> dict:
        text = case_result.final_response_text
        import re
        citations = re.findall(r"\[source:\s*https?://", text)
        return {
            "score": min(len(citations) / 3.0, 1.0),  # normalize: 3+ = full marks
            "details": {"count": len(citations)},
        }

Template — LLM-as-judge metric

from google.adk.evaluation import Metric
from google.adk.models.lite_llm import LiteLlm

JUDGE = LiteLlm(model="gemini-2.5-pro")

class MedicalAccuracyMetric(Metric):
    name = "medical_accuracy"

    async def evaluate(self, case_result):
        rubric = (
            "Score the response 0-10 on medical accuracy. "
            "Penalize: incorrect dosing, contraindications missed, "
            "non-evidence-based claims. Output JSON: {score: int, rationale: str}."
        )
        out = await JUDGE.complete(
            f"{rubric}\n\nResponse:\n{case_result.final_response_text}"
        )
        import json
        parsed = json.loads(out)
        return {"score": parsed["score"] / 10.0, "details": {"rationale": parsed["rationale"]}}

Register and run

from google.adk.evaluation import EvalRunner

runner = EvalRunner(
    agent=root_agent,
    metrics=[CitationCountMetric(), MedicalAccuracyMetric()],
)
report = await runner.run("./eval_set.evalset.json")

CLI flag form:

adk eval ./agent.py ./eval_set.evalset.json \
    --metrics my_module.CitationCountMetric,my_module.MedicalAccuracyMetric

Multi-dimensional rubric

class TripleRubric(Metric):
    name = "triple_rubric"

    async def evaluate(self, case_result):
        scores = {
            "accuracy": await self._judge(case_result, "factual accuracy"),
            "tone": await self._judge(case_result, "professional tone"),
            "brevity": await self._judge(case_result, "conciseness"),
        }
        return {"score": sum(scores.values()) / 3, "details": scores}

Validation

• Metric returns {"score": float in [0,1], "details": dict}
• Deterministic on the same input (LLM judges: temperature=0, fixed seed where supported)
• Calibrate against human-labeled sample (10-20 cases) before trusting at scale
• Save metric outputs to evals/results/ for trend analysis

See also

• eval-set-generator for the test cases this scores
• agent-optimization-loop to use these metrics as the optimization signal