fact-check

Fact Check

Verify a claim by actively searching for BOTH supporting and contradicting evidence. Counteracts LLM confirmation bias by enforcing adversarial search.

Steps

1. Decompose the claim into verifiable sub-claims:

   • Separate factual assertions from opinions
   • Identify the core falsifiable statement
   • Note the domain: scientific, technical, historical, statistical, or current events

2. Search for SUPPORTING evidence using search:

   • search(action="search", query="[claim as stated]")
   • For scientific claims: search(action="search", query="[claim] research evidence", search_type="academic")
   • Extract 2-3 strongest supporting sources with extract

3. Search for CONTRADICTING evidence (mandatory — do NOT skip):

   • search(action="search", query="[claim] debunked OR wrong OR myth OR criticism")
   • search(action="search", query="[opposite of claim] evidence")
   • For scientific claims: search(action="search", query="[claim] replication failure OR meta-analysis", search_type="academic")
   • Extract 2-3 strongest contradicting sources with extract

4. Assess source quality for each piece of evidence:

   • Tier 1: Peer-reviewed journals, official statistics, primary sources
   • Tier 2: Established news outlets, institutional reports, expert analysis
   • Tier 3: Blogs, forums, social media, opinion pieces
   • Tier 4: Anonymous sources, undated content, known biased outlets
   • Discard Tier 4 unless no better sources exist

5. Produce verdict with structured output:

   • Claim: [exact claim being checked]
   • Verdict: Supported / Partially Supported / Disputed / Unsupported / Insufficient Evidence
   • Confidence: High / Medium / Low (based on source tier and agreement)
   • Supporting evidence: [bullet list with source tier tags]
   • Contradicting evidence: [bullet list with source tier tags]
   • Nuance: [important caveats, context-dependence, or scope limitations]

Decision Tree: Choosing Verdict

• All Tier 1-2 sources agree -> Supported (High confidence)
• Tier 1-2 support, some Tier 2-3 contradict -> Partially Supported (Medium)
• Tier 1-2 sources disagree with each other -> Disputed (note the split)
• Only Tier 3-4 sources support, Tier 1-2 contradict -> Unsupported (High)
• Fewer than 2 sources found either way -> Insufficient Evidence (Low)

Common Pitfalls

• Confirmation bias: Searching only for supporting evidence. The contradicting search in step 3 is MANDATORY.
• Recency bias: Old but well-cited research may outweigh recent blog posts.
• Authority bias: A claim from a famous person still needs evidence.
• Survivorship bias: "X worked for company Y" does not mean X works in general.
• Outdated facts: Check publication dates. A 2020 statistic may be wrong in 2026.

When to Use

• Verifying claims before including them in code comments or documentation
• Checking technical claims ("X is faster than Y", "Z is deprecated")
• Validating statistics or numerical claims from any source
• Fact-checking user assumptions before building on them