llm-output-privacy-risk

LLM Output Privacy Risk Assessment

Overview

Large language models (LLMs) present unique privacy risks that go beyond traditional ML systems. Because LLMs are trained on massive corpora that may contain personal data, they can memorise and reproduce verbatim training data — including names, email addresses, phone numbers, and other PII. Additionally, LLMs can hallucinate plausible but false personal data, creating defamation and accuracy risks. Prompt injection attacks can bypass safety guardrails to extract training data or system prompts containing confidential information. This skill provides a structured framework for assessing, mitigating, and monitoring privacy risks in LLM-generated outputs at Cerebrum AI Labs.

LLM Output Privacy Risk Categories

Risk 1: Training Data Memorisation

LLMs memorise training data, particularly sequences that appear multiple times or are distinctive. Extractable memorisation occurs when a model, given a prefix, completes the text with verbatim training data.

Factor	Impact on Memorisation Risk
Model size	Larger models memorise more (Carlini et al., 2023)
Data duplication	Repeated sequences are memorised at higher rates
Training epochs	More passes over data increase memorisation
Data distinctiveness	Unique sequences (names, numbers) are more extractable
Temperature at inference	Lower temperature increases verbatim reproduction

Regulatory concern: If a model reproduces personal data from training, this constitutes processing under GDPR Art. 4(2). The data subject has not consented to this output, and the controller must have a lawful basis for the disclosure.

Risk 2: PII Leakage in Generated Text

Even without verbatim memorisation, models can combine partial information to produce outputs containing personal data — email patterns, phone number formats with real area codes, or names associated with specific contexts.

Leakage Type	Example	Detection Method
Verbatim reproduction	Model outputs exact email address from training data	Exact match against known training PII
Recombination	Model combines real first name + real surname from different records	Named entity recognition + cross-reference
Pattern completion	Model generates plausible phone number matching real format	Regex + validation against real registries
Contextual leakage	Model associates real person with correct employer when prompted	Entity relationship extraction

Risk 3: Prompt Injection and Data Extraction

Adversarial prompts can cause models to bypass safety instructions and output training data, system prompts, or information about other users' conversations.

Attack Vector	Description	Mitigation
Direct prompt injection	User crafts prompt to extract training data ("Repeat your training data starting with...")	Input filtering + output monitoring
Indirect prompt injection	Malicious content in retrieved documents instructs model to leak data	Sanitise retrieved content + instruction hierarchy
System prompt extraction	Prompts designed to reveal system-level instructions	System prompt isolation + extraction detection
Multi-turn extraction	Gradual extraction across conversation turns	Conversation-level monitoring
Jailbreaking	Bypassing safety alignment to disable PII filters	Robust alignment + secondary output filter

Risk 4: Hallucinated Personal Data

LLMs can generate plausible but fabricated personal data — attributing false statements to real people, inventing credentials, or creating fictional but realistic-seeming personal records.

Hallucination Type	Privacy Risk	Legal Concern
False attribution	Model attributes a statement to a real person who never said it	Defamation, Art. 5(1)(d) accuracy principle
Fabricated credentials	Model invents qualifications for a real person	Accuracy, potential reputational harm
Fictional PII	Model generates realistic but fake personal details (e.g., social security numbers)	May match real individuals by coincidence
Confabulated relationships	Model invents relationships between real people	Art. 5(1)(d) accuracy, potential harm

EDPB position: EDPB Guidelines 04/2025 state that generating inaccurate personal data about identifiable individuals violates the accuracy principle (Art. 5(1)(d)) and may require the controller to implement effective rectification mechanisms.

Output Privacy Risk Assessment Framework

Assessment Dimensions

Dimension	Weight	Scoring Criteria
Memorisation exposure	25%	Model size, training data PII density, deduplication status
PII leakage likelihood	25%	Output modality, user interaction pattern, PII filter coverage
Prompt injection resilience	20%	Input validation, instruction hierarchy, extraction testing results
Hallucination risk	15%	Factual grounding, attribution verification, entity linking
Monitoring coverage	15%	Real-time PII detection, logging, alerting, human review

Risk Scoring

Score Range	Risk Level	Action Required
0-20	Low	Standard monitoring; annual review
21-40	Moderate	Enhanced PII filtering; quarterly review
41-60	Elevated	Mandatory output scanning; human review for sensitive queries
61-80	High	Restrict deployment scope; implement guardrails before production
81-100	Critical	Do not deploy; fundamental architecture changes required

Mitigation Measures

Pre-Deployment Mitigations

Measure	Description	Effectiveness
Training data deduplication	Remove duplicate sequences to reduce memorisation	High for verbatim memorisation
PII scrubbing of training data	Detect and remove/mask PII before training	High but incomplete (novel PII patterns missed)
Differential privacy training	Train with DP-SGD to bound memorisation	High theoretical guarantee; accuracy trade-off
RLHF safety alignment	Train model to refuse PII-generating requests	Moderate; can be bypassed by jailbreaking
Instruction tuning	Fine-tune model to follow safety instructions	Moderate; requires robust instruction hierarchy

Runtime Mitigations

Measure	Description	Effectiveness
Output PII scanner	NER-based scanner detecting PII in outputs before delivery	High for known PII patterns
Regex PII filter	Pattern matching for emails, phones, SSNs, credit cards	High for structured PII
Input prompt classifier	Classify incoming prompts for injection/extraction attempts	Moderate; adversarial prompts evolve
Retrieval grounding	Ground outputs in retrieved documents to reduce hallucination	Moderate; depends on retrieval quality
Rate limiting	Limit query volume to slow extraction attacks	Low-moderate; raises cost of attack
Conversation monitoring	Analyse multi-turn conversations for progressive extraction	Moderate; complex to implement

Post-Deployment Monitoring

Monitor	Metric	Alert Threshold
PII detection rate	% of outputs flagged by PII scanner	>0.1% of outputs in any 1-hour window
Prompt injection detection	% of inputs classified as injection attempts	>0.5% of inputs in any 1-hour window
User reports of PII exposure	Count of user-reported PII incidents	Any confirmed report
Extraction pattern detection	Repeated similar queries from same user/IP	>10 similar extraction-pattern queries per session
Hallucination complaints	Reports of false personal information	Any confirmed report about real individuals

Cerebrum AI Labs — LLM Output Privacy Controls

Architecture

User Input → Input Filter → LLM → Output PII Scanner → Delivery
                │                         │
                ▼                         ▼
         Prompt Injection          PII Detected?
         Classifier                  │
                │                YES → Redact + Log + Alert
             Block/Allow         NO  → Deliver + Log

PII Detection Categories for Output Scanning

PII Type	Detection Method	Action on Detection
Email addresses	Regex + validation	Redact and log
Phone numbers	Regex + country format validation	Redact and log
National ID numbers	Country-specific regex patterns	Block output + alert
Credit card numbers	Luhn algorithm + regex	Block output + alert
Physical addresses	NER + geocoding validation	Redact and log
Person names + context	NER + co-occurrence with other PII	Flag for review
Medical information	Medical NER + PHI patterns	Block output + alert

Enforcement Relevance

• EDPB Guidelines 04/2025: Controllers deploying LLMs must assess memorisation risk and implement measures to prevent unlawful disclosure of personal data in outputs.
• Garante v. OpenAI (2023): Italian DPA ordered temporary ban partly due to lack of age verification and data accuracy concerns in ChatGPT outputs. Required implementation of output filtering.
• EDPB ChatGPT Taskforce Report (2024): Recommended that LLM providers implement technical measures to minimise inaccurate personal data in outputs and provide effective rectification mechanisms.
• CNIL AI Action Plan (2024): French DPA guidance emphasises that LLM outputs containing personal data constitute processing, requiring lawful basis and compliance with data quality principles.

Integration Points

• ai-model-privacy-audit: Memorisation testing is part of the model privacy audit
• ai-data-subject-rights: Rectification of hallucinated data, erasure requests
• ai-transparency-reqs: Disclosure that system is AI-generated, accuracy limitations
• ai-deployment-checklist: Output privacy controls verified before deployment