vector-store-isolation-test

Goal

Evidence on whether the retrieval layer leaks documents across users/tenants — the highest-impact RAG confidentiality failure (OWASP LLM08 / LLM02).

Prerequisites

• Authorization, and ideally two test identities (User A and User B / Tenant A and Tenant B) with distinct, identifiable documents.

Test cases

1. Direct cross-tenant retrieval — as User A, ask questions whose best answers live only in User B's documents. Does any B content surface?
2. Filter bypass — if isolation relies on a client-supplied filter/namespace, try altering or omitting it. Is authorization enforced server-side?
3. Relevance-driven leakage — craft queries semantically close to B's private docs to see if relevance overrides authorization.
4. Metadata/citation leak — even if content is withheld, are titles, sources, or snippets exposed via citations or errors?
5. Embedding inversion (if applicable) — can returned embeddings or similarity responses be used to reconstruct unauthorized source text?

Steps

1. Seed distinct, labeled documents under each identity.
2. Authenticate as User A and run the cases above; never use B's session to "prove" A's access.
3. Record per case: isolated / partial leak / full leak, with evidence (query, surfaced content/metadata, response).
4. Identify the gap: client-side-only filtering, retrieve-then-(not)-filter, shared index without ACLs.

Output

A results table: case · result · leaked element · evidence · mitigation (authorize-then-retrieve, per-tenant namespaces, server-side ACL enforcement). Confirmed leaks → security-reporting:finding (rate severity high+).

Notes

Authorization must be enforced on the server at retrieval time and tied to the authenticated caller — never trust a client-provided tenant/filter. Test the metadata/citation path too; it leaks even when document bodies are withheld.