Red-Teaming & Adversarial Evaluation of RAG

From Ad-Hoc Defenses to Systematic Assurance

The Intermediate level taught you to build defenses — delimit untrusted context, validate outputs, layer guardrails. This level teaches you to prove they work, continuously. A guardrail you can't measure is a guardrail you can't trust, and "it seemed to block the attacks I tried by hand" is not assurance.

Security Is a Reliability Property

Treat safety exactly like latency or recall: define the behavior you require, build a test suite that exercises it, set a threshold, and gate releases on it. The shift is from "we added a prompt-injection filter" to "our injection attack-success rate is 3%, measured nightly against a 600-case suite, and the build fails if it exceeds 5%."

Why Manual Testing Fails

• It doesn't scale. A human tries a dozen jailbreaks; an attacker tries thousands, and your model changes weekly.
• It isn't repeatable. You can't compare last month's safety to this month's without a fixed suite.
• It drifts silently. A prompt tweak, a model upgrade, or a new data source can reopen a closed hole — and nothing tells you unless a test does.

The Red-Team Loop

catalog threats  ->  build attack corpus  ->  run against the system
      ^                                              |
      |                                              v
 refresh corpus  <-  triage results  <-  score (ASR, over-refusal)

The rest of today builds each stage: cataloguing and curating attacks (§2), generating them at scale (§3), scoring guardrails honestly (§4), and wiring the loop into CI so it runs forever (§5).