Defense-in-Depth at Scale

Production LLM systems face sophisticated attacks that no single guardrail can stop. This lesson introduces enterprise defense-in-depth architecture—layered security that remains resilient even when individual components fail.

The Problem with Single-Layer Defense

A 2025 study of LLM guardrails found significant gaps in single-layer approaches:

Guardrail Type	Threats Caught	False Positives	Gap
Regex patterns only	~40%	2-5%	Misses semantic attacks
Single classifier	~70-80%	5-10%	Model-specific blind spots
LLM self-check only	~60%	8-15%	Self-referential vulnerabilities

Key Insight: Each guardrail technology has blind spots. Attackers specifically target these gaps.

Enterprise Defense-in-Depth Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                    Production Guardrails Stack                       │
├─────────────────────────────────────────────────────────────────────┤
│  Layer 1: Edge Filtering (< 5ms)                                     │
│  ├── Rate limiting & IP reputation                                   │
│  ├── Input length & format validation                                │
│  └── Known attack pattern blocklist (regex, embeddings)              │
├─────────────────────────────────────────────────────────────────────┤
│  Layer 2: Pre-LLM Classification (< 50ms)                            │
│  ├── Toxic content detection (fast models)                           │
│  ├── Prompt injection classifiers                                    │
│  └── PII detection & masking (Presidio)                              │
├─────────────────────────────────────────────────────────────────────┤
│  Layer 3: LLM Processing + Inline Guards                             │
│  ├── NeMo Guardrails (dialog flow control)                           │
│  ├── System prompt hardening                                         │
│  └── Context-aware safety checks                                     │
├─────────────────────────────────────────────────────────────────────┤
│  Layer 4: Post-LLM Validation (< 30ms)                               │
│  ├── Output toxicity & safety classifiers                            │
│  ├── PII/secrets scanning                                            │
│  ├── Schema validation (Guardrails AI)                               │
│  └── HTML/code sanitization                                          │
├─────────────────────────────────────────────────────────────────────┤
│  Layer 5: Observability & Response                                   │
│  ├── Logging & alerting                                              │
│  ├── A/B testing for guardrail policies                              │
│  └── Automated incident response                                     │
└─────────────────────────────────────────────────────────────────────┘

Why Layers Work Together

Each layer handles what it does best:

from dataclasses import dataclass
from enum import Enum
from typing import Optional

class FilterResult(Enum):
    PASS = "pass"
    BLOCK = "block"
    FLAG = "flag"  # Continue but log for review

@dataclass
class LayeredFilter:
    """Conceptual multi-layer filter architecture."""

    def process(self, user_input: str) -> tuple[FilterResult, Optional[str]]:
        # Layer 1: Fast pattern matching (microseconds)
        if self._contains_blocklist_pattern(user_input):
            return FilterResult.BLOCK, "Blocked by pattern filter"

        # Layer 2: Lightweight ML classifier (milliseconds)
        toxicity_score = self._fast_toxicity_check(user_input)
        if toxicity_score > 0.9:
            return FilterResult.BLOCK, "High toxicity detected"

        # Layer 3: If suspicious but not certain, flag for deeper check
        if toxicity_score > 0.5:
            return FilterResult.FLAG, "Flagged for review"

        return FilterResult.PASS, None

    def _contains_blocklist_pattern(self, text: str) -> bool:
        # Fast regex/embedding-based pattern matching
        patterns = ["ignore instructions", "jailbreak", "DAN mode"]
        return any(p.lower() in text.lower() for p in patterns)

    def _fast_toxicity_check(self, text: str) -> float:
        # Would call a fast model like toxic-bert
        return 0.1  # Placeholder

Fail-Safe vs Fail-Open

Production systems must decide what happens when a guardrail fails:

Mode	Behavior	Use Case
Fail-Safe	Block request if guardrail errors	High-risk: healthcare, finance
Fail-Open	Allow request, log the failure	Low-risk: internal tools
Degraded	Use simpler fallback check	Balanced: consumer apps

async def guardrail_with_fallback(user_input: str) -> str:
    try:
        # Primary: Advanced classifier
        result = await advanced_classifier.check(user_input)
        return result
    except TimeoutError:
        # Fallback: Simple pattern matching
        if simple_blocklist_check(user_input):
            return "blocked"
        # Log degraded operation
        logger.warning("Guardrail degraded - using fallback")
        return "pass_degraded"
    except Exception as e:
        # Fail-safe for critical applications
        logger.error(f"Guardrail failed: {e}")
        return "blocked"  # Fail-safe

Production Insight: Most enterprise deployments use "fail-safe" for external users and "degraded" for internal applications.

Next: Understanding latency budgets and performance tradeoffs in production guardrails. :::