The fallback strategy — composing multiple models

A production application that uses one model has a single point of failure. The model can be down. The model can return junk on a specific input shape. The model can be repriced out of your use case. And — the one teams plan for least — the model can be retired on a published schedule while your code keeps calling it. The mature pattern is to compose multiple models with explicit fallback logic.

The failure everyone forgets: the model is going away

Every vendor publishes a deprecation page, and every vendor eventually retires the model you are calling. This is not a rare event and it is not unannounced — it is a dated commitment you can read today:

Vendor	Where the schedule lives
Anthropic	Model deprecations — states model status, deprecation date, and retirement date per model ID
OpenAI	Deprecations — shutdown dates and the recommended replacement
Google	Model versions and lifecycle — retirement dates per model

The three models this course captured are a live example. They were current when the captures were taken in April 2026. Read those pages today before you copy any model ID out of this course into production code: at least one of them has a retirement date inside the next few months, and a request to a retired model does not degrade — it fails.

That is why the routing layer below is worth building even if you only ever use one model. A system with a fallback path already has somewhere to send traffic on the day a retirement lands. A system without one has an outage and a rewrite.

The operational habit is small: put the deprecation page for every model you call on a quarterly calendar reminder, and keep the model ID in configuration rather than inlined at the call site. Both cost minutes. Neither is exciting. They are the difference between a migration and an incident.

The three patterns that work

Pattern 1 — primary + fallback. The simplest. Send the request to your preferred model. If the response is invalid (failed JSON parse, empty body, error response, truncated output), retry on a second model. Optionally, after N failures on the primary, switch all traffic to the fallback for some cooldown period. This is the most common production setup.

async function classifyWithFallback(input: string) {
  let reason: string;
  try {
    const r = await callPrimary(input);
    if (isValid(r)) return { result: r, model: "primary", fallbackReason: null };
    reason = "invalid-response";
  } catch (err) {
    reason = err instanceof Error ? err.name : "unknown-error";
  }
  const r = await callSecondary(input);
  return { result: r, model: "secondary", fallbackReason: reason };
}

Note what that function returns. Not just the answer — also which model produced it and why the fallback fired. A bare catch {} would be shorter, and it would throw away the two fields the "What to log" section below says you cannot operate without. Error handling that discards the error is the most common way a fallback layer becomes undebuggable.

Pattern 2 — cheap-first, escalate. For tasks where most inputs are easy and a few are hard, route everything to a small model first. If the small model's answer fails a check you can actually run, escalate to a larger model. This minimises cost on the easy majority of requests and only pays for the expensive model on the hard remainder.

The whole pattern lives or dies on that check, so it has to be a real signal:

async function summariseEscalating(text: string) {
  const small = await callSmallModel(text);

  const needsEscalation =
    small.stopReason === "max_tokens" ||   // ran out of room — incomplete
    small.refusal ||                        // declined the task
    !passesTaskCheck(small.text);           // your own validity test

  if (needsEscalation) return await callFrontierModel(text);
  return small;
}

An earlier version of this lesson escalated when the small model's output was under 50 tokens. Read that against the task: this function summarises. A short summary is the goal. That condition escalates on success — it pays frontier prices precisely when the cheap model did well, which inverts the economics the pattern exists to create. Length is not a quality signal; it is a length signal. Escalate on a stop reason, an explicit refusal, or a check that encodes what "correct" means for your task.

Pattern 3 — verifier-on-top. Two models on the same input. The first generates an answer. The second verifies the answer against the input. If the verifier rejects, regenerate. This pattern is overkill for tone rewrites but essential for code generation, financial extraction, or anything where wrong is expensive.

How to know which pattern fits

The patterns are stackable, and the tree gives you the starting point rather than the final architecture.

The patterns are stackable. A real production system might use Pattern 2 with a self-hosted open-weight model as the cheap model and Claude Sonnet 4.5 as the escalation, plus Pattern 3 verifier on top of the Claude path for the highest-stakes 1% of requests.

What to log

Every multi-model setup needs three logging fields per request:

1. Which model produced the final answer. Otherwise you cannot debug "why is this user complaining about a weird output?" — the user does not know which model answered.
2. Why fallbacks fired (parse error, empty response, low confidence, manual override). This is your dataset for improving the routing logic over time.
3. Cost per request. Per-token costs vary across models; per-request costs vary even more once you account for fallbacks. Log them so you can prove the savings to your CTO.

This logging is the foundation of the comparison report you will ship in the capstone. Hagar's report will not just say "we should use these models for these tasks". It will show the actual cost-per-request distribution before and after the routing change, the fallback fire rate, and the user-facing quality scores. Without logging, none of that is provable.

Next module: the capstone — port one prompt across 8 models and ship a real comparison report. :::