Prompt budget — how long instructions degrade

Frontier APIs handle long, complex prompts well. You can give Claude a 5000-word system prompt with detailed rules, formatting requirements, and few-shot examples, and Claude will track most of it through to the final answer. Open-weight models, even the larger ones, handle this differently. Long prompts degrade in predictable ways.

What "degradation" looks like

Three patterns recur across the open-weight family:

Pattern 1 — last-instruction bias. When the prompt has many instructions, smaller open-weight models tend to weight the most recently mentioned instruction more heavily than the earliest one. If you list 10 rules in order and rule 1 is "always reply in 3 sentences", but rule 10 is "include an explanation", the model often follows rule 10 and produces 6 sentences. Frontier APIs are less prone to this — they treat the rule list more equally.

Pattern 2 — system-prompt fade. Open-weight models can lose track of system-prompt rules when the user message gets long. A 200-word user message with detailed input data may push the system prompt out of effective attention. Mitigation: repeat key rules in the user message ("Remember: respond only with JSON, no markdown."), or move the rules into the user message entirely.

Pattern 3 — format drift over conversation. In multi-turn chats, open-weight models drift away from the format you locked in turn 1. Frontier APIs hold the format better. Mitigation: re-state the format in your system prompt or in the latest user message every few turns.

Token efficiency matters more

On frontier APIs, you pay per token but you can afford to be verbose because the prompt-following is reliable. On open-weight models, every extra paragraph of instructions has a cost and a benefit, and the benefit drops faster.

The practical move on a small model is to cut the prompt down and measure what breaks. Strip the polite framing. Strip the redundant restatements. Keep the format example, drop the format explanation. The model needs less prose; it needs tighter signal.

Do not take a fixed shrink ratio from anyone, including us — how far you can cut depends on the model, the task, and how much of your prompt was doing real work. Cut it in half, run your eval set, and put back only what measurably recovers accuracy. That loop takes an afternoon and gives you a number for your prompt, which is worth more than a rule of thumb for a model you are not running.

The larger open-weight models close most of this gap and will run something close to your frontier prompt verbatim. The small ones will not. Where the line falls has moved with every release, so treat it as something to test rather than something to look up.

Where Hagar's CTO question lands

If the team's per-prompt cost is the binding constraint, the right move is not "switch from Claude to GPT-4o-mini". The right move is "for the high-volume task, test a self-hosted open-weight model with a tightened prompt, and route to it only what it passes".

The shape of the savings at scale: assume 10 million requests per month at 1500 input + 500 output tokens each. On a hosted frontier model that is a per-token bill that scales linearly with volume. Self-hosted, it is a GPU lease that is nearly flat with volume — you pay for the instance whether it runs at 10% or 90% utilisation. That difference in shape, not any particular pair of numbers, is why high volume is the condition that flips the decision. Below some crossover point the hosted API is cheaper; above it, the lease is. Compute the crossover with today's prices and today's instance rates when you make the call.

Two costs that get left out of that comparison and should not be: the engineering time to migrate and maintain the serving stack, and the utilisation you actually achieve — an idle GPU bills the same as a busy one.

The risk is that 5-10% of those requests need the frontier model anyway. Build the fallback into your routing layer from day one. The next lesson covers when few-shot saves you, which is the technique that lets a smaller model handle a task it would otherwise fail.

Next: when few-shot examples save you on a smaller model. :::