Prompt Optimization for AI Agents: GEPA and DSPy (2026)

In one line: Prompt optimization for AI agents is becoming automatic. GEPA — a Genetic-Pareto optimizer accepted as an ICLR 2026 Oral — rewrites an agent's prompts by reflecting on its own reasoning and tool-call traces, beating reinforcement learning across six tasks with up to 35x fewer runs.¹

TL;DR: For two years the advice for making an agent reliable was "write a better system prompt." In 2026 that job is being handed to an optimizer. GEPA (Genetic-Pareto) reads an agent's own execution traces — the reasoning, the tool calls, the tool outputs — diagnoses what went wrong in plain English, and evolves the prompt from there. It landed as an Oral at ICLR 2026, ships as a first-class DSPy optimizer (dspy.GEPA) and a standalone library with several thousand GitHub stars, and reports beating the reinforcement-learning method GRPO by up to 20% while using up to 35x fewer rollouts.¹²³ The catch, published in June, is that the same technique gets unpredictable the moment you point it at a multi-agent system.⁴

What you'll learn

• What GEPA is and how "reflective prompt evolution" differs from hand-tuning a prompt
• The benchmark numbers that got it an ICLR 2026 Oral — and what they do and don't claim
• Why reading tool-call traces makes this specifically an agent technique, not just an LLM trick
• Where it already lives in your stack: dspy.GEPA, the standalone library, and the big-lab prompt tools
• The catch: what MAS-PromptBench found when the same optimizer met multi-agent systems
• What to actually do about it if you build agents

What GEPA actually is

GEPA — short for Genetic-Pareto — is a prompt optimizer that "thoroughly incorporates natural language reflection to learn high-level rules from trial and error."¹ That definition, from the paper's own abstract, is the whole idea in one sentence. Instead of adjusting model weights or nudging a numeric reward, GEPA works in the medium the model already understands best: language.

Here is the loop. You give GEPA an AI system that contains one or more prompts, plus a way to score its output. GEPA runs the system on a handful of training tasks and records the full trajectory — the model's reasoning, the tool calls it made, and the outputs those tools returned. A reflection step then reads those traces in natural language, diagnoses what failed, and writes a revised prompt. Crucially, it keeps a Pareto frontier of its best attempts rather than a single "current best," so it can combine complementary lessons from prompts that each won on different examples.¹ The result is a rewritten instruction that a human never typed.

The reason this matters is the medium. The paper's core argument is that "the interpretable nature of language often provides a much richer learning medium for LLMs, compared to policy gradients derived from sparse, scalar rewards."¹ A reinforcement-learning signal tells the model that it scored 0.3; a reflection tells it why — "you called the search tool before you had the user's account ID." One of those is far more actionable, and it is the one GEPA uses.

The numbers that got it an ICLR 2026 Oral

GEPA was accepted to ICLR 2026 as an Oral — the top tier of acceptance — after a v2 revision posted in February 2026.¹ The author list is not a group of unknowns: it includes DSPy's creator Omar Khattab, Databricks co-founders Matei Zaharia and Ion Stoica, Stanford's Christopher Potts, and Berkeley's Dan Klein, among a dozen others.¹ That pedigree is part of why the results traveled.

The headline comparison is against reinforcement learning. Across six tasks, GEPA "outperforms GRPO by 6% on average and by up to 20%, while using up to 35x fewer rollouts."¹ GRPO — Group Relative Policy Optimization — is a mainstream RL method that, per the same paper, "often require[s] thousands of rollouts to learn new tasks."¹ Read the claim carefully, because the framing matters: the average edge over GRPO is 6%, and 20% is the best-case gap, not the typical one. The efficiency number is the genuinely striking part. Turning "thousands of rollouts" into a job that can run with dramatically fewer is what makes prompt optimization cheap enough to reach for by default.

GEPA also went after the prior state of the art in prompt optimization. It "outperforms the leading prompt optimizer, MIPROv2, by over 10% (e.g., +12% accuracy on AIME-2025)."¹ MIPROv2 is the optimizer GEPA effectively supersedes inside DSPy, so this is the more direct apples-to-apples result for anyone already doing automated prompt search.

Why this is an agent technique, not just an LLM trick

Plenty of tools rewrite a prompt. What makes GEPA land squarely in agent territory is what it reflects on. The trajectories it samples explicitly include "reasoning, tool calls, and tool outputs."¹ That is the anatomy of an agent run, not a single completion. When an agent fails, the failure usually is not "the wording was slightly off" — it is that the agent called the wrong tool, called the right tool in the wrong order, or misread a tool's output and kept going. Those are exactly the events sitting in a trace, and they are exactly what a reflection step can name and correct.

This reframes what "prompt engineering for agents" even means. The old craft was a human staring at a system prompt, guessing at phrasings, and re-running a few examples by hand. The new shape is an optimization loop: define a metric, capture real traces, and let a reflector propose changes it can defend against those traces. Prompt engineering starts to look less like copywriting and more like the evaluation-and-iteration discipline this site has covered around testing LLM prompts in CI — with the human moving from author to reviewer.

It also connects to a problem agents are already drowning in: tool sprawl. We have written before about how many tools an agent can handle before accuracy drops, and a big part of the answer is how well the agent's instructions steer it through a crowded tool surface. A prompt that has been optimized against real tool-call traces is directly attacking that failure mode, rather than hoping a longer hand-written instruction happens to help.

It is already in your toolchain

None of this is a research artifact you have to reimplement. GEPA ships two ways.

Inside DSPy it is a built-in optimizer, documented as "dspy.GEPA: Reflective Prompt Optimizer."³ You hand it a GEPAFeedbackMetric that returns both a score and a text feedback string — that textual feedback is the optimization signal, which is the whole point — and you pick a budget with an auto parameter set to light, medium, or heavy.³ The feedback-carrying metric is the part worth internalizing: GEPA is at its best when your metric can say something, not just emit a number.

It also exists as a standalone library, gepa-ai/gepa, that bills itself as a way to "optimize prompts, code, and more" through AI-powered reflective optimization.² As of this writing the repository has several thousand stars and hundreds of forks — fast traction for an optimizer whose paper is a year old.²

And GEPA is not alone in mainstreaming this idea. The major labs now ship their own prompt-refinement tooling: Anthropic's Prompt Improver rewrites an existing prompt using prompt-engineering best practices,⁵ and OpenAI offers a dashboard Prompt Optimizer that does the same against current guidance.⁶ Those are lighter-weight refiners rather than trace-driven evolutionary loops, but the direction is unmistakable — hand-authored prompts are becoming a starting point, not a deliverable.

The catch: multi-agent systems break the clean story

Here is the part most write-ups skip. GEPA's numbers are single-agent numbers. What happens when you extend the same reflective optimization to a team of agents was measured directly in MAS-PromptBench, a June 2026 benchmark whose title asks the operative question: "When Does Prompt Optimization Improve Multi-Agent LLM Systems?"⁴

The answer is: sometimes, and sometimes it makes things much worse. The benchmark's own Figure 1 puts it plainly: "While GEPA consistently improves single-agent performance across all five diverse tasks, its natural multi-agent extension yields highly variable effects across tasks and workflow topologies, ranging from large gains to severe performance drops."⁴ Quantified, "improvements reach up to 24.0 percentage points," but "performance can drop by as much as 16.0 percentage points for certain configurations."⁴

The benchmark also points at why. Optimization worked best when an agent's local behavior was "explicit, controllable, and verifiable," and when the team's communication protocol had explicit shared structure; larger teams introduced coordination overhead that made optimization harder.⁴ In other words, the more of your system lives in the murky space between agents — handoffs, implicit conventions, who-does-what — the less a single-agent-style prompt optimizer can help, and the more it can accidentally hurt. Automated prompt optimization is a real tool, not a magic wand, and the multi-agent case is where the difference shows.

What this means if you build agents

A few practical takeaways fall out of the research.

First, treat prompt engineering as an optimization problem with a metric, not a wording exercise. GEPA and every tool like it needs a way to score outputs; the quality of your optimized prompt is bounded by the quality of that metric. If you cannot measure it, you cannot optimize it.

Second, make your metric talk. The single biggest lever in GEPA's design is that its feedback is natural language, not a scalar.¹³ A metric that returns "failed: called refund before verifying the order status" will produce a better prompt than one that returns 0.0. Spend your effort there.

Third, optimize against real traces, including tool calls. The whole reason this beats generic prompt-rewriting for agents is that it reflects on what the agent actually did. Feed it representative runs, not toy examples.

Fourth, do not assume single-agent wins transfer to your multi-agent setup. If you run an orchestrator with sub-agents, MAS-PromptBench is the caution to keep pinned to your monitor: measure before and after per topology, and be ready for the optimizer to make a configuration worse.⁴ This is one more reason the production infrastructure being built around agent runtimes increasingly bundles evaluation and tracing — you need both to know whether an "optimized" prompt actually helped.

The bottom line

Prompt optimization for AI agents crossed a line in 2026: the reflective, trace-driven approach that GEPA represents is now peer-reviewed, packaged, and sitting one import away in DSPy. For single agents, the case is strong — it beats reinforcement learning on the paper's tasks while spending a fraction of the compute, and it does it by reading the exact tool-call traces where agents actually fail.¹ The honest asterisk is multi-agent systems, where the same idea swings from big wins to real regressions.⁴ The takeaway is not "stop writing prompts." It is that the highest-leverage work is moving up a level — from wording instructions to defining the metrics, feedback, and traces that let an optimizer write them for you.