Meta-Harness: Optimizing the System Around the AI Model.
Two systems can run the exact same weights and behave very differently, because one has a better way of deciding what the model sees. Meta-Harness turns that surrounding system into something an agent can search and rewrite.
Executive Summary is a complete, self-contained read of the argument. Technical Detail keeps everything above and adds architecture, full benchmark tables and deeper analysis throughout the same article.
- An LLM's behavior isn't just its weights; it's also the harness around it: the code that decides what gets stored, retrieved and shown to the model. Meta-Harness treats that harness as something an agent can search and rewrite, while the model itself stays frozen.
- Its search process leans on rich evidence: the full history of what happened in every prior attempt, not just a pass/fail score. That richer feedback is a big part of why the search finds improvements hand-tuning misses.
- On text classification it beats a strong hand-built baseline by 7.7 points using roughly 4x fewer context tokens. On math reasoning it lifts five held-out models by 4.7 points on average. On TerminalBench-2 it beats the hand-engineered baselines it started from.
- It's not magic and it's not free: evaluation quality, search cost, overfitting to the benchmark, and reproducibility are real open questions, covered honestly further down, not as a footnote.
Same model, different behavior
Take the same frozen model (same weights, same checkpoint) and drop it into two different applications. One retrieves the right examples before every answer and tracks what it already tried; the other retrieves the wrong ones and starts every step from a blank slate.
Take the same frozen model (same weights, same checkpoint) and drop it into two different applications. One retrieves the right five examples before every answer. The other retrieves the wrong twelve. One tracks what it already tried and stops repeating mistakes. The other starts every step from a blank slate. One formats a failing test result so the model can actually see what broke. The other truncates it mid-stack-trace.
Nothing about the model changed. What changed is the code around it: what gets stored, retrieved, and shown to the model the moment it has to act. That's the layer engineers already spend real effort tuning by hand, and it moves real numbers.
Nothing about the model changed. What changed is the code around it: what gets stored, what gets pulled back out, and how it gets presented at the moment the model has to act. Engineers already spend enormous effort tuning that layer by hand: better retrieval, smarter memory, cleaner context windows, tighter tool definitions. It's real work, and it moves real numbers.
Which raises an obvious question. If the model can be optimized, and the prompt can be optimized, why not optimize the thing that decides what the model even sees?
The model didn't get smarter. The system around it got better at deciding what the model gets to see.
That's the premise behind Meta-Harness: End-to-End Optimization of Model Harnesses, a March 2026 paper from Stanford's IRIS Lab (Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab and Chelsea Finn), accepted to COLM 2026. It doesn't touch model weights at all. It searches over the harness: the code that decides what to store, retrieve and present.
What is a model harness?
In simple terms: a model harness is everything around an AI model that decides what it gets to see and do, separate from the model's own weights. Think of the model as an extremely capable employee, and the harness as the workplace they operate in: the instructions they're given, the information handed to them, the tools on their desk, the workflow they follow, and the checks on their work before it ships. Swap out the workplace and the same employee can perform very differently, without the employee changing at all.
Model vs. harness
Model vs. harness
MODEL
- generates predictions
- fixed weights, never touched
HARNESS
- controls what the model sees
- this is what Meta-Harness searches
Technical detailPrecisely what counts as "harness code," and where the boundary sits
The employee analogy is useful but not exact, and it's worth being precise about where it stops. In this paper, "harness" means everything sitting between a task and the model's weights: what information gets written to memory, what gets pulled back out and when, how it's formatted before the model sees it, how tools are defined, and how the interaction is structured turn to turn. It's not the system prompt: a harness is a full stateful program, and the prompt is just one of the things it can generate.
The important framing shift: the harness isn't a preprocessing step that happens before the "real" model call. It's the layer that governs the entire interaction: what the model is even allowed to know at each point in a task. Two harnesses wrapped around the identical model can produce very different task performance, because they're effectively handing the model different problems to solve.
Why hand-tuning breaks down
That hand-tuning is real work, but it doesn't scale: harnesses are still designed almost entirely by hand, one engineer inspecting failure cases and adjusting a heuristic at a time, bounded by how much time a person can spend reading logs. Automating that process isn't new, either, but existing tools compress feedback down to a single score or short note, losing exactly the detail needed to diagnose why a harness failed. That's the gap Meta-Harness closes.
The previous section described that hand-tuning as "real work that moves real numbers," and it is, but it doesn't scale the way you'd want. Right now, harnesses are still designed almost entirely by hand: an engineer inspects failure cases, adjusts a retrieval heuristic or a memory-eviction rule, re-runs the eval, and repeats, across a handful of candidates, bounded by how much time a person can spend reading logs.
Automating that process isn't a new idea, either, but the existing tools for it aren't a good fit here. The paper's authors note that methods like OPRO, TextGrad, MIPRO, AlphaEvolve and GEPA all compress feedback aggressively, down to a scalar score, a short critique, or a fixed window of recent attempts. That works for optimizing a prompt, where the only thing changing is a block of text and a single number can reasonably summarize whether the new version helped. A harness has many more moving parts: a memory-eviction rule, a retrieval filter and a context-formatting choice can each fail independently, and a scalar score doesn't say which one broke. That's exactly the information you need to diagnose why a harness failed on a specific task, which is the gap Meta-Harness is built to close.
Meta-Harness: the idea
Meta-Harness moves the optimization target one layer outward. Rather than changing the model itself, it explores whether the code surrounding the model, the system responsible for context, memory, retrieval, tools and interaction, can also be improved through systematic evaluation.
In practice: try a version, see how it does, learn from what happened, try a better version. Instead of keeping just a pass or fail, the process keeps a fuller record of each attempt, sometimes called an execution trace, so the next round can target a real problem instead of guessing.
In practice, that means treating the harness the way you'd treat any other thing worth optimizing: try a version, see how it does, learn from what happened, try a better version. The part that makes this work is what "see how it does" means. Instead of keeping just a pass or fail, the process keeps a fuller record of each attempt: what the harness showed the model, and roughly where things went right or wrong. That richer record, sometimes called an execution trace, is what lets the next round target a real problem instead of guessing.
Two things distinguish this from prompt-tweaking. It acts on the harness's actual code (the logic governing storage, retrieval and interaction), not a single block of text. And it learns from a fuller account of what happened each round, not a compressed hint about how it went. Model weights stay untouched throughout.
Technical detailWhat an execution trace actually contains, and why it's stored on a filesystem
"Evidence" in the loop above isn't a single number. An execution trace is the fuller record of one attempt: what the harness wrote to memory or retrieved, how it formatted that information for the model, and where the run went right or wrong relative to the eval task. Keeping that record intact, rather than compressing it into a score, is what lets the next revision target a specific failure instead of guessing at one.
Practically, the paper's authors give the search process a filesystem interface to store and reread these traces across rounds. Against two representative program-search baselines given the same proposer and compute budget (OpenEvolve and TTT-Discover), Meta-Harness matched their final accuracy using roughly 10x fewer evaluations, a gap the authors attribute to those methods discarding execution traces that Meta-Harness keeps. Separately, none of the discovered harnesses in the paper required extra LLM calls beyond the one that actually solves the task, so the technique's gains are not simply bought with more inference.
Terminology · optional
Two meanings now share the name
One word, two different products. Open if you
want the distinction.
The term branched; it is not simply being misused, and the two branches solve different problems. An optimization meta-harness is the subject of this article: it searches over the code or configuration of a lower-level harness using evaluation traces and scores. A fleet meta-harness is something else: a common control, policy, state and execution plane sitting across several underlying harnesses. Omnigent, which describes itself as an open-source meta-harness over Claude Code, Codex, Cursor and custom agents, is the second kind; its managed Databricks edition was documented as beta as of July 17, 2026.
Optimization meta-harness
Fleet meta-harness
Saying which one you mean costs a word and saves an argument. The two can also be combined, and the interesting design constraint when they are is back-pressure: a fleet layer can host an optimizer, provided safety policy, secrets and the final grader stay outside its write access.
What the experiments show
The paper reports results across three settings: online text classification, retrieval-augmented math reasoning, and agentic coding. The claim isn't that automated harness search beats hand-engineering everywhere, by a lot, forever. It's narrower and more useful than that: given rich enough access to prior experience, automated search over harness code can find task-specific improvements that hand-tuning missed.
- Text classification: a discovered harness beat a strong hand-engineered baseline (ACE) by 7.7 points of average accuracy while using roughly 4x fewer context tokens.
- Math reasoning: a single discovered retrieval harness lifted accuracy by 4.7 points on average across five models that never saw the search process, the paper's clearest transfer result.
- Agentic coding (TerminalBench-2): the discovered harness surpassed the hand-engineered baselines it started from, though it did not top the overall public leaderboard.
Here's what was actually measured, benchmark by benchmark.
Online text classification. Following the setup used by ACE (Agentic Context Engineering), a model receives labeled examples one at a time, updates its own memory, and gets evaluated on a held-out test set, across LawBench (215 classes), Symptom2Disease (22 classes) and USPTO-50k (180 classes), using GPT-OSS-120B.
| Harness | USPTO | Symptom2Disease | LawBench | Avg. accuracy | Context used |
|---|---|---|---|---|---|
| Zero-shot | 12.0 | 63.2 | 7.0 | 27.4 | 0K |
| Few-shot (best N) | 15.0 | 78.3 | 29.0 | 40.8 | 49.3K |
| ACE (hand-engineered) | 16.0 | 77.8 | 29.0 | 40.9 | 203.0K |
| MCE | 14.0 | 83.0 | 23.0 | 40.0 | 114.0K |
| Meta-Harness | 14.0 | 86.8 | 45.0 | 48.6 | 45.5K |
The discovered harness (the paper calls it Label-Primed Query) beat ACE's 40.9% average by 7.7 points while using roughly 4x fewer context tokens (45.5K vs. 203K). The gains concentrate exactly where you'd expect automated search to earn its keep: tasks with large, confusable label spaces, where LawBench improved by 16 points and Symptom2Disease by 9.
Retrieval-augmented math reasoning. Here Meta-Harness searches over the retrieval program itself (filtering, branching and formatting logic over a corpus of 500,000+ problems) rather than the classification memory system. A single harness is evolved on a 250-problem search set, then evaluated on 200 held-out IMO-level problems, including on five models that never saw the search process at all.
| Method | GPT-5.4n | GPT-5.4m | Gemini-3.1FL | Gemini-3F | GPT-20B | Average |
|---|---|---|---|---|---|---|
| No retriever | 23.0 | 28.8 | 28.6 | 42.6 | 47.6 | 34.1 |
| BM25 retrieval | 30.2 | 29.2 | 32.8 | 46.6 | 48.9 | 37.5 |
| Meta-Harness | 31.7 | 30.4 | 34.9 | 46.3 | 50.6 | 38.8 |
One retrieval harness, discovered by searching against a subset of models, improved accuracy by 4.7 points on average (34.1% → 38.8%) across five models it never saw during search. That's the closest thing in the paper to a direct transfer test, and the result the authors lean on hardest.
TerminalBench: harness evolution in practice
The most tangible demonstration is on agentic coding. TerminalBench-2 evaluates fully autonomous coding agents on 89 Dockerized tasks with binary pass/fail grading. Meta-Harness evolves the entire coding harness, starting from two established open baselines, and the resulting harness surpasses both hand-engineered starting points, though it ranks second overall rather than first on the public leaderboard.
TerminalBench-2's 89 Dockerized tasks span code translation, distributed ML setup, systems programming, bioinformatics and cryptanalysis, with binary pass/fail grading across 5 trials per task. Here Meta-Harness evolves the entire coding harness: system prompts, tool definitions, completion-checking logic, and context management, starting from two established open baselines, Terminus 2 and Terminus-KIRA.
| Rank | Agent (Claude Opus 4.6) | Pass rate |
|---|---|---|
| 1 | ForgeCode | 81.8% |
| 2 | Meta-Harness | 76.4% |
| 3 | Capy | 75.3 |
| 4 | Terminus-KIRA its own starting point | 74.7 |
| 5 | MAYA-V2 | 72.1 |
| 6 | TongAgents | 71.9 |
| 7 | Factory Droid | 69.9 |
Meta-Harness ranks #2 among Opus 4.6 agents; it does not top the leaderboard, ForgeCode does. The paper's claim is narrower: it surpasses the hand-engineered baselines it started from (Terminus-KIRA's 74.7% and Terminus 2's 62.9%), not every agent on the board. On Claude Haiku 4.5, separately, Meta-Harness does rank #1 at 37.6%, ahead of Goose's 35.5%.
The project page also publishes a smaller, more legible run on a harder 19-task subset, chosen because most agents struggle on it, so harness-driven gains stand out clearly. Starting from Terminus-KIRA's 28.5%, repeated rounds of evidence-driven revision take the pass rate to 46.5%, purely through harness changes, with no change to the underlying model. The authors are explicit that this isn't the run behind their final reported 76.4% figure; it's a smaller, easier-to-follow demonstration of the same process.
Technical detailEvaluation design across all three benchmarks
Text classification uses three held-out test sets with markedly different label spaces: LawBench (215 classes), USPTO-50k (180 classes) and Symptom2Disease (22 classes), all run on GPT-OSS-120B, comparing against zero-shot, best-of-N few-shot, ACE and MCE baselines.
Math reasoning searches over retrieval logic against a corpus of 500,000+ problems, evolving one harness on a 250-problem search set, then evaluating pass@1 (averaged over 3 samples) on 200 held-out IMO-level problems across five models that never participated in the search: GPT-5.4n, GPT-5.4m, Gemini-3.1FL, Gemini-3F and GPT-20B.
Agentic coding uses TerminalBench-2's full 89-task suite, graded pass/fail across 5 trials per task on Claude Opus 4.6, plus a separate 19-task harder subset used only to make the improvement curve easier to follow, and a secondary run on Claude Haiku 4.5.
One interpretive point worth flagging: 5 trials per task on an 89-task binary pass/fail benchmark is a modest sample for each individual task, even if it's reasonable in aggregate across the full suite. That's part of why the article treats a 1.4-point gap between second and third place on the TerminalBench-2 leaderboard as close, rather than reading small rank differences on that board as decisive.
The benchmark had to make room for the harness
The clearest sign that this layer now counts is what happened to evaluation. Benchmarks that used to report one number have started separating raw model performance from restricted and open harness tracks. The exact winners will keep changing. The structure is the signal: once a benchmark opens a separate lane for the harness, it has conceded that the harness is part of what is being measured.
| Track | What is allowed | What the score answers |
|---|---|---|
| Raw model | One response, without external tools or scaffolding. | What can the model do in isolation? |
| Restricted harness | Tools and scaffolds within a bounded evaluation contract. | What can a controlled model-plus-system do? |
| Open harness | Tools, execution and solver code are allowed. | What can the complete AI program achieve? |
A practical data-analytics example
Conceptual illustration, not a result from the paper — Meta-Harness wasn't tested on this scenario; it just makes the idea concrete. Ask a data-analysis agent "Why did revenue fall in Q2?" A basic system generates SQL, runs it, and returns whatever comes back. A harness-driven version of the same task, same model, looks very different:
Basic system
Question → LLM → SQL → answer. Whatever the query returns is treated as the explanation, correct or not.
Harness-driven system
Same model. Planning, SQL validation, result inspection, error recovery and evidence-checking now sit around it before it's allowed to answer.
The model hasn't changed between the two panels. The harness has. Meta-Harness asks whether a search process (evaluating candidate harnesses, reading their execution traces, proposing edits) could discover a better version of that surrounding workflow automatically, the same way the paper's authors discovered better retrieval and memory harnesses for classification and math tasks. Whether that specific discovery process transfers cleanly to a data-analytics agent is an open question, not a claim the paper makes.
What this means for AI engineering
Traditionally, harness design is bounded by how many candidates a person can hand-build and inspect. A Meta-Harness-style workflow reorders that: evaluate a baseline, collect execution traces automatically, let an agent propose a change, keep or reject it, repeat, bounded instead by compute and evaluation budget.
None of this is a guaranteed production outcome. What's demonstrated is that, on the tasks tested, richer access to prior experience let automated search find improvements a fixed-budget human process didn't reach in the same time.
Framed as potential benefits deliberately: none of this is a guaranteed production outcome, and the paper doesn't claim it is. What's demonstrated is that, on the tasks tested, richer access to prior experience let automated search find improvements a fixed-budget human process didn't reach in the same time.
Technical detail · interpretationWhat adopting a workflow like this actually requires operationally
This section is engineering interpretation drawn from the risks described later in the article, not a deployment recipe from the paper itself. Running a Meta-Harness-style loop against a real system implies infrastructure most teams don't already have for prompt tuning: a way to capture full execution traces from every candidate run, not just a final score; enough evaluation budget to run dozens of candidates through the full task set repeatedly, since the search's value comes precisely from trying more variations than a person would; and, per the safety and control risk noted below, an explicit boundary on what part of the harness the optimizer is allowed to touch unsupervised versus what stays under human review.
Put differently, the operational cost of harness search doesn't disappear, it moves. Instead of engineering time spent hand-designing a harness, the cost shows up as evaluation infrastructure and compute spent running and re-running candidates, plus the governance work of deciding what an automated process is trusted to change in a production system.
Where the approach can fail
None of this makes harness search a free upgrade. Here's where it can go wrong in practice, as engineering considerations rather than an attack on the research.
The strongest of these is probably evaluation quality: every other failure mode is downstream of it. An optimizer with a filesystem full of rich execution traces is still only as good as the score it's chasing. Rich feedback tells you why something failed; it doesn't tell you whether the thing you're measuring is the thing that actually matters.
Not the same as retraining the model
Worth stating plainly, because "AI optimizes AI" headlines invite the wrong mental model: Meta-Harness does not retrain or fine-tune the LLM's weights. The model checkpoint used to run a candidate harness is identical to the one used to run every other candidate. The entire search happens one layer up.
Meta-Harness vs. prompt optimization
A prompt optimizer rewrites text. Meta-Harness rewrites the program that decides what text the model ever sees, which is a broader surface, with a correspondingly larger space of things that can go right or wrong. The two aren't mutually exclusive; a production system could reasonably layer both.
These approaches operate at different altitudes, from hand-written prompts up through full harness code:
| Approach | What gets optimized |
|---|---|
| Manual prompt engineering | A prompt, by hand |
| Prompt optimization e.g. OPRO, MIPRO | Prompt or instruction text, automatically |
| Context optimization | What gets retrieved and presented as context |
| Agent workflow engineering | The sequence of steps an agent follows |
| Meta-Harness | The harness code itself, the system that governs all of the above at once |
Technical detail · interpretationWhy code-level search is a harder verification problem than text-level search
This is engineering interpretation, not a claim from the paper. A prompt optimizer edits a string; checking whether the new string is well-formed is trivial, and the space of "things that could go wrong" is limited to what the string says. A harness is a stateful program: editing it can change what gets written to memory, what a retrieval filter returns, how a tool call gets constructed, or when the interaction loop terminates. Each of those is its own way for a change to be syntactically fine but behaviorally wrong, which is exactly why the paper's authors lean so heavily on full execution traces rather than a pass/fail score: a scalar can tell you a candidate got worse, but not which of several moving parts caused it.
That also explains why several of the failure modes described above cluster together. Evaluation quality, search cost, and reproducibility aren't independent risks; they're downstream of the same fact, that the object being searched has a much larger and more structured space of possible states than a prompt does, so the process needs correspondingly richer feedback and correspondingly more evaluation runs to search it safely.
The bigger shift
Zoom out and there's a pattern to which question AI engineering has spent its attention on, year over year. First: which model should we use? Then: how should we prompt it? Then: how should we give it tools and context? Meta-Harness is an argument for the next question in that sequence: can the system that controls all of the above be optimized automatically, instead of by hand?
That's a genuinely different kind of question from "is this a good prompt." It's a claim about where the leverage sits in a deployed LLM system, and the paper's results (a real accuracy gain at a fraction of the context cost, on tasks the search process never saw during development) are early evidence for it, not proof that every harness should now be discovered rather than designed. Whether it holds up outside the three settings tested here is exactly the kind of thing worth watching for in the next round of research that cites this paper.
The takeaway
The system around the model becomes the product
The urgent shift is not this month’s leaderboard. It is the move from hand-authored AI behaviour to software that can systematically improve how intelligence is elicited, tested and governed — and that improves a little on every run rather than whenever an engineer next has an afternoon.
A harness turns a model into a worker. A meta-harness turns every run into evidence for the next system.
FAQ
What is a model harness?
The code surrounding a language model that decides what information gets stored, retrieved and presented to it at each step: memory management, retrieval logic, tool definitions, and interaction structure. It's everything except the model weights themselves.
Does Meta-Harness retrain the model?
No. Model weights stay fixed throughout every experiment in the paper. Meta-Harness searches over the surrounding code, the harness, never the parameters of the LLM itself. See the diagram above.
What's the difference between Meta-Harness and prompt optimization?
Prompt optimizers rewrite text. Meta-Harness rewrites executable code: what gets retrieved, how memory updates, how context gets assembled, how the model interacts with tools. It operates one level higher than the prompt; see the comparison table.
What results does the Meta-Harness paper report?
A 7.7-point accuracy improvement over a hand-engineered baseline on text classification using roughly 4x fewer context tokens, a 4.7-point average accuracy gain on retrieval-augmented math reasoning across five held-out models, and a TerminalBench-2 pass rate that surpasses the two hand-engineered baselines it started from. Full numbers are in the results section.
Who wrote the Meta-Harness paper?
Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab and Chelsea Finn, from Stanford's IRIS Lab, published on arXiv in March 2026 and accepted to the Conference on Language Modeling (COLM) 2026.
Is Meta-Harness open source?
Yes. The reference framework and two paper experiments are published by the Stanford IRIS Lab on GitHub, alongside a separate artifact repository holding the optimized TerminalBench-2 harness. Links are in Sources below.
Sources & further reading
Every arXiv identifier below was checked against the arXiv API and resolves to the title shown. Both repositories are linked from the paper’s own project page.
- PaperLee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O. & Finn, C. Meta-Harness: End-to-End Optimization of Model Harnesses (2026). The primary source for this article.
- Projectyoonholee.com/meta-harness Official project page: interactive demo, full results tables, and the method figure this article’s diagrams are based on.
- Codestanford-iris-lab/meta-harness Reference framework plus two of the paper’s experiments.
- Codestanford-iris-lab/meta-harness-tbench2-artifact The optimized coding harness from the final reported TerminalBench-2 run.
- PaperZhang, Q. et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (2025, rev. 2026). The hand-designed baseline Meta-Harness is measured against on text classification.
- PaperYao, Y. et al. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows (2026). Evidence that the harness, not just the model, moves the score.
- PaperZhang, A. L. et al. Recursive Language Models (2025, rev. 2026). Adjacent line of work on models that call themselves.
- PaperYe, H. et al. Meta Context Engineering via Agentic Skill Evolution (2026). The same outer-loop idea aimed at skills rather than harness code.
- PaperWen, C. et al. HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety (2026). Treats configuration, capability, runtime, persistent state and recovery as distinct safety surfaces — the reason back-pressure matters.
- BenchmarkLongCoT track definitions (2026). A worked example of a benchmark splitting raw-model from harness tracks.
- ProjectOmnigent The fleet sense of the term: one control plane over several coding-agent harnesses. Managed edition documented as beta, 17 July 2026.