That loop above is the whole idea in one picture. The rest of this article exists to unpack it, starting with why the loop is needed in the first place.

What is a Skill?

Before we get to SkillOpt, we need one idea straight: a Skill.

A Skill is a document that tells an agent how to do a recurring job: how to close out a finished branch, how to triage a failed deploy, how to format a changelog entry. Think of it as an instruction manual the agent reads right before it acts.

What one looks like

In practice, a Skill is often written down as a document called SKILL.md. Here's a small, made-up example:

---
name: repo-closeout
description: Close out a finished feature branch cleanly
---

## Overview
Use this Skill when a feature branch is merged and ready to retire.

## Rules
- Delete the branch only after CI confirms main is green
- Squash merge commits before deleting
- Date ranges in the changelog are inclusive of the end date

The document itself isn't the point. The content is. What SkillOpt cares about is one question: does this Skill give the agent the right instructions to succeed, consistently, on the cases it will actually meet?

Most Skills are written once by hand and never touched again. That's the problem. A Skill that was right for the first few cases someone imagined quietly stops being right as new edge cases show up, and someone has to notice, guess at a fix, and rewrite it, usually with no way to know if the rewrite actually helped. SkillOpt turns that guesswork into something structured and tested.

The model stays frozen. The Skill gets better.

SkillOpt draws a hard line between what it's allowed to touch and what it isn't.

The model that actually performs the task, the target model, has its weights frozen for the entire process, before, during, and after. Nothing about how it was built or trained ever changes. The Skill is the only thing that changes, and only when a change earns its place.

What does "frozen" actually mean?

It's worth spelling this out, because it's the whole idea. Frozen means the model's internal weights (the numbers that make it behave the way it does) are read-only. They're the same before SkillOpt runs and after. Nothing is retrained.

"Gets better" has one narrow meaning here: a strictly higher score on tasks the Skill wasn't shown while it was being edited. Not a rewrite that reads more confidently. Not phrasing that sounds more thorough. A measured improvement on cases the edit wasn't built around. The rest of this article is about how that measurement actually happens, because that mechanism is what makes "better" mean something, instead of just being a claim.

How SkillOpt works

Two separate models are involved, and they do two different jobs.

The target model, frozen throughout, attempts real tasks using the current Skill. The optimizer model reads how those attempts went and proposes a change to the Skill. Neither model's weights change. Only the Skill does.

The full loop

Put together, one round of SkillOpt looks like this:

The target model runs training tasks with the current Skill, and those attempts get scored. The optimizer reads that scored evidence, not the raw transcripts alone, and proposes a change. That change is bounded: a capped append, insert, replace, or delete to the existing Skill. Never a free rewrite of the whole document. The cap matters for a specific reason: it keeps any single accepted change small enough to read and audit, rather than letting one noisy batch of evidence rewrite the Skill from scratch.

The candidate only replaces the current Skill if it strictly beats it on data it wasn't tuned against. If it doesn't clear that bar, it's rejected, and the rejection itself becomes part of the record the optimizer works from next time, so it isn't likely to propose the same failed change twice.

Why evaluation matters

Here's the trap SkillOpt is built to avoid. It's tempting to "improve" a Skill the lazy way: look at one failure, tweak the wording, test it against that same failure, and declare victory because it now passes.

SkillOpt avoids this by keeping three roles of data strictly separate. None of them is allowed to substitute for another:

• Training tasks generate the scored evidence a candidate edit is proposed from.
• A selection set the candidate wasn't shaped around decides whether it survives. This is the validation gate: the candidate is scored against it, and only replaces the current Skill if that score is strictly higher.
• A sealed test set is held back entirely and used only for final reporting, never consulted while deciding which edits to keep.

Why bother separating training from selection? Because an edit that only helps on the examples that motivated it isn't necessarily a general improvement. It could just be a narrower Skill that happens to fit those particular cases. Testing against data the edit wasn't built around is what turns "this reads better" into "this measurably performs better."

Why a third, sealed test set? Because the selection set gets reused across many rounds of proposals, and anything reused repeatedly to make accept/reject decisions can gradually be overfit to, the same way tuning too long against one validation set starts to flatter that set specifically. A test set that's touched only once, at the very end, is what keeps the final number honest.

A closer look: accept or reject

SkillOpt doesn't accept an edit because the wording sounds better. It accepts an edit only when the evaluation shows it performs better.

A worked example

Illustrative only. Not a result from the SkillOpt paper or any benchmark. This is built from the documented process, with invented numbers, purely to make the mechanism concrete.

Say a Skill's rule treats an end date as exclusive, when the task actually wants it inclusive. A handful of training tasks come back scored as wrong because of that one rule, and that becomes the evidence for a proposed edit.

Here's what that looks like as an actual run, with invented numbers standing in for a real evaluation:

# optimizer proposes a bounded replace on one rule
edit  replace: "date ranges end on the stated day"
       -> "date ranges are inclusive of the end date"

# candidate scored against the held-out selection set (hypothetical numbers)
gate  candidate: 0.91*   current: 0.84*
      candidate > current  →  accepted

* illustrative scores, not measured results
Skill updated · 1 accepted edit

Those two numbers are the whole decision. Laid out as a chart, the comparison is even easier to read at a glance:

The edit reads better on its own, which by design proves nothing. That's exactly the judgment the validation gate exists to replace. What would make a real accepted edit trustworthy is that it had to handle date-range cases it hadn't seen in training, before the gate accepted it.

And when an edit doesn't help?

Not every proposed edit clears the bar. Most don't, and that's fine, because a rejected edit still does useful work.

A change that only fixed the one example it was shown, while quietly breaking something else, would score no higher than the Skill it was trying to replace, so it would be rejected, exactly as it should be.

What actually changes for you

Each of these follows directly from the mechanism above, not a separate list of claims.

• No added cost at deployment. Optimization happens offline, before deployment. In production, the frozen target model just reads whatever Skill won the gate. No extra model call, no added latency.
• A Skill you can review like a code change. Because edits are bounded, an accepted Skill is a small, specific diff against the version before it, something you can put in front of a reviewer, not a wall of regenerated text.
• A record of what didn't work. Rejected edits are kept, not discarded, so the Skill's history becomes part of what shapes future proposals.
• A compact result, though not a guaranteed one. The published figures, 300 to 2,000 tokens after one to four accepted edits, are a consequence of the edit cap, not a size limit SkillOpt enforces directly.

Installing and running SkillOpt

SkillOpt isn't a hosted product. It's a codebase you run against your own tasks. Before installing, have three things ready: a set of task examples, a way to score whether an attempt succeeded, and access to an LLM endpoint for the optimizer's edit proposals.

A note on exact commands: the specific script paths, flag names, and example config filenames aren't reproduced here as copy-pasteable syntax, because they weren't independently verified against a live checkout of the repository for this article. A fast-moving research codebase's exact interface can drift. Check the repository's own README for the current, authoritative command syntax; the steps above describe the shape of the workflow, which is stable, not the precise flags.

The bottleneck for a first run usually isn't installing the code. It's having a real evaluator and enough examples to fill three separate splits without any one of them becoming too small to trust. A first run doesn't need production-scale data: a small, well-scoped task with a scorer simple enough to trust completely is the fastest way to see what an accepted edit actually looks like, before pointing SkillOpt at anything that matters.

Who SkillOpt is for, and its limitations

SkillOpt fits a specific situation. It works well when all of these hold at once, and runs into trouble wherever one of them doesn't.

• A repeated task, with a reliable scorer. The validation gate is only as trustworthy as what it optimizes toward. If the scorer measures the wrong thing, or the task is genuinely subjective, the gate enforces the wrong thing just as rigorously as it would enforce the right one. A one-off task, or one nobody can score without a person judging every case, isn't a fit.
• A stable execution environment. The model, harness, and tools the Skill runs against need to stay the same between training and deployment. Change any of them and the Skill's validation no longer describes the environment it will actually run in.
• Enough data for three honest splits. Training, selection, and test each need to be large enough to trust on their own; this is a real data requirement, not a formality.
• Tolerance for real, ongoing cost. Every training run involves LLM calls from both the target model's rollouts and the optimizer model's proposals.

And even once a Skill clears the validation gate, that score is evidence, not a production guarantee. A held-out test set describes performance on the data available at training time. It says nothing directly about traffic patterns, edge cases, or environment drift the Skill will meet later. Treating a newly trained Skill the way you'd treat any other change to a live system (introduced carefully, watched, and reversible) is a reasonable precaution the validation process on its own doesn't provide.

Next step

The fastest way to understand SkillOpt is to run it once against a small task of your own and read what it produces. The complete loop shown earlier is enough to make sense of the result.

Get SkillOpt on GitHub →

We often think improving an AI agent means changing the model. SkillOpt keeps the model fixed and improves the instructions instead, but it never judges those instructions by how good they sound. Only by whether the agent actually performs better.

FAQ