Agent Skills Are Runbooks, Not Magic Memory

Agents

Systems

What agent skills actually do, why they help, and where they become another source of failure.

Author

Chirag Sehra

Published

August 29, 2026

An agent can know how to solve a problem and still fail while solving it.

The experiment behind this paper makes that failure concrete. The researchers run software tasks in Harbor, keep the successful and failed attempts, and turn those attempts into two different kinds of advice. One version keeps cleaned execution traces. The other asks a model to write a single SKILL.md file.(Jiang et al. 2026)

The file is meant to contain sections like these.(Jiang, Zhiyuan and contributors 2026)

Use This Skill When
Preconditions
Steps
Common Failure Modes To Avoid
If A Failure Happens
Verify

That is a very practical question. If five earlier runs discovered that a service must be started before a client connects, and that the final result needs a particular format, should the next agent see five noisy transcripts or one short procedure?

The paper Demystifying Agent Skills: Why They Work, Until They Don’t studies what changes when the answer is the short procedure. Instead of asking only whether a skill improves the final success rate, the authors ask three more useful questions.(Jiang et al. 2026)

When does a skill help?
Why does it help?
Where does it fail?

The short answer is that the procedure can make execution steadier. It usually does not give the agent a new store of facts. It gives the agent a better order of operations and a reminder to check the result. That summary is my plain-language reading of the paper’s reported mechanisms.(Jiang et al. 2026)

The paper reports controlled experiments across 8,135 trial records, three benchmark families, and several agent and model settings.(Jiang et al. 2026)

What is an agent skill?

In the study, a skill is a reusable SKILL.md file made from earlier execution experience. It describes things such as these.(Jiang et al. 2026; Jiang, Zhiyuan and contributors 2026)

Use this skill when...
Before you start...
Do these steps...
Avoid these failures...
If something fails...
Verify the result by...

The important part is not the file extension. The important part is the transformation:

messy execution history
        │
        ▼
short, reusable procedure

The authors compare this with Workflow Memory. Workflow Memory also gives the agent earlier experience, but it keeps more of the trace-like form of that experience.(Jiang et al. 2026)

Form	What the agent receives	Main strength	Main risk
Raw	No earlier experience	The agent can explore freely	It may repeat known mistakes
Workflow Memory	Cleaned execution traces	It preserves useful detail	It can be long and noisy
Skill	A distilled procedure	It gives a stable path	It can be applied too rigidly

This comparison matters because the experiment holds the source experience, task, agent, model, benchmark harness, and trial budget constant while changing the representation.(Jiang et al. 2026) If the skill does better than Workflow Memory, the difference is not simply that the skill had more history. The history was packaged differently.

What skills actually change

Across the paper’s matched comparisons, the reported success rates were:(Jiang et al. 2026)

Raw              59.1%
Workflow Memory  55.9%
Skill            61.9%

The most important comparison is Skill versus Workflow Memory. Skills improved the result by 6.06 percentage points, with a reported 95% bootstrap interval of 0.76 to 11.36 percentage points.(Jiang et al. 2026)

The authors then looked at what changed in the agent’s behavior. They found that 65.7% of the skill cases were best described as procedural anchoring. In plain language, the skill gave the agent a usable order, checklist, tool sequence, implementation routine, or verification plan.(Jiang et al. 2026)

Only 4.5% of cases were best described as knowledge injection, where the skill supplied a concrete fact the agent otherwise lacked.(Jiang et al. 2026)

The distinction is:

Knowledge injection:
    "Here is a fact you did not know."

Procedural anchoring:
    "Here is a dependable way to carry out the work."

The second explanation fits the results better.

Where the improvement appears

Skills were most useful against failures that look operational and repeatable.(Jiang et al. 2026)

The paper reports reductions such as:

Environment or infrastructure failures   5.3% → 0.2%
Output or schema mismatches               7.4% → 3.2%
Background service lifecycle failures    2.7% → 0.8%
Shell or code corruption                 1.1% → 0.2%

These are exactly the failures a good runbook can prevent. Start the service first. Use this output shape. Do not paste this command through an extra layer of quoting. Check that the background process is still alive.

Skills were less powerful against deeper reasoning failures. Algorithmic logic errors remained visible, as did cases where the agent inspected code or ran a static check without actually testing the behavior at runtime.(Jiang et al. 2026)

So a skill can help an agent carry out a plan. It does not guarantee that the plan is correct.

Better execution is not the same as better understanding.

Distillation removes noise, but adds a new risk

Workflow Memory has a clear problem. It can preserve failed branches and exploratory detail. The paper reports timeout exhaustion at 10.6% for Workflow Memory, compared with 4.4% for Skill.(Jiang et al. 2026)

But the shorter skill has its own failure mode. Skill guidance was misapplied or ignored in 10.0% of skill cases, compared with 0.4% for Workflow Memory.(Jiang et al. 2026)

The trade-off looks like this:

Workflow Memory
    keeps too much of the journey
    and can slow the agent down

Skill
    removes much of the journey
    and can make one route look more universal than it is

This is where the benefit turns into a risk. A skill should say when the procedure applies and what to check before trusting it. Otherwise, a useful pattern can become a bad rule.

Failed attempts can teach, but only if failure is visible

The authors also test whether the skill writer is told which earlier trajectories succeeded and which failed.(Jiang et al. 2026)

When all source trajectories are successful, hiding the outcome labels changes little. When failed trajectories are mixed in, the labels become much more useful.(Jiang et al. 2026)

That makes sense. A failed trace can contain both:

something worth keeping
and
something that must not be repeated

Without an outcome signal, the skill writer may confuse the two.

The paper gives a notable example from one Terminal-Bench setting with three successful and two failed source traces. With outcome labels visible, the skill condition reached 74.62% success. Without those labels, it reached 40.00%.(Jiang et al. 2026)

The point is not that every skill needs a perfect label. A real run is not automatically a good lesson. A failed run can contain one valuable warning and five steps that should never be copied.

Retrieval is a separate problem

So far, we have assumed the right skill is already in front of the agent. Real systems have libraries of skills.

The paper tests pools of 5, 10, 20, 50, and 100 candidates. It uses three separate tests: embedding ranking, agent selection without execution, and full task execution followed by verification.(Jiang et al. 2026)

These are independent measurements. The result of embedding ranking is not passed into agent selection or full execution. This matters because a good offline choice is not the same thing as useful behavior during a real task.

As the pool grows from 5 to 100, the precision of skills actually used during execution falls from 29.6% to 3.3%. Yet downstream task success changes only from 36.4% to 39.3% in the reported aggregate result.(Jiang et al. 2026)

Consider a simple case. The benchmark may say that service-startup is the correct skill for a task. The agent may instead open test-first, notice the instruction to run the service health check, and finish successfully. The label is wrong, but the advice helped. The reverse can also happen. The agent can select service-startup, read it, and still skip the health check.

So we need to separate two questions:

Did the agent use the benchmark's exact expected skill?

Did the agent receive something useful enough to finish the task?

They are not the same question.

A related skill may contain a useful command or verification step. The exact expected skill may be selected and then ignored. A parser may also miss silent or indirect use.

Exact retrieval accuracy is therefore not a complete measure of skill quality. A useful evaluation must connect what was selected, what was followed, and what the verifier observed.

What the code makes concrete

The paper’s repository shows the experiment as a fairly direct pipeline:

Harbor runs
    → collect successful and failed attempts
    → export cleaned workflow records
    → generate SKILL.md from selected traces
    → inject Workflow Memory or Skill
    → run the same target task
    → use the verifier's result

The clean experiment path lives under pm2s-clean/. export_workflows.py reads run artifacts and turns them into compact per-trial records. generate_skills.py samples a fixed mixture of successful and failed traces, then asks a model to write one general skill. run_context_comparison.py evaluates Raw, Workflow Memory, Skill, and compact procedural baselines in matched environments.(Jiang, Zhiyuan and contributors 2026)

The repository also makes an important distinction. Its broader src/procmem2skills package describes a more general architecture with segmentation, workflow induction, clustering, and atomic skill mining. The clean path used for the reported comparison is simpler. It parses traces and asks a model to produce a task-level SKILL.md.

That difference matters when trying to reproduce the result. The paper’s findings should not automatically be attributed to every more elaborate skill-generation architecture.

There are other measurement limits worth keeping in view. Actual skill use in the full execution arm is inferred from text, JSON, and file-path patterns. It is not guaranteed activation telemetry. The released candidate-pool construction also uses token overlap for similar distractors, while the paper describes similarity in embedding space. These are good reasons to treat the results as evidence from a specific experimental system, not as universal laws about all skills.(Jiang, Zhiyuan and contributors 2026)

The full lifecycle is therefore longer than “retrieve the right file”:

flowchart TB
  A[Useful or noisy experience] --> B[Distill a procedure]
  B --> C[Find a candidate skill]
  C --> D[Read and follow it]
  D --> E[Adapt to the current task]
  E --> F[Verify the result]
  B -. wrong lesson .-> X[Bad guidance]
  C -. wrong context .-> X
  D -. ignored or rigid .-> X
  F -. missing check .-> X

Each arrow is a different place for the system to fail. The paper measures these stages through separate retrieval tests, execution outcomes, and trajectory analysis. The repository shows that some of the execution evidence is inferred from logs rather than captured as direct activation events.(Jiang et al. 2026; Jiang, Zhiyuan and contributors 2026) That is why “the skill was retrieved” and “the skill helped” should never be treated as interchangeable claims.

The maintenance problem

A procedure is written against an environment that can change.

The command may be renamed. The service may expose a new health endpoint. The output schema may change. A repository may move the file that the skill tells the agent to edit.

That creates a question the headline success rate cannot answer:

Who decides that an old skill is no longer safe to follow?

In practice, a useful skill needs at least four pieces of context:

scope       Where does this procedure apply?
assumptions What must be true before it starts?
evidence    Which runs or checks support it?
freshness   When should it be reviewed or retired?

This is my engineering extension of the paper’s results, not a claim that the paper proves a complete skill-maintenance system. The experiments show that distillation helps and that misapplication is real. Versioning and invalidation are the natural response.

The engineering lesson

If you are building a skill system, start with repeatable execution failures.

Capture the setup order, tool sequence, checks, and recovery steps that repeatedly matter. Keep the skill general enough to transfer, but specific enough to guide action. Mark assumptions next to the steps they constrain. Include failure boundaries, not only the happy path.

Then evaluate more than final success. A successful run does not tell you whether the agent read the skill, followed it, or merely succeeded for another reason.

Ask:

Did the skill reduce a known operational mistake?
Did it make the agent slower or more confused?
Did the agent use it in the right context?
Did it verify the behavior, or only produce a plausible patch?
Did the skill help because of a procedure or because of a fact?
What happens when the skill library gets larger?

The paper supports a modest claim. A skill can make good behavior easier to repeat, especially when the problem is a recurring setup, tool-ordering, formatting, or verification mistake.

But a skill is also a decision made from old evidence. It can be stale. It can be too narrow. It can be read without being followed, or followed without being checked.

So the design obligation is sharper than “write better instructions”:

A skill should carry not only instructions, but the conditions under which those instructions are no longer trustworthy.

That means tracking its scope, source runs, assumptions, and last useful verification. It means testing whether the skill changed behavior, not only whether the agent eventually passed. And it means giving the agent permission to stop following a procedure when the current environment no longer matches it.

Sources

References

Jiang, Zhiyuan and contributors. 2026. Demystifying Agent Skills: Research Code and Artifacts. https://github.com/zhiyuanjiang04/demystify-agent-skills.

Jiang, Zhiyuan, Fangrui Huang, Hanwen Xing, et al. 2026. Demystifying Agent Skills: Why They Work–Until They Don’t. https://arxiv.org/abs/2608.14036.