Is the 200-instruction skills ceiling still real?

The 200-instruction scare

If you keep a skills file packed with if-X-then-Y, always-include-Z, and never-say-W rules, the working anxiety is not that the agent will crash. It is that the output will look right while a subset of those rules quietly disappeared.

Laurie Voss, head of developer relations at Arize AI and a co-founder of npm Inc., now spends his time on how to test AI systems. In a talk uploaded to the AI Engineer YouTube channel on 9 September 2026—the conference session date is not in the recording—he locates the scare in a number he heard as an aside. A few months earlier, at AI Engineer Miami, Dexter Horthy mentioned in passing that an agent can follow about 200 instructions before it starts forgetting, a 2025 figure Horthy said might already be better. The talk was not about skills files. Voss stopped listening anyway, because 200 is not very many instructions.

His practitioner framing is that a decent skills file blows past 200 almost immediately. Each discrete rule counts separately: if the user says X, do Y; always include a section on Z; never use phrase W. If the model quietly stops tracking after 200, that is a hard ceiling on what you can build. The research is aimed at the feeling you get when you hand over pages of tone, formatting, and edge cases, get a plausible result, and cannot tell whether the agent followed the file or handed back a close simulacrum.

• Laurie Voss at AI Engineer (YouTube)

Where 200 came from, and why keywords are a ceiling

Voss says the 200 figure is not folklore. He traces it to IFScale, a benchmark paper from the previous year by an author whose name he pronounces as “Jeroslowitch,” plus co-authors. Radar did not fetch that paper, so the printed author spelling is not reconstructed here.

The protocol, as he describes it, is simple. Ask the model to write a business report. Give it a growing list of exact words that must appear—customer, revenue, and so on. Each required word is one instruction. Density n is how many rules you piled on; accuracy is what share of those exact words showed up.

Required keywords are a proxy for named constraints of the same shape: include a pricing section, or never use this phrase. Voss calls the resulting score a ceiling. If a model cannot track 200 words in one prompt, it will struggle with 200 more complicated instructions; harder real rules should lower the number, not raise it. That is still a proxy task. It is not a census of real skills files, and it is not a test of whether an agent reasoned over conflicting brand, legal, and tool rules.

Voss’s replication of the old ceiling

Before chasing new models, he says, good science meant rerunning the original set. Models live and die quickly. Of about ten original IFScale models, he says only GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro were still reachable by API in his window. After he first published the research, one of those three was later retired, so the lineup was already down to two. Availability and retirement stay as his retest-window claims; Radar did not confirm a catalog.

On a log-scale plot of accuracy versus rule count, he says his curves matched the paper within noise. By about 500 rules the remaining models were missing 30–50% of constraints. A year earlier, frontier models started falling apart around 200–300 rules. Those figures are Voss’s replication result. Radar did not rerun the benchmark.

Current models walked through 500, so he moved the wall

He then pointed the same original test at what was then the current frontier: GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro. He tested Opus 4.7 because, he says, 4.8 shipped a week later. All four scored 100% immediately. The old benchmark topped out at 500 words, and the models walked through that ceiling without noticing it was there.

To find a new wall he moved the goalposts: 500 to 1,000, then 2,000, then out to a 10,000-word list. On the extended log-scale chart, he says newer curves hold until roughly 2,000 named constraints, and the best until about 5,000—close to 10x simultaneous instruction-following in about twelve months. Those are his headline numbers from this extended IFScale-style run, not Radar-tested facts.

Everyday feel and the metric can disagree. He says the jump from GPT-5.1 to GPT-5.5 felt incremental, but this score moved a lot. He also says Opus 4.8 already made the chart slightly stale, so skills- and prompt-length assumptions older than about six months should be re-engineered. That is dated advice from this talk, not a calendar law for every stack.

Four failure modes instead of quiet forgetting

The unexpected result was that new models no longer fail mainly by silently dropping rules. Each of the four failed in a distinctive way that broke the original scoring story.

DeepSeek 4, in his run, is the traditional case: it starts forgetting around 750 rules and by 2,000 is dropping nearly half. He trusts that pattern because it is measurable. It is his DeepSeek curve, not a product-wide ranking.

Opus 4.7 treated random-word combinations as dangerous and refused at the API. A safety classifier, he says, would bail if words such as anthrax and cyanide landed in the same prompt. After he filtered the vocabulary through an OpenAI safety filter, Claude performed well—but he says it can refuse as early as 200–300 instructions on medical or dual-use content. That is his observed behavior on this random-word prompt, not a general safety evaluation.

Gemini 3.1 Pro stayed strong out to 5,000, then spent its thinking-token budget trying to satisfy every rule and returned a short answer missing the required words. In one anecdote from the same run, a 10,000-token budget became about 9,500 tokens of thinking and a 500-word response with none of the required terms.

GPT-5.5 was strongest on the chart at about 99% accuracy to 5,000 rules. Pushed further, it started the report, then abandoned the task as a stupid request—a polished half-answer that still counts as a fail and is hard to notice unless you read to the end.

His taxonomy: DeepSeek quietly forgets, Claude refuses, Gemini overthinks into silence, and GPT-5.5 finishes part of the job and declines the rest. Instruction-following no longer has one recognizable failure mode.

What that changes in a skills workflow

A year ago, he says, the smart move was keeping skills under about 200 instructions and sharding the rest into a labyrinth of subskills. He now says that compression is no longer required and skills files can be much longer. If a use case needs 100 or 300 specific rules, they can live in one prompt without the old anxiety that the model silently ignored a subset.

Two thousand named constraints, in his illustration, is an entire style guide of brand and legal rules that a year ago would have been sharded across specialized agents. That is an analogy, not a measured brand-guide experiment.

The old question was whether the model could hold the rules. The new one is whether stuffing thousands of instructions is worth a larger, slower, more expensive prompt. The hard wall became a cost-and-latency trade-off. He does not offer a public price list; cost stays qualitative.

• Skills files no longer have to stay under about 200 instructions with a maze of subskills.
• A few hundred specific rules can live in one prompt, in his view, without the old silent-drop anxiety.
• The binding constraint is now cost and latency, not a hard capacity wall.

What the keyword test does not prove

Stuffing random words into a fake report is evidence long skills might work, not proof they do. Models in his set hit the wall anywhere from about 750 to 9,000-plus rules, so the model choice still matters.

The test does not measure clear reasoning over a giant prompt, or conflict resolution among rules. Tracking 2,000, 5,000, or possibly 10,000 instructions is not the same as reasoning over them.

He cites later context-rot work from Chroma, as he summarized it: across 18 models, long-input accuracy can fall 30–50% well before the context-window limit, and coherent structured text is more vulnerable than shuffled instructions. He said he had not fully read the report. Radar has no paper body, so those figures stay as Voss’s recap.

A later paper he cites, testing 46 models on revisiting language-model reliability in instruction following, found a model can ace a keyword-style benchmark and still collapse if the same instructions are reworded or reordered. Capacity went up; reliability is still a problem. He names Firebench, CCR Bench, and Guidebench as newer attempts to measure messy real constraint-following. Those are emerging related work he mentioned, not Radar reviews.

The hard part is now verification

Claude refusals are loud API errors. GPT’s polite half-finished report looks like a real answer unless you read to the end. Silent failure is the dangerous one.

Because frontier models can fail silently on tricky production tasks, he argues you need output monitoring—another model checking the result, which he calls an eval. That is monitoring advice from this talk, not a product ranking. As an experiment-cost aside, he said the full query set cost him $29 for about 2,300 calls across seven models; that is not a public API price list.

His close: a year ago writing a skill was a compression problem of fitting rules without losing the plot. That ceiling moved about 10x on his metric. The hard part is now verifying that the model actually did what it was told, which a better prompt does not solve. Revisit prompt- and instruction-size assumptions from six months earlier because they may already be wrong. He pointed to an on-screen GitHub URL for code and data; the captions do not speak the URL.