Most “vs” pages are not comparisons. They are three product write-ups stitched together. One tool is the one the author already uses. The other is opened long enough to take a screenshot. The task changes halfway through. A price is copied from memory. The comments then argue about a comparison that never quite happened.

Claude Radar is going to publish comparisons anyway. Never putting two tools next to each other would just dodge the job. The honest version is slower: lock a task, lock a stop condition, write down what the operator did, and refuse claims that look like evidence but are not.

This article is that method: a protocol a second person could follow from the page alone. Times, dollar figures, and win rates wait until we have a public fixture and a test with a recorded run date.

The problem with tool roundups

A coding agent is not a camera. You cannot put two of them on a tripod and shoot the same brick wall. The operator is in the loop: the prompt, the moment they interrupt, the files they refuse to let the tool touch, the test they decide is “close enough.” If that loop is invisible, the comparison is a personality test.

Roundups also mix categories. A terminal agent, an editor with inline completions, and a browser app-builder can all “write a to-do app.” They do not all leave you with the same kind of repository, the same kind of tests, or the same kind of work the next day. Treating them as interchangeable products is how you get a grid of logos and a vague closing paragraph.

If a second operator cannot rerun the comparison from the page, it was a diary entry with a comparison headline.

We would rather publish fewer pieces, with the diary part labelled as diary and the comparison part labelled as comparison. This protocol is for the second kind.

What we compare, and what we do not

The magazine is English-first and global. The centre of gravity is Claude Code, because that is what this magazine covers. Cursor, Codex, and vibe-coding tools sit in the same field of view because readers actually move between them. Automation catalogs, template marketplaces, and “1,000 workflows” directories can appear later as a side note if a builder’s work collides with them. They are not the core of this magazine.

A comparison belongs on this site when all of the following are true:

  • The reader is trying to produce working software, not a slide.
  • Each tool can be aimed at the same public fixture.
  • We can name a Definition of Done that is not “it felt faster.”
  • We are willing to publish misses, not only the run that behaved.

A comparison does not belong here when the only source is a marketing page, a remembered launch demo, or a social thread. We will quote official documentation with a retrieval date. We will not launder it into a benchmark.

The shared fixture

Every comparison in the first editorial set should start from one small public web app — the same repository, the same failing tests, the same README. The point of a fixture is not to look cinematic. It is sameness. If Run A is “rebuild our production monolith from memory” and Run B is “scaffold a counter,” we have compared two different jobs.

The fixture should be boring on purpose:

  • Small enough to read in one sitting.
  • Public, so a reader can clone it.
  • Free of client secrets, production logs, and private names.
  • Already equipped with tests that fail for a known reason at the start of a run.
  • Unrelated to this magazine’s own codebase, so we are not grading a tool on how well it flatters us.

The first public fixture will be linked from every comparison that uses it, with a commit hash. Until that hash exists, comparisons stay in method rather than results.

The protocol

These are the steps, written so they can be copied. If a published comparison skips a step, the piece should say which step and why — not hide the skip in a smoother narrative.

  1. Record the tools. Record tool names, channels or build identifiers, model choices the tool exposes, editor or CLI version, OS, and the date the run started. If a tool does not expose a version, write that down. Do not approximate.
  2. Reset the fixture. Start from the agreed commit. No leftover node_modules cleverness, no stashed WIP, no extra skills or rules files unless the comparison is specifically about those files. The starting tree is part of the method.
  3. Deliver the same prompt packet. One packet per run, stored in the article or an adjacent file. No “I also told it privately.” If the operator must add a constraint, it is an intervention (step 4), not a secret prologue.
  4. Cap the interventions. Decide in advance how many operator turns are allowed after the packet: a number, not “until it looks right.” Each intervention is logged in one line: what was said, why, and whether it leaked information the other run will not get.
  5. Stop at the same Definition of Done. For the first fixture we expect: the agreed tests pass, the app serves locally, and no extra feature was smuggled in to make a screenshot prettier. If the tool cannot get there inside the cap, the result is an incomplete run, not a poetic “almost.”
  6. File the log before looking at the other log. Write the notes for Run A as if Run B does not exist. Then the reverse. Only after both files are saved may we put them in one table. Peeking is how diaries pretend to be trials.
  7. Publish the limits next to the table. A comparison page that cannot list what it did not measure is not ready.

Example prompt packet

The packet below is a specimen of tone and scope. Copy it if you want to try the run yourself. A local try is not a published result.

Task in plain language: add a session-note parser to a small TypeScript app so a raw text file becomes a structured note, with tests that already describe the shape.

You are working in a small public web app at the tagged commit.

Add `src/notes/parseNote.ts` so the tests in
`src/notes/parseNote.test.ts` pass. Do not add features
the tests do not describe. Do not rewrite unrelated files.

Constraints:
- TypeScript, no new dependencies.
- Refuse sample data that looks like a real person.
- If a test is unclear, stop and ask one question.
  Do not invent a business rule.

Stop when `npm test` is green for that file, or when you
cannot proceed without a decision from me. Write a short
summary of files touched and any test you could not satisfy.

The tests the packet points at should already exist in the fixture, so the tool is not grading its own homework.

What we will record

The table is the contract for a later comparison page. Empty cells in a published piece mean “we did not measure this,” not “fine.” We will not fill them with adjectives.

Criterion What we record What we will not infer
Done or not Whether the agreed tests passed inside the intervention cap. That the tool is “better at engineering.”
Time to a working run Elapsed time on a named machine, start to stop, if we choose to measure time at all. Team productivity, or how you would do on a tired afternoon.
Edit quality Diff size, files touched, tests still passing, obvious unrelated rewrites. Taste, seniority, or “clean architecture.”
Operator effort Count and text of interventions we actually sent. The skill of operators in general.
Cost Only metered usage the tool exposes, with the date and plan name. Otherwise: “not disclosed.” Your monthly bill, or a ranking by sticker price.
Failure modes Where it stopped, what it broke, how we recovered — or that we did not. A personality for the model.
Setup friction Install, login, and project-hook steps we had to perform, as a checklist. That a longer setup is always worse.

What a published comparison will contain

When a comparison leaves the candidate list, the page should carry:

  • The fixture hash and the prompt packet.
  • The tool list from step 1.
  • Both logs, or a fair abridgement that does not drop the misses.
  • The criteria table, with empty cells left empty.
  • Links to official docs used for plan names or limits, with retrieval dates.
  • A corrections stub, even if it only says “none yet.”

If we later add affiliate links or a sponsor, they will be labelled and kept off the scoring table. A rank that moves after a commercial relationship is a correction, not a refresh.

Limitations

This protocol is already biased, and we would rather name the bias than dress it up.

  • One fixture is not the industry. A parser in a small TypeScript app says little about native mobile work, data science notebooks, or a million-line monorepo.
  • The operator is a variable. Another person will interrupt sooner, or not at all. We can log our interruptions; we cannot subtract ourselves.
  • Tools move. A test with a recorded run date is a result from that date. We will not silently restyle old tables as current events.
  • Exposed meters are incomplete. If a product hides token use or bundles usage into a seat plan, we cannot invent a cost column.
  • No comparison has been run yet. Empty tables stay empty until a test with a recorded run date exists.
  • Independence is a practice, not a mood. Using Claude in the name does not let us favor that tool. If a comparison reads as favoring a tool, the method failed.

Sources policy

We will distinguish four kinds of source, and not let them mix:

  1. Official documentation — quoted or paraphrased with a URL and a retrieval date. Pricing and limit figures come from here or they do not appear.
  2. Our fixture and logs — public enough to inspect. Private client work stays out.
  3. Other journalism and primary posts — linked when they are the source of a claim about the world, not used as a substitute for a run we skipped.
  4. Marketing language — usable as a quote about what a vendor says, never as a measurement.

We will not scrape private chats, paste internal runbooks, or treat a social screenshot as a version number. When an affiliate or sponsor relationship exists, it will be named on the page that carries the link.

Correction policy

If a published comparison is wrong, the page changes in public.

  • Factual errors (wrong version, wrong plan name, misread test) get a dated note at the top or in a corrections block, and a fix in the body.
  • A new run is a new run. It does not overwrite an old table without saying so.
  • We will not silently reorder a ranking after a commercial deal, a vendor email, or a bad night of traffic.
  • If we cannot defend a sentence against this protocol, we delete the sentence, not the embarrassment.

No comparison has been published yet, so there is nothing to correct. The corrections block is here so later published pages use the same process.