Heroes / Why / measured

The project’s claim, held to account in public: what has been measured, what was written down before measuring, and what this page does not state yet. Every number on this page is copied from the file in the repository that measured it, and that file is named beside it.

• Measured: what the compiler catches. It refuses the plausible mistakes its own rules exist for, and with those rules switched off the same mistakes get through. The mechanism →
• Written down before measuring: the test of the claim. The one number it will report, what each outcome will mean and what the author expects, frozen in a commit together with the sealed tasks. The rules of the test →
• Tried, and not evidence: a practice run. Ten practice tasks proved that the instrument works, and say nothing about the claim. The practice run →
• The first run on the sealed tasks ended level. With the rules and without, the models solved the same tasks within three tries; on the measure fixed before the run, that is called inconclusive. The figures are stated at the publication gate. Why the figures wait →

The claim

A language designed so that every plausible LLM mistake is a compile error will produce working programs in fewer total tokens than a conventional language, even if each program costs more tokens to write.

An LLM is a large language model, the kind of AI model that writes code. Tokens are the pieces of text a model reads and generates. The claim rests on one formula from the project’s design document:

real cost = program tokens × (1 + rewrite rate)

Shorter syntax reduces the tokens needed to write a file. Fixing an error requires more steps: the model reads the message, rereads the code, rewrites a function and tries compiling again. Avoiding a correction can therefore save more tokens than shorter syntax. The whole argument →

The formula has two factors, and the design document says in its own words that only the first is measured. The second, how often a program has to be rewritten, needs a model in the loop, so until a model is put there the formula is a design rule and not an audited one. The test that puts a model in the loop is below, with its rules written down before it runs.

Measured: what the compiler catches

heroes mutate takes working programs and plants one plausible mistake at a time: two arguments swapped, a name misspelled, a case forgotten. Then it asks whether the compiler refuses the result, twice: once with every rule, and once with the rules that are Heroes’ own switched off (--permissive), so the difference between the two is what those rules add. A mutant is one modified program, and killed means the compiler refused it. The tool gives a rate for each kind of mistake and warns never to pool them into one headline, because the rate for each kind is the measurement.

Read one kind at a time, the table shows three things. Some mistakes only Heroes’ own rules catch: a positional argument where a label was required, a name that hides another of the same spelling. Some ordinary checks already catch, a typo in a name or integers mixed with decimals, and there the rules add nothing and are not counted in their favour. And some no check catches, such as a typo inside a string, which the table reports as such. The table itself, every row in plain words and re-run each time it is published, lives in the chapter on errors.

What this measures is the compiler: whether it refuses a mistake a tool planted. It does not measure whether a model makes those mistakes, how often, or whether refusing them saves anything. That is the claim itself, and the test of it is the next section.

Written down before measuring: the test of the claim

The claim is about models, so its test puts models in the loop. The project’s design panel set the rules of that test and the author ratified them, before any model was run on the tasks it will be scored on; the rules are in the record of that sitting. In plain words:

• Tasks kept out of sight. 40 programming tasks written by the author, and twenty more in reserve, each described in English and judged by hidden cases: arguments and input files go in, the printed output and the exit code come out, and apart from the one example a task shows, the model never sees the cases. Every task tempts a plausible mistake on purpose. The tasks are sealed, as the next part shows, and no model had been given one before the run that scores them, as the author reports.
• Heroes against itself, with its rules switched off. The test does not set Heroes against another language. A model writes one program for a task, and that one program is judged three ways, the three arms of the test: strict, with every rule and the compiler’s full message; brief, the same rules and a short message; and control, with the 27 rules that are Heroes’ own switched off, each given a fixed meaning so that the program is built and run anyway. Strict against brief is what the message is worth; brief against control is what the rules are worth.
• The same help in every arm. After a refusal the model is shown the compiler’s message; after a program that compiles, the output of the one example the task shows, beside the output expected; the hidden cases, never. Five attempts at most.
• Three ways to fail. Refused by the compiler; stopped loudly while running; or silent, a program that ends normally and prints the wrong answer. The silent one is the failure the claim is about.
• One number, chosen in advance. How many programs are green, the project’s word for passing every hidden case, by the third attempt: the strict arm minus the control, on one model, Claude Sonnet 5.5, with the specification in the prompt, sent through Anthropic’s API with a fixed prompt. The other models, Haiku 5.5, Opus 5.5 and Fable 5.1 as a ceiling to compare against, and everything else are reported too, each labelled exploratory.

Why a count of working programs, when the claim is about tokens: the sitting’s reason is that what the claim promises a reader is a program that works, with the feedback a user would have. The run also reports, for each arm, the formula’s two factors (the program’s tokens and how many rewrites it took) and the tokens spent until a program works; those are exploratory, and say so where they are reported.

The prediction, as it was frozen

The pre-registration is one file, prediction.md, written in the same commit as the sealed tasks. Its own words follow, copied from that file and checked against it every time the site is built. First, the one number:

**`green@3`(strict, full message) − `green@3`(control) on `claude-sonnet-5-5` in the `spec`
context**, over the 40 tasks at three samples each (panel 226's R2). `green@k` is *the hidden cases
passed by turn k under the compiler's feedback*. The interval is **the cluster bootstrap over
tasks**, the task the cluster and its three samples never nine independent pairs, 2,000 resamples,
the seed in the run's provenance; the test is **McNemar's mid-p over the discordant pairs**, the
exact conditional p reported beside it and never as the primary's. **The smallest effect of
interest is 10 percentage points**: a confirmed difference under it is reported as confirmed and
small, never as success. Everything else is exploratory and says so where it is reported.

In plain words: the share of strict programs that are green by the third attempt, minus the control’s. Beside it goes a range the true difference is likely to lie in, computed by resampling the tasks 2,000 times, and a test that counts only the pairs where the two arms disagree. A difference smaller than 10 percentage points (a difference between two percentages) is reported as small, never as success. Then, what each outcome will mean:

- **Confirmed**: the bootstrap interval's lower bound is above 0.
- **Falsified**: its upper bound is below 0.
- **Inconclusive**: otherwise. **Rule, the author's choice (1a)**: an inconclusive result is
  published as inconclusive, with its interval as measured. n is not extended once the result is
  seen, since extending a sample after reading it inflates the false positives it then reports;
  the reserve is drawn only to replace a task the canary burns or the adjudication finds
  ambiguous.

Confirmed if the whole range lies above zero, falsified if the whole range lies below it, inconclusive otherwise. An inconclusive result is published as inconclusive, and no tasks are added after the result has been seen. Then what the author expects:

- `green@1`: the strict arm **below** the control by **5 to 15 points**, the price of refusing at
  turn 1, before any feedback.
- `green@3`: the strict arm **above** the control by **at least 10 points**, the smallest effect
  of interest.
- The silent rate at turn 3: the strict arm's **at most half** the control's.

The trial on the assistant's ten tasks (measurement 050) is not evidence for or against these:
those tasks tempt no trap, and its arms were level by turn 3 with no discordant pair of 90.

The author expects the rules to cost at the first attempt, the strict arm 5 to 15 points below the control, and to pay by the third, at least 10 points above it, with at most half the control’s silent failures. The number of tasks is fixed: 40 tasks, 3 samples each, for every model and arm. The practice run below was meant to size it, from how often the arms disagree; by the third attempt it found no pair where they did, so how likely the test is to detect a real effect is declared unknown rather than computed from a number that is not there.

The same file closes with the protocol and the digests of what the run will use, in one block: the compiler, the prompt and its layout, the instrument, the specification the prompt carries, the models and their settings. The report sets every run against that block field by field, and prints every difference as a deviation.

The seal, and how to check it

A test a model has read in training measures its memory, not the language, and this repository is public, so a task committed as plain text would reach the next crawl. The tasks are therefore committed encrypted, under keys the author holds, never written on a command line or in a file of the repository. A digest is a fingerprint of a text: change one character and it changes completely. These two lines are what the freeze wrote for the suite and for its reserve, each digest computed from the digests of the tasks inside:

suite c24a5b813f6bba6da2f8ca28f63aa15f76a790be5e012e74468f6924f7ad2274 held-out 40 canary

suite b27ea1a9949cd555646f816f73ac5473276156eea7bd5556eeb019a700f77dd4 held-out 20 canary

Those digests are over the tasks’ text and cases, so they can be checked once the archive is open. Two things can be checked today, from a clone of the repository, with nothing but the files:

$ cd tests/thesis/held-out
$ shasum -a 256 -c suite.enc.sha256 reserve.enc.sha256
suite.enc: OK
reserve.enc: OK
$ git log --format=%H -- prediction.md hashes.txt hashes-reserve.txt suite.enc reserve.enc
a4e09e061130af837a1ed66d61cf247fb10afe4a

The first command checks the two encrypted archives against the digests committed beside them. The second lists every commit that ever changed the prediction, the digests or the archives, and it prints one: the commit that froze them. A later change to any of them would print a second line; and any result, once committed, can be set against this commit to see which came first.

The project’s CI, the automatic checks that run on every push to GitHub, has a job that opens the archive inside GitHub’s own machine, with the keys held as its secrets, and prints counts and digests alone, never a task’s text; where the keys are not there, as in a copy of the repository, it says that it attested nothing rather than passing as if it had. Each task also carries a unique random string, a canary: before a run every model is asked to complete each one, and a task whose string a model completes exactly was in its training data, so it is replaced by the next task in reserve. The same probe, run on the practice tasks before the freeze with four models: 44 probes, 0 completed.

The archive opens by itself. Within 30 days of the first release tagged after the first run on these tasks, or of the suite being declared burned by its canaries, whichever comes first, the author commits the keys and every task becomes readable by anyone. Until then the key is given on request, on the one condition that it is not republished, and every delivery is written down in the sealed folder’s README.

The practice run: proof the instrument works, not evidence for the claim

Before the freeze, the instrument was tried on ten tasks written by the AI assistant that writes the compiler. By the project’s own rule those ten tasks never enter a published number: a suite the assistant wrote measures what the assistant already expects. None of them was written to tempt a mistake. What the run can show is that the instrument measures, what a run costs and where the protocol needed repair. It cannot show whether the claim holds, and nothing in this section is offered as evidence that it does.

Three models, the ten tasks, three samples each, the specification in the prompt; how many of the 30 programs of each model and arm were green at the first attempt, and by the second, third and fifth:

| model | green@1 strict / brief / control | green@2 | green@3, every arm | green@5 | mean turns to green |
|---|---|---|---|---|---|
| Haiku 5.5 | 16 / 16 / 16 | 27 | 28 | 28 | 1.5 |
| Sonnet 5.5 | 18 / 18 / 20 | 29 | 29 | 29 | 1.4 strict, 1.3 control |
| Opus 5.5 | 23 / 23 / 24 | 29 strict, 30 control | 30 | 30 | 1.3 strict, 1.2 control |

• The strict arm pays at the first attempt and is level by the third. A pair is one task and sample judged in two arms, and the strict arm loses one where the control is green and it is not. It lost 3 pairs at the first attempt (Sonnet 5.5 2, Opus 5.5 1), 1 at the second (Opus 5.5), none at the third, and won none at any attempt. The one number the real test will report, read here on Sonnet 5.5, is +0.0 points: 29 and 29, and not one pair where the two arms disagree.
• The silent failures were the same in every arm. By the third attempt: 1 against 1 for Haiku 5.5 and for Sonnet 5.5, 0 for Opus 5.5, the same mistakes in every arm, which no rule names. The same run through Claude Code’s command line instead of the API, in a table of its own and never pooled with it, had every one of its 270 samples green by the third attempt, in every arm.
• Without the specification, no model wrote a program that ran right. 0 green of 270, every model and every arm, with five attempts and the compiler’s messages each time; for each model and arm, thirty tries, the likely range still runs from 0.0 to 11.4%, because thirty tries with no success do not prove a rate of zero. The first answers wrote the main of some other language (Sonnet 5.5 wrote def main 26 times of 30), and five of Opus 5.5’s thirty first answers said the language was unknown to them. One program did run to the end and print a wrong answer, in the control arm alone, and the record calls it an anecdote, not a rate. So the rows above measure what the specification teaches, from nothing the models brought with them.

Read as the record reads it: level is what ten tasks can say. These tasks were not written to tempt a mistake; the sealed tasks, every one of them a trap, are where a difference could show. The pre-registration says the same thing of itself, in the last lines quoted above.

The practice run ran twice. The record of the first says, in corrections written under it, that every first request had gone to Haiku 5.5 whatever model it named: a defect of the instrument, found when a rule that holds each answer to the model its request named stopped a later run. The run was asked again with every model named, and the numbers above are the second run’s.

Not on this page yet: the claim’s number

The run on the sealed tasks has happened. With the rules and without, the models solved the same tasks within three tries, and every program that worked one way also worked the other. Read against the prediction above, the one that stood before the run, that result is inconclusive: the run cannot tell whether the rules help. An inconclusive result is published as inconclusive, and this is it. The figures behind it, and what else the run read, are not on this page yet.

The site states the claim’s number at the project’s publication gate, the gate for a finished thing being announced, and not before: in the words of that gate’s page, stating it unmeasured publishes an opinion with a decimal point.

Why the claim is made: the design choices. What the compiler says when it refuses: the chapter on errors. And what Heroes turned down: the refusals.