A bug in my scorer cost Qdex-1.5B three points on HumanEval

Qdex-1.5B scores 42.1% on HumanEval. For about a month it scored 39.0%, and the model never changed in between. What changed was the thing doing the counting.

#The model solved five more problems than my scorer credited it with

HumanEval is 164 problems. Under the harness I wrote, Qdex passed 64 of them. Under the fixed one it passes 69. Nothing about the weights moved — I did not retrain, re-tune or re-quantise anything between those two runs. Five solutions that worked were being filed as solutions that did not.

Three percentage points is not a dramatic number and I am not going to pretend it is. What makes it worth writing down is that I published the wrong one, and that the process which produced it looked exactly like a process that was working.

#The extractor discarded correct answers over how they were formatted

A code benchmark does not read a model's mind. It reads a blob of text, cuts the program out of it, runs that program against the problem's tests, and records a pass or a fail. The cutting-out step is called extraction, and it is the part nobody writes about because it feels like plumbing.

Mine was brittle. A 1.5B model asked to write a function does not reliably produce one clean fenced block — it wraps things in prose, adds a second example, closes a fence it never opened, or trails off mid-comment. My extractor handled the tidy cases and dropped the rest on the floor. A dropped answer is not scored as unparseable. It is scored as wrong, which is a different claim entirely.

So the fix was not clever. Harden the extractor against the malformed shapes, add a regression test holding the specific outputs that had been discarded, re-run the whole benchmark rather than the affected rows. That last part matters more than it sounds: re-scoring only the five would have carried over 159 results from a harness I had just admitted was faulty.

#I audited the number because it disappointed me, and that is not a reason to be proud of

Here is the part I would rather leave out. I did not go looking for a scoring bug out of rigour. I went looking because the number was implausible in one specific way.

The base model, Qwen2.5-Coder-1.5B, already solves 40.2% of HumanEval when you let it complete raw code — 66 of 164. It scores 1.2% when you ask it a question instead, because it was never taught to answer one. The entire point of the fine-tune was to close that gap without losing the underlying ability. At 39.0% my model was sitting below the base model's latent skill, which would have meant the tuning had quietly cost it something.

That did not fit. So I opened the harness. If the bug had run the other way — if the extractor had been generous and handed me 45% — it would have fit perfectly, I would have published it, and I would still be quoting it today.

Which is an uncomfortable thing to notice about your own process, and I think it generalises. You only ever audit a number that disappoints you. A flattering bug produces a result that confirms what you hoped, and nobody opens the code to ask why the good news is good. This is reasoning rather than a measurement — but it does predict that published scores across the field skew high, and it costs nothing to assume that about your own.

#Public evaluation harnesses have shipped the same class of bug

This is not a story about being an amateur, which is the reading I expected and the reason I checked before writing.

EleutherAI's lm-evaluation-harness — the harness a great many published results are produced with — carried a scoring bug in its Math Verify path where the code stripped \boxed{} notation from an answer before handing it to a parser that relies on that notation to find the answer. Same shape as mine: the model was right, the pipeline threw the answer away, the row was recorded as a failure. The same project has also had evaluation documents leak into their own few-shot prompts on several tasks, humaneval_instruct among them.

And HumanEval+ exists as a project largely because the original benchmark shipped with test harness bugs, misplaced assertions, import faults and mistakes in its own reference solutions. A 2026 paper on benchmark-validity audits puts the general case better than I can: audit conclusions can be silently manufactured by implementation details.

#A benchmark number measures two things and names only one of them

"Qdex-1.5B scores 42.1% on HumanEval" reads like a property of the model. It is not. It is a property of the model *and* the harness, jointly, and the sentence names one of them.

That is why leaderboard positions move when nobody has retrained anything, and why two people evaluating the same open weights on the same benchmark get different figures and both are being honest. The weights are the reproducible half. The extraction rules, the prompt template, the stop tokens, the timeout, how a malformed output is classified — none of that travels with the model, and all of it lands in the number.

The practical version is short. Publish the harness with the score, or expect the score to be treated as a claim rather than a measurement. Mine is on GitHub for that reason, and if somebody re-runs it and gets 41.4% I would rather have that argument than the alternative, which is nobody being able to have it.

#The old number is still live on the Hugging Face model card

39.0% was wrong in five places on this site and it took until 10 September to find the last one, sitting on the homepage — which AEO_PLAYBOOK records as the most-crawled page here, so of the five it was the one being read most.

One is still wrong. The Hugging Face model card quotes 39.0% (64/164) and updating it needs a login I cannot reach from the repository, so it has not been done as I write this. That is the honest state of it, and the reason I would rather say so than quietly fix it before publishing: a post about auditing your own numbers that presents a tidy finished picture is doing the thing it is arguing against.