Runtime Pass Is Not Correctness

We tried to train Qwen3.8-27B to reason more efficiently. The model missed the bar—and the evaluator failed more fundamentally.

A program can run. It can return the expected output. It can pass every visible test.

It can still be wrong.

Today I am releasing Vinci Technical Report No. 3, Runtime Pass Is Not Correctness: A Negative Reasoning-Efficiency Post-Training Result and Verifier Audit on Qwen3.8-27B.

The study began with a model-training question: could a small supervised fine-tuning stage followed by conservative Direct Preference Optimization teach a reasoning model to spend less time overthinking easy work, finish reliably, and preserve correctness?

The configured intervention did not meet that bar.

Then the study found something more important. The executable-code evaluator we had used to measure correctness accepted every broad adversarial non-solution we tested—24 out of 24. Some of those programs selected the expected answer from the length of the input without examining the input values at all.

The evaluator executed the code. The outputs matched. The run passed.

The implementation did not solve the task.

That failure changed the meaning of our earlier correctness numbers. It also changed what this research program is about.

The result in one line: we did not establish a useful reasoning-efficiency improvement, and we learned that executable success is not evidence of correctness until the evaluator has survived serious attempts to make wrong work look right.

No trained checkpoint is being released. No evaluation bank from this lineage is being promoted as qualified. The technical report is the release.

The original goal: less wasted reasoning, not generic brevity

Reasoning-capable models can spend thousands of tokens reconsidering a task, repeat a derivation they have already completed, or exhaust the generation budget before delivering a usable answer.

Reducing those tokens sounds straightforward. It is not.

A model can become “efficient” by refusing more often, guessing early, truncating necessary work, skipping verification, or learning what the evaluator rewards rather than satisfying the task. Token reduction is useful only when the shorter trajectory still delivers the correct outcome.

Our intended policy was task-conditional:

• easy requests should receive brief reasoning and a direct final answer;
• closed-form problems should get one sufficient derivation instead of repeated derivation;
• code tasks should produce a verified implementation in the requested format;
• agents should use tools, incorporate feedback, and verify the resulting state;
• unanswerable tasks should identify missing information instead of guessing or looping; and
• genuinely difficult tasks should retain enough reasoning depth.

The intervention used 440 accepted SFT demonstrations and 700 accepted DPO preference pairs on a pinned Qwen3.8-27B checkpoint. The DPO data was deliberately not a simple “shorter is better” corpus. In 56.6% of accepted pairs, the chosen response was not shorter than the rejected response. Correctness and verified delivery were supposed to outrank length.

The development evaluation covered 400 nominal tasks across direct, closed-form, executable-code, agentic, and answerability strata, under two fixed inference seeds.

The target was demanding on purpose: at least a 20% reduction in median reasoning tokens at both seeds, alongside lower total generation and no material regression in verified delivery.

A bar matters only when it can say no.

The configured recipe missed every positive efficiency target

For the one-epoch SFT-to-DPO recipe, a positive reduction means fewer reasoning tokens than the base model. A negative value means the trained model used more.

At one seed, the trained model reasoned longer than the base. At the other, it improved by less than one quarter of the required amount. Adding DPO after SFT changed median reasoning by approximately zero.

The cap-exhaustion point estimate declined modestly, but that did not rescue the intervention. The contract was not “find one favourable termination metric somewhere.” It was a material and reproducible reduction in reasoning and generation while preserving verified delivery.

The configured recipe failed.

Figure 1 — What the one-epoch recipe actually did. Grouped bars for both seeds show SFT alone, the full SFT+DPO arm, and what DPO added on top of SFT. Positive means fewer reasoning tokens, negative means more. The target was +20% at both seeds; the arm landed at −10.9% and +4.9%, and the DPO increment is the pair of near-zero bars.

A stronger training dose appeared to work—until we tested the serving policy

After the one-epoch result, we authorized one bounded screen with two DPO epochs.

At first, the stronger dose looked more promising, especially on executable-code tasks under the study’s reasoning_effort=xhigh condition. Reasoning length and cap exhaustion moved in the intended direction as the DPO dose increased.

That looked like evidence that stronger preference optimization was the lever.

It was not enough to establish that the trained weights caused the change.

The Qwen serving template did not treat reasoning_effort as passive metadata. Under xhigh, it injected an instruction telling the model to think carefully, validate assumptions, and consider alternatives. Under medium, it injected no effort-specific instruction.

So we ran a serving-control screen.

On the screened executable-code shard, changing the untrained base model from xhigh to medium moved observed cap hits from 13 of 20 to 0 of 20 and reduced restricted mean reasoning length from 6,407 to 608 tokens.

The screen was small and clustered. Zero observed cap hits out of 20 is not a known population rate of zero, and the formal matched-effort comparison remained underpowered.

But it established the decision-relevant point: the apparent trained benefit under xhigh was not identifiable as a weight effect. A serving-only change reproduced nearly the entire termination improvement on untrained weights.

Continuing to increase the DPO dose before treating the serving controller as a comparator would have meant training the model to resist an avoidable harness choice.

We retired the adapter.

Figure 2 — The same improvement, without any training. Untrained base weights, run at medium effort instead of xhigh on the screened code shard. Cap exhaustion falls from 65% to 0% and mean reasoning length from 6,407 to 608 tokens. This is the control that cost us the dose-response result: a serving setting reproduced the effect we had been attributing to the adapter.

Then the correctness evaluator failed

The more consequential finding came from attacking the original executable-code bank.

We constructed broad shortcut programs, plausible alternatives, near misses, and metamorphic variants across all eight content-distinct code families.

Every broad adversarial non-solution passed:

24 out of 24 deliberately incorrect shortcut programs were accepted by the original evaluator.

The simplest class mapped the evaluator’s three known inputs directly to their expected outputs. Another selected the answer from one incidental scalar. The most revealing used only properties such as len or sum to identify the expected result without implementing the requested function.

In one family, input length alone uniquely identified all three test cases. The program did not need to inspect a single input element. It ran, printed the expected values, and received a pass.

That is not functional correctness. It is specialization to an inadequate test set.

The bank also contained less independent evidence than its task count implied. Its 80 nominal executable-code tasks represented eight content-distinct problems cloned ten times. Names and identifiers changed, but the prompts and tests were otherwise repeated within each family.

Repeated identifiers are not repeated evidence.

Once the evaluator failed qualification, we withdrew absolute code-correctness rates from the original bank. We did not assume relative rankings were safe either, because different policies may produce different kinds of shortcuts, alternative implementations, and edge-case errors.

The previously reported 0.844 fixed-policy correctness frontier is suspended. It is not a product result.

24 of 24 deliberately incorrect shortcut programs were accepted by the original evaluator.

Runtime feedback and certification are different jobs

We replaced the original design with two channels:

• Runtime tests are small and visible. They help a model decide whether to accept, retry, or repair its current attempt.
• Certification tests are protected and broader. They decide whether the completed artifact supports a scientific or release claim.

The split substantially improved broad-shortcut rejection.

Of 322 broad shortcut probes, 154 passed the visible runtime channel. None passed certification.

That is not an argument that runtime tests are useless. It proves they are doing a different job. A visible suite can be a good local workflow signal while remaining a terrible final authority.

The replacement certification bank still did not qualify.

• It accepted 7 of 157 near-miss or partial programs.
• Its frozen abstract-syntax restriction rejected 7 of the bank’s 40 correct reference implementations for a non-behavioural reason.
• A later independent mutation population generated 1,862 provably wrong programs; 52 still passed certification.
• Those leaks affected 19 of the 40 protected tasks under that mutation population.
• Combining two non-exhaustive audit methods established a lower bound of at least 21 of 40 tasks with a known certification flaw.

That last count needs care. It is a cross-method lower bound, not an estimate that exactly 52.5% of all tasks or evaluators are defective. The mutation methods were designed to find necessary qualification failures, not estimate real-world prevalence.

The replacement was much better than the original bank. Better is not the same as qualified.

Figure 3 — Where the evaluators let wrong programs through. Each bar is a false-accept rate against a different attack set. The original bank accepted 24 of 24 broad shortcuts. The replacement rejected all 322 at certification — a real improvement — but still accepted 7 of 157 near-miss programs and 52 of 1,862 independently generated wrong mutants. The last two bars are why it is not qualified.

A test suite should be measured by the wrong alternatives it kills

Evaluation coverage is often described with category labels: boundary case, empty input, tie, malformed structure, hidden test.

Those labels can create false confidence.

A test containing an empty input does not detect every empty-input bug. A test containing a tie does not test the required tie-break unless two plausible implementations behave differently on that case. Separate tests for conditions A and B do not necessarily test the conjunction A-and-B.

The operational question is sharper:

Which plausible wrong implementations does this evaluator reject, and which legitimate alternatives does it continue to accept?

Mutation testing turns that into a measurable procedure. Construct a program that violates a stated requirement, identify an input where it differs from correct behaviour, and ask whether certification kills it.

But even mutation testing can overfit. A repair built from one known mutant family can catch those exact mutants while leaving the underlying defect class open. Our targeted repairs killed the witnessed failures. A fresh mutation population generated by a different method found new leaks.

A repair cannot be the sole author of the population that declares the repair complete.

We also measured the wrong quantities before correcting them

The evaluator audit was not the only methodological correction.

An early scorer removed cap-exhausted attempts from the denominator by treating them as extraction failures. That estimated correctness conditional on producing a parseable final answer—not correct delivery across all attempts.

But budget exhaustion was the central failure mode the study was supposed to measure. Removing those episodes made non-delivery disappear from the metric.

In the retained calibration census, 88.5% of logged extraction failures were cap hits. After correcting the denominator, one pooled reconciliation quantity moved from 0.9622 to 0.8386. That pooled number is not itself a model verdict. It demonstrates that the two scoring procedures answered materially different questions.

We also used medians in a code stratum where more than half of some arms hit the 8,192-token cap. Once more than half the observations are censored at the same boundary, the observed median is not a meaningful estimate of the uncapped median. It is a lower bound pinned to the limit.

And we initially treated decoding seeds and renamed task clones as independent experimental units. They are not. Seeds estimate within-task variation. Renamed copies do not establish generalization across problems.

These are not cosmetic statistical details. Each one changes the scientific sentence a number can support.

Reproducible can still be wrong

This program used content-addressed artifacts, frozen hashes, schemas, and repeatable execution.

Those controls prevented silent drift. They did not guarantee semantic validity.

We found cases where:

• a production path selected the wrong grading authority even though tests injected the correct one;
• the same object exposed hidden grader evidence to the model;
• stale Python bytecode could execute after a rapid same-length source repair;
• a supposedly atomic protected-bank update validated all changes first but then wrote them through a non-transactional loop; and
• the exact digest was correct for bytes that represented the wrong authority or estimand.

A deterministic wrong evaluator remains wrong.

A reproducible mistake is still a mistake.

Vinci Eval Integrity 0.1

The reusable output of this work is not a checkpoint. It is a provisional seven-check admission record for evaluators that may influence model selection, optimization, release, or product claims.

E1 — Scope, identity, and independence. Declare semantic families, near duplicates, reference identity, and the real statistical unit.

E2 — Contract clarity. Independent prompt-only implementers should be able to satisfy the tested behaviour without seeing the reference or tests.

E3 — Runtime and certification separation. Visible feedback may guide repair. It may not become the final correctness score or training reward.

E4 — Bidirectional discrimination. Reject broad shortcuts, near misses, boundary faults, conjunction faults, and independent mutants while accepting legitimate alternative implementations.

E5 — Estimand, censoring, and retention integrity. Define the denominator, unit, direction, comparator, and cap treatment before opening outcomes, and retain sufficient raw evidence to reconstruct the result.

E6 — Production-path and authority binding. Qualify the actual loader, authority, environment, information boundary, write path, and evidence sink used in production.

E7 — Exact-procedure power and independent requalification. Simulate power against the complete decision rule at the correct clustered unit, and use a fresh post-repair population built independently of the patch.

This is an internal and provisional standard, not an external certification. Neither P-BREVE evaluator bank cleared all seven checks at the report cutoff.

Figure 4 — The seven checks, as a single admission record. Each check is reported separately and on purpose: collapsing them into one “evaluation passed” flag would recreate the compression error this report documents. Neither P-BREVE bank cleared all seven at the report cutoff.

What we are releasing—and what we are not

This is development-tier internal evidence on one Qwen3.8-27B revision, one narrow LoRA intervention, one accepted training corpus, and a bounded one-to-two-epoch dose range.

It does not establish that DPO cannot improve reasoning efficiency. It does not prove that medium is globally optimal. It does not estimate how often ordinary users encounter cap exhaustion. It does not establish broad safety, capability preservation, or a universal mechanism of “overthinking.”

No external laboratory reproduced the work. No external peer review or independent external audit was completed at the report cutoff. Several important evaluator audits were designed after the original model outcomes were visible, which makes them valuable diagnostic evidence rather than pre-registered confirmation.

The model checkpoint is not being released as a candidate. The executable evaluator bank is not being released as a qualified reward. The original correctness story is not being preserved through softer wording.

We are releasing the negative intervention result, the serving-control confound, the evaluator failures, the correction history, and the resulting evaluation standard.

What changes next

The next reasoning-efficiency experiment should not begin by training another adapter.

It should begin by comparing controllers:

1. the stock or product-default serving policy;
2. static medium;
3. a simple rule-based controller over effort settings;
4. a learned task-conditional controller; and
5. only then, a trained model under the same controller and matched compute budget.

That design asks the right incremental question: do weight updates add value beyond a cheaper serving-time policy?

Evaluator work will also become its own research program with a separate charter, budget, claims ledger, and stop conditions. Model results should not remain open indefinitely while the same lineage repairs the instrument after seeing each outcome.

A future P-BREVE-02 replication may still be valuable. It should begin only after the evaluator is qualified, the serving controller is declared as a comparator, and the intervention recipe is frozen before protected outcomes are opened.

Why publish this

There was no successful model at the end of this study.

That does not make the work empty. It makes the claim narrower.

We learned that our configured training recipe did not produce a material, reproducible efficiency improvement. We learned that a simple serving choice dominated the apparent stronger-dose termination effect on the screened code tasks. Most importantly, we learned that a deterministic, executable, content-addressed evaluator can still be incapable of distinguishing a real solution from a trivial shortcut.

The wrong response would be to hide the failed model, quietly replace the bank, and present only the repaired pipeline.

The useful response is to preserve what happened: which claims survived, which were narrowed, which were withdrawn, which were suspended, and which were contradicted by later evidence.

At Vinci, we want “verified” to mean more than a green check beside a process that happened to complete.

A runtime pass is useful feedback.

It is not correctness.

This time, the model is not the release candidate.

The report is.

---

Read the research

• Vinci Technical Report No. 3: Runtime Pass Is Not Correctness
• GitHub evidence package
• Report PDF
• DOI record

The technical report and this blog post are authored by George Pu.