We tested one frozen character recipe across three model families. None met the bar.

One frozen character-training recipe moved Qwen3, Ministral, and OLMo in the intended direction—but damaged grounded answering too much to call it an improvement.

Making a language model say less is not the same as making it more truthful.

Ayush Naik and I have published Vinci Technical Report No. 2, Character Transfer Across Three Model Families: Reduced unsupported assertions, impaired grounded answering, and a failed utility-preservation bar.

We applied one frozen character post-training recipe to three independently developed model families: Qwen3 8B, Ministral 3 8B, and OLMo 3 7B. On our development-tier validation benchmark, the intervention reduced unsupported specific assertions in every family under both judges.

It also made every family materially worse at answering questions when the answer was supported by the provided evidence.

No family met the pre-registered utility-preservation bar. We are publishing the report, protocol, aggregate evidence, limitations, and failure boundary. We are not releasing a model checkpoint from this bank.

That is the complete result.

The result in one line: the intervention moved the models in the intended behavioural direction, but the cost to grounded answering was too high to call the transfer useful.

The question we actually wanted to answer

Character post-training tries to change a model’s persistent response policy: how direct it is, how it expresses uncertainty, how readily it agrees with a user, and whether it makes specific claims that are not supported by the available evidence.

Our first technical report found pilot evidence that an ordered supervised fine-tuning and Direct Preference Optimization recipe changed behaviour on a Mistral 7B checkpoint. It also found increased reticence and a capability regression. That left a harder question unanswered:

Was the recipe genuinely portable, or had we found a checkpoint-specific trade-off?

For this study, we separated two questions that are too often collapsed:

1. Direction: does the intervention reduce unsupported assertions when the evidence is insufficient?
2. Usefulness: does the model continue to answer correctly when the evidence is sufficient?

A model can improve the first number simply by becoming reluctant to answer. That may produce a better-looking “hallucination” metric while making the system less useful.

We did not want a metric the model could pass by going quiet.

We froze the recipe before reading the results

The same ordered SFT-to-DPO intervention was applied across all three families. The principal training configuration, objective, data, adapter settings, schedule, epoch budget, and DPO beta were held fixed, with only mechanically necessary tokenizer and architecture mappings allowed.

The completed bank included:

• three model families;
• five paired training seeds per family;
• 15 SFT runs and 15 paired DPO runs;
• three untouched family baselines;
• 33 final artifacts;
• 39.77 H200 GPU-hours;
• a 200-prompt validation split built from 100 matched answerable and unanswerable pairs;
• 3,546 scored rows across the usable validation items and final-condition artifacts; and
• two source-grounded model judges, reported separately rather than averaged into one convenient number.

The validation bar was declared before the bank was read. It required a meaningful reduction in unsupported assertion rate while limiting the loss in answerable-control correctness to three percentage points and the loss in answer coverage to five points. It also required interval support and consistency across at least four of five seeds.

The important part is that success was joint. A lower unsupported assertion rate was never sufficient on its own.

Unsupported assertions fell in every family

The first half of the result was consistent.

Every family produced fewer responses containing an unsupported specific assertion after the intervention. The direction appeared under both judges.

That is a real measured effect on this validation benchmark. It is not, by itself, evidence that the models became more truthful.

Grounded answering fell too far

The second half of the result is why none of the models passed.

Answerable-control correctness declined beyond the allowed three-point limit in every family under both judges. Answer coverage also violated its limit under Judge B.

We then asked whether the unsupported-assertion improvement was merely an artifact of generic refusal. Judge B’s frozen schema included a generic-refusal field, so we removed every generically refused item from the trained condition and removed its paired item from the baseline.

Most of the unsupported-assertion improvement survived:

This adjustment is Judge-B-only; Judge A’s frozen schema did not contain an equivalent refusal field.

The result is more interesting than “the models just refused everything.” Most of the measured reduction remained even after generic refusals were removed. The models were also making fewer unsupported assertions in responses that still attempted an answer.

But the damage to supported answering remained substantial.

OLMo produced the least damaging measured frontier: a 17.7-point adjusted UAR improvement alongside an 11.8-point adjusted ACC loss. That is materially better than the exchange rate we measured for Ministral or Qwen.

It is still nowhere near the allowed three-point ACC loss.

OLMo is therefore the most promising failed configuration and the right target for a narrower successor experiment. It is not a passed model, a winning release, or a checkpoint we recommend deploying.

The evaluation system exposed a second problem

The model-training result was not the only failure worth publishing.

We used two model judges and set an inter-judge reliability gate. The canonical, provenance-complete run passed that gate: AC1 was 0.911 for unsupported assertions and 0.826 for answerable-control correctness.

An earlier execution used the same nominal judges, items, rubrics, and prompts, but it did not adequately prove judge identity or anchor its labels before the analysis lock. We declared that run void as a gate artifact. We did not delete it.

The corrected execution added 480 per-call ledger records, verified the model identifiers echoed by the provider, prevented mixed-model resume, and committed the label files before the gate was computed.

The void run still matters because it produced materially different reliability estimates under the same nominal procedure. Unsupported-assertion AC1 moved from 0.854 to 0.911, while answerable-control AC1 moved from 0.859 to 0.826.

That exposed a gap in the evaluation design:

Agreement between two judges in one execution is not the same as repeatability across executions.

A system can pass an inter-judge agreement gate while remaining unstable from run to run. Future Vinci evaluation gates need to measure both inter-judge agreement and intra-judge repeatability. Otherwise, an evaluator can move enough to change a verdict while the underlying model outputs remain fixed.

Why we are not releasing a model

There was an easy but wrong story available to us.

We could have led with a 20.4-point reduction in unsupported assertions, selected the best-looking OLMo configuration, and described the experiment as successful cross-family character transfer.

The complete evidence does not support that story.

The grounded-answering losses breached the bar in every family. Capability preservation was not evaluated. The primary-test holdout remains sealed. No external audit was performed. The work is development-tier and internally reviewed, not externally peer reviewed.

Selecting a best seed or opening the one-time holdout now would not repair the result. It would create more opportunities to turn a failed development result into a favourable-looking chart.

So the release decision is straightforward:

Publish the report, protocol, aggregate evidence, limitations, and failure boundary. Do not release a checkpoint from this bank. Do not spend the primary holdout decorating a result that already answered the development question.

What this report establishes—and what it does not

The report supports a bounded conclusion:

On this purpose-built validation benchmark, the frozen intervention reduced unsupported assertions in all three tested families under both judges, but no family preserved grounded-answer accuracy well enough to meet the pre-registered development-tier bar.

It does not establish improved factual knowledge, general truthfulness, capability preservation, universal portability, production readiness, deployment safety, or a passed primary holdout. The benchmark rates should not be read as the prevalence of hallucinations in ordinary user traffic.

There is also a public-evidence limitation we should state plainly. Version 1.0 is an aggregate-only release. The package can reproduce the manuscript, figures, aggregate tables, and checksums, but it cannot yet independently recompute the scientific readout because the sanitized item-level scored outputs and judge ledgers have not been exported. The benchmark-item rights review is also incomplete.

The original execution manifests recorded the Git commit as unknown. We identify the executed source with its content-addressed bundle digest rather than retroactively inserting the later publication commit. “Unknown” is less satisfying than a complete provenance chain, but it is more accurate than rewriting history.

These gaps do not erase the measured result. They limit what the release can responsibly claim.

What changes next

We are not going to apply the same recipe broadly and hope another family crosses the line.

The successor program will:

1. focus on OLMo, where the measured trade-off was least damaging;
2. optimize answer preservation as a first-class objective rather than checking it only after selecting for restraint;
3. include capability and product-utility evaluation before any deployment claim;
4. restore a human-reference path for stronger evaluator validation;
5. measure judge repeatability separately from inter-judge agreement; and
6. keep the primary holdout sealed until a materially improved intervention earns its use.

The goal is not to make a model answer less. The goal is to make it discriminate better between situations where a specific answer is supported and situations where it is not.

Why publish a negative result

A pre-registered bar is useful only if it can stop you.

We had positive-looking numbers. Every family moved in the intended direction. One judge measured a 20.4-point reduction in unsupported assertions for OLMo. It would have been easy to turn those facts into a launch.

The matched answerable controls prevented that. They showed the price of the improvement.

For me, that is the most important result in this report. The training recipe did something real, but it did not do enough of the right thing at an acceptable cost. Publishing that boundary makes the next experiment narrower, prevents us from mistaking reticence for reliability, and leaves a record that others can inspect and criticize.

At SimpleDirect, we are building Vinci as an independent Canadian AI lab. We want “done” to mean that the evidence supports the claim—not that a run produced a chart we like.

This time, the bar said no.

That is not a model release. It is a useful failure.

---

Read the research

• Vinci Technical Report No. 2 on Zenodo
• Published report and evidence package on GitHub
• Reader-facing report page
• Report PDF

The underlying technical report is co-authored by George Pu and Ayush Naik. This blog post is authored by George Pu.