Skip to main content

Vinci research

3 technical reports

What holds up when AI is tested?

See what changed, what failed, and how we tested it.

Start with the findingsExplore released models
Vero examines a layered ceramic sphere through a magnifying lens on a sunlit terrace.
Illustration.

What the studies found

  1. Vinci Technical Report No. 3Does passing a check mean an answer is right?
  2. Vinci Technical Report No. 2Does the same training help different AI models?
  3. Vinci Technical Report No. 1Can training reduce unsupported claims?

Technical reports

Findings, written up to be checked

Numbered reports, methods, limitations, and a citable PDF.

Page presentation updated:

Added direct paths from a research question to its finding, limits, and source report. Original evidence, model records, measurements, and publication principles remain accessible.

The original report titles, publication dates, report text and English source language are unchanged. Full reports and downloads remain accessible below. This page update does not represent a new evaluation.

Vinci Technical Report No. 3

Originally published
Version
1.0
Authors
George Pu

Released technical report · not peer reviewed

Does passing a check mean an answer is right?

Runtime Pass Is Not Correctness

Why it matters A passed test can look reassuring. It only helps if the test can tell a correct answer from a wrong one.

What we learned
Our training experiment missed its targets. An audit also found that the original code-checking test accepted deliberately wrong shortcut programs. We withdrew or paused the claims that depended on that test.
What this does not establish
The replacement test had not yet been shown to catch the required mistakes. This work produced no model for release and does not establish reliable checking for other tasks.
Read the full studyDownload PDF

Vinci Technical Report No. 2

Originally published
Version
1.0
Authors
George Pu, Ayush Naik

Released technical report · not peer reviewed

Does the same training help different AI models?

Character Transfer Across Three Model Families

Why it matters Training that helps one model may not help another. We also need to check whether it makes useful answers worse.

What we learned
In tests used to develop the approach, the same training reduced unsupported claims across three types of AI models. But it also reduced correct answers to questions the models should be able to answer. None met the study’s acceptance criteria.
What this does not establish
The answers were judged by two other AI models. The separate final test set was not used; broader abilities were not tested, and no trained version was recommended for release.
Read the full studyDownload PDF

Vinci Technical Report No. 1

Originally published
Version
1.0
Authors
George Pu, Ayush Naik

Released technical report · not peer reviewed

Can training reduce unsupported claims?

Transferring Character Post-Training to Mistral 7B

Why it matters An AI answer can sound confident without evidence. Fewer unsupported claims would help only if the model remains useful.

What we learned
One experimental model made fewer unsupported claims on deliberately challenging prompts. It mostly made fewer specific assertions; it did not show improved factual knowledge. Math performance fell, and refusals on ordinary requests increased.
What this does not establish
Another AI model judged the claims. This result is limited to one starting model and its trained versions, on the measured tests; it is not a general reliability score or a recommendation to use the model in production.
Read the full studyDownload PDF

Models for specific kinds of work

Explore each release’s weights, licence, and evaluation record.

Explore models

Vinci MLE 1.0

Research models for machine-learning engineering. For version 1.0, a valid evaluation on MLE tasks kept out of training was not completed.

Valid held-out ML-engineering and external MLE-bench evaluations were not completed for 1.0. General-capability comparisons with each model’s own parent document measured gains and tradeoffs, but do not establish performance on new ML-engineering tasks.

Compare Vinci MLE engineering modelsRelease announcement

Vinci Cyber 1.0

Defensive cybersecurity specialist models. Licences and evaluation evidence differ by size; consult each model’s record.

Current model cards report a paired infrastructure-as-code evaluation across all three sizes and their own parent models. Results depend on the test set, runtime and format, and include failed repairs and unwanted changes. Scanner clearance alone does not establish a valid repair; syntax, security properties and operational impact still need independent checks.

Compare Vinci Cyber cybersecurity modelsRelease announcement
Earlier model research: Piccolo, Bozza, and Prova
Browse on Hugging Face

A closer look at the measurements

We tested whether character post-training transferred to Mistral 7B. It did — with trade-offs.

It transferred to Mistral — development-set model-judged fabrication fell from 53.8% to 8.6% — but it cost 5.6 points of GSM8K, and the model now sometimes refuses ordinary work.

Read the article
Development-set model-judged fabrication; matched internal-harness GSM8K comparisons. Released records disagree on the GSM8K test extent, so the scores are not leaderboard-comparable. These are different evaluation populations, not general real-world hallucination rates.
Measured result
Measured resultBase modelAfter trainingChange
Judged fabricationlower is better53.8%8.6%-45.2 percentage points
GSM8Khigher is better51.6%46.0%-5.6 percentage points

The record

All research and release records

Newest first, including findings that failed or were disqualified.

Vinci ProvaResearch report

We tested whether character post-training transferred to Mistral 7B. It did — with trade-offs.

It transferred to Mistral — development-set model-judged fabrication fell from 53.8% to 8.6% — but it cost 5.6 points of GSM8K, and the model now sometimes refuses ordinary work.

Read the article
Technical report (HTML)Technical report (PDF)Merged weightsGGUF builds (f16, Q4_K_M, Q5_K_M, Q8_0)

How we publish

The standard we hold ourselves to

  • The question comes before the headline.
  • Success and failure criteria are set against a named baseline, not chosen after the numbers land.
  • Gains are reported next to their costs.
  • Measurements that failed or were disqualified stay in the record, marked unusable, with the reason.
  • Limitations are stated, not implied.
  • Provenance, versions, and pinned revisions are preserved.
  • Artifacts are released where it is responsible to, with the license and access state named.
  • When a finding changes a mainline model, the record says what changed.

Research writing

More from the research notebook

Research

Our 4B model kept changing a plan it was asked to fix

Conestoga’s Piccolo conversations exposed budget errors, unwanted edits and missing answers. What to check before relying on a small local AI model.

By George Pu
Newsroom

What Conestoga’s Piccolo tests can tell us about local AI

Meet the Conestoga researchers testing Piccolo on Macs, small computers and a phone, and see what the early results mean for choosing a local AI setup.

By George Pu
Research

Piccolo on a 6 GB Android phone and the wait for an answer

Conestoga tested Piccolo on a 6 GB Samsung phone. See the generation times, the memory warning and what to check before relying on local AI.

By George Pu
Research

Piccolo on Jetson and Raspberry Pi

Four Piccolo prompts ran faster on Jetson than Raspberry Pi, but neither setup returned a requested $226 price. Read the configurations, results and limits.

By George Pu
Research

Piccolo on Mac mini and MacBook Air

Conestoga’s 24-prompt Piccolo comparison found faster generation on Mac mini and nine truncated responses across both Macs. Here’s what to check before buying.

By George Pu
Why Vinci started with post-training
Research

Why Vinci started with post-training

We want to pre-train our own 4B and 10B models. George shares why that matters for Canadian AI, what the economics look like, and why post-training came first.

By George Pu
Introducing Vinci MLE 123B 1.0
Research

Introducing Vinci MLE 123B 1.0

Our largest MLE model is now available with BF16 and GGUF weights. Built on Devstral 2, it extends Vinci’s work on models for machine learning engineering.

By George Pu
We tried a new AI architecture. It didn't work.
Research

We tried a new AI architecture. It didn't work.

Our first native architecture experiment fell short on basic language modelling. We stopped it, kept the useful engineering, and changed how the next idea will earn more compute.

By George Pu
Introducing Vinci Cyber: Open-Weight Models for Defensive Cybersecurity
Newsroom

Introducing Vinci Cyber: Open-Weight Models for Defensive Cybersecurity

Vinci released Vinci Cyber, a family of three open-weight models built for defensive cybersecurity: 8B, 30B, and 123B parameters.

By George Pu
Research

We tested one frozen character recipe across three model families. None met the bar.

We applied one frozen character post-training recipe to Qwen3, Ministral, and OLMo. Unsupported assertions fell in every family, but grounded-answer accuracy fell too far in every family. We are publishing the failure, not a model checkpoint.

By George Pu
Research

Beyond fine-tuning: what Vinci is becoming

Fine-tuning was enough to change our models' voice. It was not enough to build the models we now want to release. We are moving from light behaviour tuning to full-weight model redevelopment - open foundations, rebuilt by Vinci. Here is the plan, stage by stage.

By George Pu

Have a question worth exploring?

Work with us