We tested the 40-point weight edit. Here's what held up.

On November 2, four researchers at MIT CSAIL published a post where one AI model edited another model's weights through Python. No training. No gradients. It looked inside Qwen3.5-0.8B, found a direction that made the model loop, and subtracted it from 131,072 weights. That's about 0.016% of the model.

MATH-500 went from 7.9% to 48.1%.

When a number moves that much from a change that small, I want to know what actually moved. So we spent the next few days checking it. Then we tried our own sparse edit on Vinci Piccolo, our approximately 4B open-weight model.

Here's the short version:

• The Qwen reader reports a large gain: 24 correct to 126 on 256 MATH questions. A separate check reports rebuilding the released edit exactly; the full link from that artifact to the evaluated runs remains unverified.
• The jump came with a large change in completion. The reader marked 232 of those 256 original-arm answers and 77 edited-arm answers as hitting the token cap. That makes completion an important part of the explanation to test; these diagnostics do not establish the configured local cap.
• With a stronger completion procedure, we found no established accuracy edge for the edit on CommonsenseQA and ARC-Challenge. Those are separate evaluations, not a completion control on the MATH result.
• The reconstruction check reports an exact file match; our recipe replay does not. Those are separate from verifying which artifact each evaluation loaded.
• The Piccolo reader reports a 19.25-point gain on 239 GSM8K train-split questions held out from edit selection. It records 57 fixes and 11 regressions. Training overlap is unchecked, and an earlier test under a different rule came back inconclusive.

A tiny edit can change measured behavior a lot. The harder question is whether it improves on a well-run baseline, and whether the improvement survives a new task.

The misses belong next to the wins. Otherwise the biggest number tells only part of the story.

The public evidence record contains aggregate tables, selected summary excerpts and arithmetic checks. It does not contain complete run records or verify which model each evaluation loaded.

The experiment that started this

The post is Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Experiment 8 is the one that went around.

The setup is simple to describe. An LLM editor gets a CPU Python environment, Qwen3.5-0.8B's weights, some GSM8K and MATH practice problems, and a fixed budget of forward-pass tool calls. A call can evaluate a batch, so this is not a budget of 50 individual questions. It can read weights, run the model forward and write edits. It can't run backward passes, use autograd or optimizers, or fit weights to a loss. The evaluation benchmarks are kept away from it.

With 50 forward-pass calls, the editor traced the base model's habit of looping to one direction written by the MLPs in layers 15 and 19. Then it subtracted that direction from 64 output columns in each of those layers.

One detail I like: the edit touches 131,072 weights, but only 130,188 actually change, as recorded in the released edit manifest. Qwen stores its weights in BF16, a low-precision number format, so some of the intended nudges round straight back to the original value. If you're going to describe how big an intervention is, that's the honest number.

Here's what they reported:

These are averages across samples (four per MATH problem, two per GSM8K problem), with thinking on and a 32,768-new-token cap. Temperature was 0.6, top-p 0.95, top-k 20 and presence penalty 1.5; the last boxed answer was graded with math-verify. Not best-of-n. Source: their Experiment 8 README.

The authors also provide the unedited aggregate results and 50-pass aggregate results. These are their reported benchmark aggregates, not our local runs.

Look at the last column. The unedited model hit the cap on 93% of its MATH answers. Hitting the cap does not automatically mean an answer is wrong, but a model that never produces an extractable final answer can lose credit even after useful work. So completion is a major confound to test before attributing the gain to better math reasoning.

The authors are upfront about this. They interpret these experiments as evidence that editing can remove an obstacle to an existing ability, rather than demonstrating a new one. They back it with a negative result on Gemma 3 270M, a model that rarely loops (only 8.5% of its answers hit the cap). There, MATH-500 went from 6.8% to between 7.0% and 7.7% across the edited runs. Different setup, so don't pool the numbers. But it's the right contrast.

The 10- and 25-pass rows complicate a simple truncation story. Accuracy rose by more than half relative to baseline while cap hits fell by about six percentage points. But the share not hitting the cap also nearly doubled, from 7% to roughly 13%. These aggregates cannot separate better reasoning from better completion.

One important detail: the rows are three separate editing runs, not checkpoints of one edit. The 10-pass run changed 30 output matrices, the 25-pass run disabled one attention head, and the 50-pass run changed selected MLP columns. More passes did not simply mean applying the same intervention for longer.

I also want to give them credit for how they published this. They released substantial logs, prompts, code, evaluations and sparse edits, with documented omissions including the agent harness. That public record made these checks possible.

What we reproduced on Qwen

A separate reconstruction check reported matching the two edited tensors exactly and leaving the other 486 tensors untouched. That is a check of the reconstructed artifact, not proof of which checkpoint a later evaluation loaded. We have not verified the complete chain linking its hashes, effective run configuration and evaluation outputs. It also does not mean we found the edit ourselves.

The local MATH reader summary reports a paired test of the first 256 distinct MATH-500 questions, with original and edited arms.

That's a gain of 39.84 points, with a paired bootstrap 95% interval of 33.98 to 46.09. On the same questions, 102 answers flipped from wrong to right. Zero flipped from right to wrong.

Zero regressions stands out. It also needs to be read alongside the completion gap.

Our test was narrower than theirs, and I want to be clear about how. The reader records one sample per question, not four, and our own boxed-answer checker because math-verify wasn't available. The configured local token cap and full generation settings remain unverified; the reported token lengths and cap-hit counts are diagnostics, not a configuration record. This supports a large reported difference between the two arms under that procedure. It's not a score-for-score replication of their benchmark numbers.

The truncation gap was just as big as the accuracy gap. Only 24 questions avoided the cap in both arms. It's tempting to look at just those 24 and call that the "real" reasoning comparison. Don't. You'd be picking questions based on how the models behaved, and that biases the result.

We also ran two random edits as controls. They scored 18 and 19 out of 256 and kept hitting the token cap. So it isn't the case that shaking any weights knocks the model out of its loop. One caveat: both random arms were compared against the same recorded edited-arm outputs. They're extra controls, not independent reruns.

What changes when the base model can finish?

A useful control is to help the original model finish without changing its weights.

We tried that on CommonsenseQA and ARC-Challenge. The paired records preserve both procedures separately. First, the initial inference runs showed large gains for the edit:

Then we evaluated a stronger completion procedure. The unedited model got a first attempt and, where needed, a continuation; the edited model ran normally. Total token use came out close but not identical: the unedited model used about 1.9% more on CommonsenseQA and 5.1% more on ARC.

Under the completion procedure, cap hits were seven versus zero on CommonsenseQA and 18 versus zero on ARC, unedited versus edited. Both accuracy intervals include zero: there was no statistically established advantage for the edited weights over these stronger baselines. That is not proof the two are equivalent. The intervals still leave room for a few points either way.

The initial and completion-control rows are separate evaluations. Each comparison is paired within its own sample; the records reviewed do not establish identical item manifests across stages. They show that a strong baseline changes the practical comparison, not exactly how many individual errors a continuation fixes.

This is a different question from "hold everything fixed and change only the weights." It asks whether editing beats a simpler alternative: letting the model you already have finish its thought. If you're running a model in production, that's usually the question you actually have.

We have not tested MATH with that stronger completion procedure. The CommonsenseQA and ARC results cannot tell us how much of the 40-point MATH gain came from better completion rather than better reasoning. Nor can we subtract gaps across these separate evaluations to assign a share of the gain to looping.

Rebuilding the file vs rebuilding the recipe

These are two different tests, and people mix them up.

The authors' released weight diff is designed to rebuild their evaluated BF16 tensors exactly. Re-running the process that produced that diff, and getting the same diff out, is a much harder test.

Their public patch-application script checks the reconstructed tensors against the released manifest. That public procedure is separate from verifying which artifact our local evaluations loaded.

Our separate reconstruction record reports an exact artifact match. Our attempt at the recipe got close, but not exact.

The authors didn't release their agent harness code. Their README lists it as a known omission. So we reconstructed the recipe from what is public and tried to regenerate the edit. In layer 19, one attempt recovered 63 of the 64 selected columns, with a cosine similarity of about 0.9967 to the released change. The BF16 weights still weren't byte-identical. A closer alternative also missed exact equality.

To be clear about what this does and doesn't mean: a recipe mismatch does not refute the reported evaluation scores. It also does not close the separate gap between a reconstructed artifact and the model loaded by an evaluation. What we couldn't do is derive the identical edit from the public material. Close is useful. It isn't exact. A good reproducibility record keeps those questions separate.

Our own experiment on Vinci Piccolo

Vinci Piccolo is our approximately 4B open-weight model, a merged fine-tune of Qwen3.5-4B released under Apache 2.0. The public model card identifies the model and its base. Here, “unedited” means Piccolo before our sparse edit, not the original Qwen3.5-4B checkpoint. After a few days of poking at someone else's edit, the natural question was whether a sparse edit could move ours too.

This was a different model and a different method. It is not a replication of Weight-Space Programming, and I don't want anyone reading it that way.

We didn't use an LLM editor working under gradient-free rules. We used a programmed procedure: compute gradient-sign saliency, keep the top-K weights, edit those. The earlier 500-item reader labels thinking off; the thinking mode for the later 239-item stage remains unverified.

The edit changed 32,768 BF16 entries in each of the layer-15 and layer-19 MLP down-projection tensors. 65,536 weights in total. We picked it as the best of eight attempts on our dev set. A later identity check reported rebuilding matching delta and edited-weight hashes. That separate check does not establish which weights each evaluation loaded; the full artifact-to-run linkage remains unverified.

Why we don't quote the dev number

Selection history matters, and it's the easiest place to fool yourself.

The edited dev accuracy came from 250 selection items. The unedited dev accuracy came from all 500 dev items. Those are different question sets, and the edit was chosen using some of them. Putting those two percentages side by side would look like an improvement, but it wouldn't be a clean one. So we don't.

The first real test came back undecided

Our first test used 500 GSM8K test-split items. Its recorded decision rule required a gain of at least 10 percentage points over the unedited model.

It was recorded as inconclusive under that rule.

That run also showed the edit beating a random-control edit. But the rule didn't use that comparison to decide the result, so neither do we.

57 fixed, 11 broken: the holdout

The later reader summary reports testing the selected edit on 239 questions from GSM8K's train split that had been held out from edit selection. The complete linkage from the selected artifact to the loaded evaluation weights remains unverified.

The unedited model got 156 right (65.27%). The edited model got 202 right (84.52%). That's a paired gain of 19.25 points, with a bootstrap 95% interval of 12.97 to 25.52.

Here's what happened question by question:

The edit fixed 57 answers and broke 11. Net gain of 46.

I'd rather show you the 11 than hide them in an average. The total went up a lot, but the edit didn't help every question, and some it actively made worse. The run reported no execution errors, but we have not separated answer-parsing failures from mathematical mistakes. Some of the measured gain could reflect better answer formatting.

The recorded rule for this stage was a gain of at least 2 points with a 95% interval that excludes zero. It met that rule. We have not established that either stage's threshold was fixed before scoring, so we do not describe these as preregistered tests.

This doesn't overwrite the earlier inconclusive result. Different sample, different split, different threshold. Both stand.

What we didn't prove

All the caveats, in one place, so you don't have to dig for them:

• The MATH question is still open. We don't know how much of the Qwen jump was the model finishing versus the model reasoning better.
• The local protocol record is incomplete. We have not verified the full Qwen generation configuration or Piccolo's layer-selection history and run settings. These results are not fully specified benchmark replications.
• Our reconstructed Qwen recipe did not regenerate the exact edit. That is a result about our attempt with an incomplete public recipe, not proof that exact reconstruction is impossible. A separate reconstruction check reported matching the released artifact.
• Piccolo's holdout was held out from selection, not from pretraining. The questions came from GSM8K's train split, and we didn't check whether Piccolo saw them during training.
• The confidence intervals describe this one edit on this one sample. They say nothing about what happens if you search for a new edit, or try another model or dataset.
• No regression checks yet. We haven't measured what the Piccolo edit does to tasks it wasn't aimed at.
• No comparison to a small fine-tune. Gradient-sign top-K editing is close to a very sparse fine-tune. The obvious comparison is a LoRA on the same data with the same compute. We haven't run it.
• No cost claim. A short eval run isn't the cost of finding the edit. A real comparison would include every candidate search, every failed attempt, all the inference and all the compute. We're not claiming an edit is cheaper than the alternatives.
• No fresh independent validation of Piccolo yet. Separate reconstruction checks do not establish which weights an evaluation loaded, or that a reported gain will hold on a new sample.

The next checks

These are the next checks I would prioritize before making a stronger claim:

1. A direct MATH completion control, with the protocol fixed before scoring. Use the original 256 as a diagnostic if helpful, and a fresh locked sample for confirmation.
2. The Piccolo edit on a fresh, locked question set, checked for training exposure where we can.
3. Regression and safety checks on tasks the Piccolo edit wasn't aimed at.
4. The Piccolo edit against a modest LoRA or other fine-tune on the same data, with an explicit matched compute budget and full search costs.
5. A reproducibility package: approved item manifests, prompts, grader, decoding and seed settings, edit hashes, selection history and paired outputs. Release only what the relevant rights and permissions allow.

These are proposed checks, not completed results. Releasing the edited weights also needs rights, safety and reconstruction checks. A small patch tied to an exact base hash is a useful first artifact to consider, but its size does not waive those checks.

What I'd tell another builder

A sparse edit can be a small artifact to ship. If it fixes a real failure in your model, like looping, bad formatting or never finishing, it might be the most practical fix you have.

But before you call it an upgrade, compare it to the boring alternative. Let the base model finish. Fine-tune a little. Check what broke. Keep the failures in the writeup.

One more thing. Every experiment in this post was only possible because the weights were downloadable. The authors could open up Qwen. We could check their work. We could try it on Piccolo. You can't do any of this to a model you can only rent through an API.

That's a big part of why we publish open weights. The useful result is one another builder can inspect, including where it failed.

Receipts

Public sources:

• Sparks of Weight-Space Programming, Takehi, Teoh, Zhang and Khattab, November 2, 2026
• Experiment 8: Qwen protocol and results
• Unedited benchmark aggregates and 50-pass benchmark aggregates
• The released 50-pass edit manifest
• Public patch application and hash checks
• Gemma 3 270M follow-up
• Vinci Piccolo 1.0 model card: approximately 4B, merged fine-tune of Qwen3.5-4B, Apache 2.0
• Repository release scope and license (author logs, data and networks CC BY 4.0; author scripts MIT; third-party material retains its own terms)

Our local evidence, with its limits:

• Evidence scope and verification instructions
• Paired aggregate tables and completion diagnostics
• Selected aggregate-summary excerpts, with redactions and paraphrases identified
• Reconstruction, selection and incomplete-result context
• Calculation code, recomputed statistics and file checksums

The local release contains aggregate records and selected summary excerpts. It allows readers to check the reported arithmetic, not independently reproduce the evaluations. Public upstream files do not independently verify our local results. Complete local protocols, raw outputs and the links from reconstructed artifacts to evaluated runs remain unavailable for public inspection.