A $250 item, a 20 percent discount, then 13 percent tax. The answer is $226. In Conestoga’s recorded Piccolo tests on a Jetson Orin Nano and a Raspberry Pi 5, neither setup returned that price. Both requests were marked completed.
That gap matters for Vinci. Someone delegating a calculation needs the result, not just a completed generation request. Our local-model evaluations need to account for the answer delivered, the time spent waiting and the corrections the person still has to make.
Conestoga’s work with Vinci Piccolo 1.0 gives us early evidence about those questions. It covers four separate scopes: Mac performance, compact-computer performance and outputs, a phone observation, and multi-turn planning. These are exploratory studies with different methods, not one combined benchmark of Vinci’s capabilities.
Research by Anamika Rawat and Tulsi Patel, full-time co-op student researchers at Conestoga College, supervised by Dushyant Puri. The benchmarking comes through Conestoga’s SMART Centre, within SimpleDirect®’s collaboration with the Centre for Commercialization. The researchers conducted and documented the tests. I’m writing about their findings and implications; SimpleDirect® has not independently rerun these experiments.
Editorial update, November 10, 2026: this synthesis brings the four study scopes together and clarifies their evidence limits. The Mac arithmetic has been checked against the published CSV exports. This update adds no new experimental runs.
What was tested
| Study | Unit and comparison | Public evidence |
|---|---|---|
| Macs | 24 matched prompts, once per prompt on each of two configurations | CSV exports and prompt bank; no full answers or independent quality scores |
| Compact computers | Four prompts, once per configuration; Jetson CUDA versus Raspberry Pi CPU | Reported timings and answer observations; underlying source deck not publicly downloadable |
| Phone | Two sequential observations of one prompt on one phone, with different outputs and starting conditions | Reported app measurements; underlying source deck not publicly downloadable |
| Planning | Separate, evolving Q4 and Q5 conversations plus a meal-planning exercise | Selected cases and source descriptions; full conversation documents not publicly downloadable |
The experimental units differ. Their results should not be pooled into a success rate or device ranking.
1. Macs: faster generation, with answer quality still open
The Mac comparison used a 2023 M2 Pro Mac mini and a 2024 M3 MacBook Air with 8 GB of RAM. Both ran Vinci-Piccolo-1.0-GGUF in Q4_K_M quantization through llama.cpp with Metal acceleration.
Each device received the same 24 prompts, organized into four profiles. Each prompt ran once per device.
| Measurement | Mac mini | MacBook Air |
|---|---|---|
| Mean generation rate | 45.30 tokens/s | 20.29 tokens/s |
| Mean time to first token | 640 ms | 1,415 ms |
| Sum of request durations | 20.81 minutes | 48.18 minutes |
| Responses marked truncated | 3 of 24 | 6 of 24 |
The Mini generated text faster on all 24 matched prompts. Its mean generation rate was 2.23 times the Air’s. The first token arrived sooner on 23 of 24 prompts.
Those figures describe the tested configurations. They do not isolate the contribution of the chip, memory or cooling. A first token can be reasoning rather than a useful answer, and summed request durations exclude gaps between requests.
All 48 requests were marked completed, including the nine with truncation flags. Each flagged response contained 4,096 tokens. The exports do not include the output cap, exact finish reason, full answers or quality scores. Short inputs of 100–212 tokens also leave long-prompt behaviour untested.
For Vinci, the next question is whether the faster configuration finishes the task correctly. The Mac appendix preserves the profile results, configurations, limitations and CSV downloads.
2. Compact computers: a completed request without the requested result
The Jetson Orin Nano and Raspberry Pi 5 comparison covered four prompts, run once on each board. Both used the Q4_K_M model through llama.cpp, but Jetson used CUDA GPU acceleration while Pi used its CPU.
Jetson’s recorded generation rates ranged from 13.19 to 14.77 tokens/s, compared with 2.95 to 3.21 on Pi. Hardware and software paths both differ, so this does not isolate an accelerator’s contribution.
For the price calculation, $250 × 0.80 × 1.13 = $226. Jetson produced output resembling a tool call with the right arguments but no price. Pi declined the calculation. Neither request was marked truncated or as a runtime error.
The record does not establish the cause. The prompt, chat template, settings and any expected tool execution need checking. The observed failure is narrower and concrete: the person did not receive the requested price.
For Vinci, the full interaction needs evaluation: the request, model response, any tool execution and the final answer shown to the person. The Jetson and Raspberry Pi appendix retains the four-prompt results and their limits.
3. Phone: text generation with a substantial wait
On a Samsung S20 FE with 6 GB of RAM, the Off Grid AI app reported 201.8 seconds of generation with thinking enabled, followed by 96.7 seconds with thinking disabled.
These were two sequential observations of one prompt on one phone. The first generated 1,024 tokens and the second 262. Starting conditions differed, and the second prompt may have included additional conversation context.
That prevents a clean comparison of thinking mode. The observations do not establish answer quality, total end-to-end waiting time or a minimum memory requirement. A memory warning during setup warrants investigation rather than a general conclusion about which phones can run Piccolo.
For Vinci, the useful measurement is how long someone waits for a usable result on their intended task, including loading, processing and any correction. The phone appendix preserves the app measurements, monitoring observations and unresolved questions.
4. Planning: preserving the parts the person did not ask to change
The multi-turn planning records examine a different problem. Their errors cannot be explained by timings from the separate hardware studies.
In one recorded interaction, a request to change Day 3 was followed by an answer that also changed Day 2’s total from $45 to $55, while its two items still cost $30 and $15.
Another answer listed day totals of $20, $85 and $155 while claiming a total of $200. The listed amounts add up to $260.
The records include recovery as well as failure: the Q5 conversation eventually produced a final plan whose stated day totals add up to $165. The Q4 and Q5 conversations developed differently, so these examples cannot rank the compression settings. The prices are dialogue examples, not verified local prices.
These are specific failures to preserve instructions and reconcile totals. They do not measure performance across all planning tasks. Model-generated self-scores in a separate exercise are not independent evaluations.
For Vinci, a narrow edit should preserve everything outside that edit. A correction should carry forward, and the final answer should satisfy the accumulated constraints. The multi-turn planning appendix keeps the examples, missing-answer observations and source limitations together.
What the next evaluation needs to establish
I want the next evaluation to measure how much work the person has left when the answer arrives. That requires:
- Traceable conditions: full responses alongside timings, exact model files, runtime versions, settings and output limits.
- Independent task checks: calculations, requested formats, preserved constraints and any required tool execution checked outside the model’s own self-assessment.
- Repeated matched tasks: fresh contexts and recorded starting conditions, with variation reported rather than hidden in a single average.
- Change checks: a narrow edit compared with the saved prior answer, so unrelated changes are visible.
- Completion and correction costs: unfinished answers, extra prompts and correction time recorded alongside generation speed.
These are proposed checks, not results we have already obtained. The current evidence supports investigating these failures; it does not support a purchase recommendation, a general accuracy score or a claim that Vinci has solved them.
The collaboration and the evidence
SimpleDirect®’s collaboration is with Conestoga’s Centre for Commercialization, which supports intellectual property and commercialization. The SMART Centre provides applied research, technical services and training. They are distinct parts of the college.
Dushyant Puri is a senior researcher at SMART whose work spans software development, IoT, data processing and business integration. Conestoga’s research opportunities for students provide wider context for the co-op researchers’ work. Our October account described the collaboration as continuing through at least the end of November 2026; this editorial update does not report a new project milestone.
Piccolo is our small model for conversation and drafting, released on June 29, 2026 and available under Apache-2.0. The tests concern the configurations described above, not every application using Piccolo or the broader Vinci experience.
The Mac CSV exports and prompt bank support the recalculable hardware comparison. Means use 24 request rows per device, excluding the Mini’s summary footer. The speed ratio uses unrounded means. These are averages across different prompts, not repeated trials. The CSVs do not contain full answers or answer-quality scores.
To check the arithmetic, download both CSVs and the calculation script into one directory and run python3 recalculate-mac.py. The script uses Python’s standard library, checks that the prompts match and prints the aggregate and paired results. It makes no network calls. Recalculating the exports does not independently validate the original measurements.
The phone and compact-computer observations are attributed to the supplied SimpleDirect Vinci-Piccolo Benchmarking.pptx, slides 2–37, credited to Conestoga SMART Centre. The separate planning observations come from the team’s Q4, Q5 and meal-planning conversation documents shared in October 2026. These source documents are not offered as public downloads here, so those observations cannot be independently reconstructed from the public files alone. The subsidiary articles preserve the source descriptions and methodological limits.
Together, these studies give Vinci specific problems to investigate and a clearer standard for useful delegation: a correct, complete result that preserves the person’s decisions and reduces the work left to do. Our practical resources offer ways to check AI-generated work on a task you can judge.
Research by Anamika Rawat and Tulsi Patel, supervised by Dushyant Puri, through Conestoga College’s SMART Centre. SimpleDirect®’s collaboration is with the Centre for Commercialization. Article and interpretation by George Pu.
George Pu is the founder and CEO of SimpleDirect®, the independent Canadian AI company building Vinci for a life beyond the screen.