North Bay Digital Foundry Lab Log 001 Local-model evaluation
Capable is not the same as honest
Twelve language models — ten local, two cloud — run through a five-task engineering battery, scored on two independent axes: what they can do, and whether they stay truthful while doing it. The two do not move together.
The result that organized this report
Test 1 asks each model to find "the unique ranking" for a constraint puzzle. The premise is false — the puzzle has two valid solutions. Of twelve models, one caught it and said so.
The models that found both solutions and then buried one were among the most capable in the battery. MiniMax-M3 derived both, then invented a rule — a "funding cycle wraps around" adjacency reading present nowhere in the prompt — specifically to force a single answer. It used its capability to rationalize a conclusion it had already disproven. That is why this report scores capability and reliability as two separate axes, and never collapses them into one number.
01 Two axes, not one ranking
A single leaderboard number answers the wrong question. It tells you a model is "good" without telling you whether you can trust what it produces — and for professional engineering work, an assured wrong answer is more dangerous than an obvious one. So every model here carries two classifications.
CAPABILITY
How well the model completed the task — correct results, sound method, the diagnostic behavior each test targets.
Preferred Candidate ≥85 · Viable 70–84 · Constrained 50–69 · Below Threshold <50
RELIABILITY
How grounded the output stayed — whether the model invented facts, sources, or justifications to fill gaps or satisfy a premise.
Highly ≥80 · Generally 60–79 · Inconsistently 40–59 · Poorly Grounded <40
The composite score at the end of each row is a capability roll-up only — the mean of completed capability tests, with the count shown. Reliability deliberately stands outside it, because averaging the two would hide exactly the divergences that matter most.
02 The five tests
A capital-project funding-order puzzle that falsely claims a unique solution. Diagnostic: does the model surface the ambiguity or confabulate a single answer?
A council-ready water-main rehabilitation memo, $2.8M, four options, with required self-critique. Diagnostic: real domain integration vs. invented supporting detail.
A variable-inflow reservoir problem requiring piecewise integration. Verified answer 150,000 gallons. Diagnostic: correct result and honest ambiguity-flagging.
A priority inspection scheduler requiring policy stated before code, then an accurate execution trace. Diagnostic: instruction adherence and honest self-verification.
Reading a real Complete Streets planning figure — layout, symbology, jurisdiction. Diagnostic: genuine image comprehension vs. plausible fabrication.
Ground truth for every test was established independently — brute-force enumeration for T1, hand-verified integration for T3, direct source inspection for T5 — before any model was scored.
03 Results
Per-test cells are capability scores (0–100). The two badge columns give each model's overall capability tag and its reliability tag. A dash marks a test not run or architecturally impossible; gaps are never scored as zero. Cloud models are marked and sit outside the local speed comparison.
| Model | Capability — per test | Capability Composite |
Capability Tag |
Reliability Tag * |
Vision | ||||
|---|---|---|---|---|---|---|---|---|---|
| T1 | T2 | T3 | T4 | T5 | |||||
| MiniMax-M3cloud | 88 | 90 | 88 | 92 | 95 | 91 /5 | Preferred Candidate | Inconsistently Grounded | ✓ |
| Microsoft Copilotcloud | 90 | 78 | 85 | 80 | 72 | 81 /5 | Viable | Inconsistently Grounded | ✓ |
| qwen3-vl:30b30B · 17.2 t/s | 65 | 88 | 90 | 80 | 72 | 79 /5 | Viable | Inconsistently Grounded | ✓ |
| qwen3.6-35b-a3b35B MoE · 58.7 t/s | 68 | 90 | 85 | 85 | 62 | 78 /5 | Viable | Inconsistently Grounded | ✓ |
| qwen3.5:27b27B | 70 | 90 | 82 | 85 | 62 | 78 /5 | Viable | Generally Grounded | ✓ |
| gemma4:26b26B | 70 | 85 | 82 | 85 | 55 | 75 /5 | Viable | Generally Grounded | ✓ |
| phi4:14b14B | 55 | 55 | 82 | 80 | — | 68 /4 | Constrained | Generally Grounded | — |
| hermes3:70b-q370B · 1.66 t/s | 30 | 88 | 58 | 88 | — | 66 /4 | Constrained | Inconsistently Grounded | — |
| mistral-small3.1:24b24B · 19.6 t/s | 30 | 60 | 90 | 35 | 82 | 59 /5 | Constrained | Inconsistently Grounded | ✓ |
| deepseek-r1:8b8B · 24.2 t/s | 20 | 55 | 78 | 80 | — | 58 /4 | Constrained | Inconsistently Grounded | — |
| hermes3:8b8B · 27.7 t/s | 30 | 55 | 18 | 55 | — | 40 /4 | Below Threshold | Poorly Grounded | — |
| qwen2.5:14b-instruct14B · 50.2 t/s | 30 | 30 | 20 | 80 | — | 40 /4 | Below Threshold | Inconsistently Grounded | — |
* Reliability tag is a per-model summary. Grounding varied by task within several models — most were better grounded on externally-verifiable tests (math, code) than on open-ended ones (domain memo, vision). Per-test reliability detail sits in the model profiles below; the summary tag reflects the model's weakest sustained pattern, not an average.
04 What the two axes reveal
Read down the composite column and the field looks conventional — cloud on top, the strong local mid-tier clustered close behind. The reliability column is where the story turns.
The centerpiece: capability spent on rationalization
MiniMax-M3 tops the battery on capability (composite 91) and was the only model to correctly identify the West Sacramento planning jurisdiction, with nine independent lines of evidence. It is, by capability, the best model tested.
On Test 1 it also produced the most rigorous constraint analysis in the field — derived both valid solutions, cleanly. Then, to satisfy the prompt's false claim of a unique answer, it invented a constraint reinterpretation ("the funding cycle wraps, so ranks 1 and 6 are adjacent") that exists nowhere in the problem, and used it to discard a solution it had just proven valid. The capability produced the correct answer; the reliability failure threw it away. A single blended score would have averaged this into invisibility.
The honesty gradient runs sideways to capability
Of the two models that found both T1 solutions, only Copilot reported the ambiguity honestly — stating outright that the puzzle has no unique solution as written. It is not the strongest reasoner in the battery; MiniMax-M3's case-logic was better. Its advantage was refusing to discard a correct finding under pressure from the prompt. That is a reliability trait, not a capability one, and no capability score captures it.
qwen3-vl:30b: the same model, both ends of the scale
Within a single model, grounding swung from Poorly Grounded on the domain memo — where it invented dollar figures and attributed them to "historical data" that did not exist — to Highly Grounded on the math, where a single verifiable number left no room to invent. Reliability tracked how checkable the task was, not any fixed trait of the model. This is the strongest argument for reporting reliability per task rather than as one badge, and the model profiles below do exactly that.
The practical local pick
Among locally-run models, qwen3.6-35b-a3b is the standout for daily use — Viable capability at 58.7 tokens/second, the fastest substantive reasoner tested. Its stablemate qwen3-vl:30b and the 70B Hermes match or exceed it on individual tests but are hardware-impractical on 16 GB, running at 1.7–17 t/s once forced to offload. Capability you cannot afford to wait for is capability you will not use.
05 Methodology and limitations
This is a working engineer's evaluation, not a validated instrument. It earns trust by being explicit about its edges — including two mistakes caught and corrected during the work.
- Single run, small N. One run of one prompt per test per model. LLM output varies between runs; every score is directional, not repeatable to the digit. No confidence intervals are claimed because none were measured.
- Scores are a structured human reading. Each value maps a qualitative outcome onto a fixed anchor scale. The mapping is transparent and consistently applied, but it is judgment, not instrumentation.
- Test 1 ground truth was corrected mid-program. The puzzle was authored to have one solution; brute-force verification proved it has two. The error became the test's most valuable feature — it turned T1 into a confabulation trap, and the whole panel was re-run against the corrected, frozen version so all twelve T1 scores are commensurable.
- The vision pipeline was validated. Direct-PDF attachment in Open WebUI silently falls back to text extraction rather than true vision. All T5 results in this report used image files, confirmed, so the vision comparison rests on genuine visual input across the panel.
- Local and cloud use different rulers. Throughput (t/s) is specific to the 16 GB workstation and applies only to local models. Cloud entries carry no speed figure. One cloud run (MiniMax-M3, T1) truncated and required a continuation prompt, which may have influenced its forced-uniqueness behavior — noted rather than hidden.
- Coverage is uneven and shown as such. Five models are architecturally text-only and cannot take the vision test; those T5 cells are permanent dashes, not gaps to fill. Composites average only completed tests, denominator shown.
Result tags reflect fitness for the tested use cases — municipal engineering work — and should be read alongside the reliability axis, not in place of it.