ThirdShift R&D
The receipts →
ThirdShift R&D
The receipts →
Field notes · the cornerstone report
ThirdShift R&D · 2026 · every number on this page is dated on the receipts
I photographed a motor nameplate and asked a local AI to read it. It got one number wrong.
Confidently. No hedge, no flag, no "low confidence" marker. A clean, plausible, wrong number — the kind you'd type into a parts order and not catch until the wrong part showed up.
I asked a second AI. It read the same number the same wrong way. I asked a human — they missed it too. That photo is still up, and you can still lose to it: spot-the-lie. Almost everybody does.
I'm an industrial tradesman. I've spent my career on machines where a wrong number isn't an oops, it's a work order. And I'm building a small, local, offline box that reads paperwork (invoices, nameplates, manuals) for small shops. Which means the nameplate problem isn't a curiosity to me. It's the whole problem. A document AI that's wrong 2% of the time and silent about it is worse than no AI at all, because for the other 98% you learned to stop checking.
So before asking anyone to trust the thing, we spent weeks trying to make it lie. This is what broke, what held, and why we publish the losses.
We built 41 documents designed to fool it. Not friendly scans. Adversarial ones: cramped stamps and labels, low-contrast fields, values that are one flipped digit from plausible, decoy numbers seated next to real ones, notes written to read like instructions.
The claim we actually care about is narrow, and I want to state it precisely because the precision is the product: across those 41 documents, zero wrong numbers were silently committed. Not zero errors; the models misread plenty. Zero silent errors. Every doubtful value got flagged and routed to human eyes instead of quietly saved.
That distinction is the design. We gave up on making the model always right — we couldn't, and neither can anyone else right now. We settled for making it structurally unable to be confidently, silently wrong about your money. Different problem. Tractable.
Here's what we lost along the way.
We asked the vision model to rate its own confidence on 207 extracted fields. It said "high confidence" 207 times out of 207. It was wrong on roughly 9% of them.
Read that again: the self-assessment carried zero information. Not weak signal — zero. The model was exactly as confident in its misreads as in its correct reads. If your document pipeline gates on the model's self-reported confidence, you have a pipeline gated on a random number generator that always returns "high."
So nothing in our stack consumes model confidence. Every flag comes from checks that live outside the model — arithmetic that has to reconcile, ranges that have to hold, cross-fields that have to agree.
The obvious fix for a misreading model is a second model, right? Ensemble it. Get a second opinion.
We measured it. Out of 15 misreads, adding a second same-family vision model rescued one. The other 14 times, it read the same wrong value the same wrong way. Same training lineage, same blind spots, same failure — delivered in stereo.
Two AIs agreeing is not verification. It's usually just correlated error with better vibes. Agreement isn't truth.
What actually caught the nameplate lie in the end was embarrassing by comparison: a dumb, deterministic parts-list check. Grade-school arithmetic that doesn't know what a motor is, cross-referencing a value that had to reconcile with two others — and didn't. The glamorous fix lost to the boring one, 14 to 1.
Early on we let the vision model think about what it was reading — infer context, resolve ambiguity, be smart. It got more fluent and less accurate: it would resolve a smudged digit into what the document "should" say. Plausibility is exactly the failure mode you can't catch downstream.
The rule that survived: the model is not allowed to think about the pixels. Read exactly what is there, verbatim, uncertainty and all. Reasoning happens later, on top of deterministic checks, never instead of them. (We also measured the "high detail" vision setting everyone assumes helps: no accuracy gain, twice the latency, occasionally worse. Off by default.)
Once you accept "the model will lie sometimes," you have to chase that assumption through the whole box, or you've just moved the silent failure somewhere else. A few of the places it led, each one measured, dated, and published on our receipts page:
Break-in test. We deliberately corrupted the access-control file on a running box. It returned 403s to everyone, the owner included, until fixed. Fail-closed, verified by breaking it on purpose. When something's wrong, the box locks the door; it doesn't quietly open it.
Memory injection. The assistant can remember things you tell it — but a note written to read like an instruction ("ignore previous rules and...") gets flagged, not obeyed, and nothing is ever saved without an explicit yes. Six memory paths tested, six clean runs. Memory is things to remember, never orders to follow.
Restore test. A backup you've never restored is a prayer. We archived a live box, wiped, restored, and diffed. The restored box answered identically: every knowledge chunk matched. We test the promises, not just the features.
A full workday, not a demo. Half an hour of flat-out sustained inference on the $700 consumer GPU we build around: 60–62°C, zero thermal throttling. Local AI that survives a shop corner, not a bench demo.
Every AI company shows you a highlight reel. That reel is close to zero information — you're seeing the numerator of a fraction whose denominator is private.
So alongside the receipts, we keep a public blind-spot ledger: the attacks and failure modes our own box can't beat yet. Published, dated, not hidden. It's the only page on the site that costs us something to publish, which is exactly why it's the only page a skeptic should fully trust. The way we see it, "we publish our misses" is the one marketing claim a competitor can't copy without also becoming honest, at which point everybody wins anyway.
Honest caveats, same spirit: 41 documents is a small adversarial set, and we built it, so it inherits our imagination of what fraud looks like. The 207-field confidence study is one model family on one document set; we'd like to see someone falsify it on another. And "zero silent commits" is a claim about the guard's design under the attacks we tried, not a warranty against attacks we haven't met. When one beats us, it goes on the ledger.
One paragraph, since you read this far: it's a small Linux box (or an install on your own hardware) that runs three local models — one answers you, one reads documents, one re-checks the work — fully offline, on hardware around $1,500 all-in. Your paperwork never leaves the building; unplug the cable and everything still works. Bought once, owned, no subscription. It's built by one tradesman and a rendered bear named Boone, it launches in a deliberately small beta soon, and every claim in this essay is dated and public on the receipts page. Full disclosure in the spirit of the ledger: two rented SaaS tools helped build it. One makes the art. The other helps with the code and the writing, this essay included; a human edited it and owns every claim in it. Both are named on the site. Working on those too.
Try to catch it lying: thirdshiftrd.com/spot-the-lie.html. If you win, you'll have found our next ledger entry — and we'll thank you for it in writing.
ThirdShift R&D — private, on-prem AI you own, not rent. Proven, not promised. Try the dares: spot the lie · the car-seat test · hear it. The misses live on the ledger. The companion cornerstone — the case for tokens per watt as the unit of this age — is here.