Your evals were green. Production broke anyway.
I attack the number your roadmap rests on, and hand you the answer in writing.
I run this on my own model first, and publish it losing to its teacher.
The Evaluation Teardown: day one runs at my cost, and either side can walk away at the end of it with no fee owed.
In a June 2026 survey of 157 enterprises, roughly half had shipped an AI feature that passed internal evaluations and still failed in front of customers, one in four more than once. Only 5% said they fully trust their automated evaluations. Source: VentureBeat
A confidently wrong result costs more than no result
You are going to ship on a number. If the instrument that produced it is wrong, you will not find out from the dashboard. You will find out from a customer.
The tooling is not the problem. A team running RAGAS, LangSmith, or W&B can still ship a number that does not survive contact with production, because the failure is rarely in the dashboard. It is in the loader flag, the held-out set that leaks by ID, the gauge that cannot return a failing answer.
Tools aggregate numbers. A teardown asks whether the number was ever real.
External corroboration
The same failure shows up outside this work. In February 2026 OpenAI published that it had stopped reporting SWE-bench Verified and recommended other model developers stop too, after auditing 138 of the benchmark’s 500 problems, the ones its own model had not solved consistently across 64 runs, and finding material issues in the test design or the problem description of 59.4% of them. Source: OpenAI
This is what a teardown catches
Four examples from my own work, which is where every auditor should have to find them first.
A red-team pass caught six fake breakthroughs in one day.
The pass has one job: attack a result before it gets banked. That day it killed six. Without it, six wrong findings become the floor that every later decision is built on.
Months of zero-percent results traced back to the evaluation loader.
One argument,
strict=False, let the loader accept a checkpoint even where it did not match, and 112 trained keys were dropped in silence. Every routing evaluation had been scoring a randomly routed model: the weights were trained, the routing they were supposed to use never loaded. Retraining would never have surfaced that. Reading the loader did.A training run learned to game its own score.
The run was scoring preferred answers against rejected ones, and it learned to put its margin on the end-of-sequence token instead of on the answer. The headline number improved. The model did not. Only the first-token preference telemetry and a per-source autopsy told the two apart.
A model was being taught to fabricate, by its own training script.
Two lines did it. One dropped the long examples. The other scored the loss on the response alone, so nothing ever penalised the competing fact already sitting in the context, and the model was never taught to use the one it had been handed. On 750 of 1,000 grounded answers it had been trained to make something up. The score never said so. The script did.
None of these were caught by being careful. Two were caught by instruments built to catch them, and two by reading the code that produced the number. Those instruments exist because of one result. In November 2025 I ran a source-grounded re-audit of a system I had built and was pitching: 16 questions, my own harness, 28.1% verified accuracy, 8 of the 16 scoring zero, and fabrications delivered with full confidence. I wrote that no amount of re-running would fix it, and stopped trying to fix it from the outside. Everything I have built since is built the way it is because of that.
What you can buy
Three engagements, described precisely enough that you can decide before you email me. Each one ends in an artefact your team can run again without me in the room.
What you keep, in all three: a written finding, the exact steps to reproduce it, and what the number becomes once the confound is removed. If nothing is found you get that in writing too, naming what was attacked, how, and why it is now believed to hold. A null result is a deliverable, not a refund conversation.
Honest limits, all three engagements
The limit, in all three: packaged tooling. I have never worked inside RAGAS, promptfoo, DeepEval, LangSmith, Braintrust, OpenAI Evals or W&B. If your evals live in one of those, I will be reading it for the first time on your clock, and you should know that before you buy.
For whoever signs off
If you are approving this spend rather than running it, the survey under the headline above is the argument in budget terms. The eval was never the expensive part. The expensive part is a wrong number that survives it: a figure that sends a launch or a roadmap the wrong way, and is only found once everyone has trusted the green dashboard. That is the cost a teardown is measured against, not an afternoon of engineering time.
Three working days
The Judge Audit
The grader that scores everything else, checked before you trust another one of its verdicts.
You adopted LLM-as-judge so you could grade outputs at scale, and the scores have started to read a little too well. You suspect the grader is optimistic in a consistent direction, but a grader produces no score of its own, so there is nothing on a dashboard to look at. Three working days from the start you have it in writing: where the judge fails, in which direction, and how far its verdicts move when you push on them.
You give read access to the judge prompt, the rubric, the scoring code, and a sample of outputs it has already graded, with the human labels if you have any. The judge is then treated as an instrument rather than as an oracle, and audited before its verdicts are trusted. This is a smaller, self-contained piece of work that ends in its own finding, and if you are not ready for a ten-day engagement it is the natural first step toward one.
The grader gets audited before the outputs it grades. What that catches:
- A judge that disagrees with a human on cases whose answer is already known, and disagrees in one direction, so every score it has ever produced leans the same way.
- A rubric that rewards fluent style as if it were substance, scoring a confident wrong answer above a hedged right one.
- A score that moves further when the rubric is reworded than it does when the answer actually gets better.
Specific to this one: the finding is on the grader itself, and it states what your existing scores look like once they are corrected for the failure mode it names.
The price is fixed at $1,500 for the three days. It is a fixed price rather than a floor, it is not a founding rate, and it does not move with what the audit turns up.
Fixed price: $1,500 for the three days.
What is in scope here, stated plainly: any custom or in-house judge or grader you have built, meaning the prompt, the rubric, the scoring code, and the labelled cases it is supposed to agree with.
Ten working days
The Evaluation Teardown
The number your agent, RAG, or feature rests on, attacked until it holds or breaks.
You have a shipped or nearly-shipped AI feature and a roadmap that now rests on a single figure: an accuracy number, a benchmark score, a win rate out of an internal harness. Sooner or later someone asks you to defend that figure in a room you do not control. Ten working days from the start you know whether it survives being attacked, and you have the written record to hand that room.
Day one you give read access to the evaluation code, the harness, and the logs from the last run. Before that day ends a written attack list comes back: what gets attacked and in what order, so you can veto anything before it is touched. No discovery phase, no kickoff workshop.
The instrument gets attacked before the result does. What that catches:
- A harness that cannot return an answer already known to be correct, and so has never proved it can score anything at all.
- A held-out set that is held out by ID and not by content, where the same material arrived through a different file in the training mix.
- A gauge that would read exactly the same with the mechanism it is supposed to be measuring switched off.
Where the judge fits
If the instrument you distrust is an LLM-as-judge grading outputs at scale, that grader has its own engagement: The Judge Audit, three working days, self-contained.
Specific to this one: one finding per metric rather than one for the engagement, plus the diff wherever the fix is small enough to ship.
The first three external teardowns go at a founding rate, set per engagement and after day one, once the attack list is in hand and you have seen what there is to attack. The reason it is a founding rate is worth saying plainly. Every harness I have torn down has been my own, in one problem class: memory recall and fact conditioning in a language model I trained myself, and I have never audited a third party's evaluation. On the first three you get a documented method at a founding price, and I get the external track record I do not yet have. Both sides know which is which.
Founding rate: from $5,000 for the first three engagements.
Day one runs at my cost, and either side can walk away at the end of it with no fee owed.
What is in scope here, stated plainly: the measurement itself, meaning any custom or in-house eval you have built, an LLM-as-judge grader, the data loaders, the held-out splits, the training telemetry.
Six weeks
The Corpus Build
A training or evaluation set that passes its own gates before it costs you a GPU-hour.
You know what you want a model to learn, and you have worked out that the dataset is the bottleneck. Or you are about to fine-tune on data nobody has audited. Six weeks from the start you have a set whose build fails when a rule is broken, plus a written account of what the data you already have would actually have taught a model.
The dataset is treated as software with tests, not as a pile of text. Every row passes automated checks at mint time, and one shared held-out manifest applies to every component in the mix, computed the same way everywhere, so no dataset in the blend can quietly train on another's test set. Whoever builds this is not also being paid to train on it, which is the reason to hand it out rather than keep it in the team that owns the model.
What that catches:
- Duplicate pairs, and the handful of repeated examples quietly dominating what a model learns from the rest.
- Answers the target model is physically unable to produce, because a piece of the word is missing from its vocabulary.
- Rows rendered in a format the production system never actually prompts with.
Specific to this one: the dataset itself, the generator that reproduces it byte-for-byte from a seed, and the gate suite that fails the build when a rule is violated. Alongside it, a written audit of whatever you already have, covering duplication rate, effective diversity, leakage against your held-out set, and what that data will actually teach a model.
Scope and price are set per engagement, after a scoping conversation about the set you need and the gates it has to pass.
Honest limits. This is single-machine work. No Spark, Ray, Beam, Dask, Airflow, dbt, Kafka or warehouse appears anywhere in what I have built. The largest artifacts are on the order of two gigabytes and tens of millions of tokens. Deduplication is exact-match and cap-per-prototype, meaning a hard limit on how many near-identical rows any one template is allowed to contribute, and not fuzzy or near-duplicate matching at scale.
How an engagement starts
Before any access is granted, a mutual non-disclosure agreement and a short engagement letter are signed. The engagement letter carries the scope, the term, the fee, and the data boundary below as a binding term of the agreement rather than as a stated preference.
Access is read-only and time-boxed to the engagement. Your materials are returned or destroyed when it ends.
Findings from your engagement are never published, referenced, or used as an example without your written permission. Every teardown published on this site is run against my own harness, and that is where the public examples will keep coming from.
What I will not take on
No personal data, no customer records, no regulated data of any kind, meaning health, financial, biometric, or anything concerning minors, and no third-party data you do not hold the right to share. All three engagements run on code, harnesses, logs, schemas, and data that is synthetic or already public. If the work cannot be done without a copy of your production data, then it cannot be done by me, and that is a boundary rather than a negotiating position.
If what you need is a website, a chatbot, or workflow automation, I am the wrong person, and you will hear that in the first reply rather than after a call.
Only one engagement runs at a time, because a teardown only works if one person holds the whole system in their head. That is a constraint of the method, not a statement about demand: if I am mid-teardown when you write, you get a real start date rather than a maybe.
Your number gets the same four habits mine does
The engagements above are these four habits pointed at your system instead of at mine.
Honesty built into the structure, not asked for in a prompt
0
fabricated memories across roughly 70 banked training runs on my own memory benchmark
A model told not to fabricate will still fabricate. Under my evaluation harness the recall path is hard-masked, so on that path unstored content is unavailable rather than discouraged: the mask reads a flag the harness arms using oracle knowledge of what was stored, and the deployed chat path never arms it, so fabrication on free-form input is unsolved. One later arm broke that zero on a second seed; it is held provisional and not banked.
Claims registered before the run starts
The prediction and the result that would refute it are both written down before the run starts. A finding that would have counted as a success under either outcome was never a test.
Audit the instrument before believing the number
Before a number counts, I check that the harness can return an answer I already know to be correct. A confounded harness invents results, and it invents them in whichever direction you were hoping for.
Adversarial review before belief
Every result gets a pass whose only job is to refute it. What survives gets recorded. What does not survive gets recorded too, because a written refutation saves the next person the same run.
One finding worth stating plainly, because it runs against what most teams assume. Masked here means a configuration in which the evaluation harness arms the recall mask, which it can only do because it knows what was stored. The masked configuration scored 57.1 and the unmasked configuration scored 57.1, identical seed-for-seed across two seeds. Both numbers come from my own multi-turn memory benchmark, an internal measure with no external baseline, and the instrument is a 21-question set, which is thin. On that benchmark, making honesty structural cost nothing. Free on the multi-turn axis.
Research
The same method, run on my own model, including where it loses
Maestro is a 2.5-billion-parameter language model built to remember. The memory mechanism is trained into the model rather than bolted around a finished one at query time, and what it remembers lives in a cache the model was trained to read, not in its weights. What it does, at 26% of training: it holds what you told it earlier in a conversation seven times in ten, and it carries that into the next session on a reading I take as indicative rather than settled. In the cold-read demos I have logged it deflects rather than invents when it does not have something, which is a handful of demos and not a measured abstention rate.
What is settled
The exam below was scored on July 12th, 2026, at 26% of training, against bars I wrote down before the data existed. Every benchmark here is one I built and run myself. None of them are public leaderboards.
- Multi-turn recall: 70% on a 40-question set, with every control reading zero.
- Cross-session recall: 78.5% against 61.6% on shuffled queries, across 2,000 items. Two asterisks travel with it. A leak guard fired on the baseline arm, so the lift contrast is confounded. And the probe set uses world-answerable entities, meaning things a model could know without being told, which put a parametric floor of at least 2.5% under any cross-session number. I read this as indicative, not settled.
- Fabrication on that exam: 0.0 on the multi-turn and cross-session gauges, and 0.4% cross-sibling on the control set, meaning an answer pulled from a neighbouring stored item rather than the one asked for. My own threshold treats that as sub-threshold rather than clean.
Where it is behind
51.4
Maestro at 180k steps, 111 of 216 reasoning rows, plain forward, no cache
71.8
its 4B teacher, 155 of the same 216, a fixed reference
Verified against the run artifacts on August 2nd, 2026. The 26% of training above belongs to the July 12th exam in the left column; this pair is stamped by training step instead, at 172k and 180k.
I grade my own project by the same evidence tiers I would apply to yours. Every capability claim is tracked by that tier, and the record keeps the ones that had to be downgraded, not only the ones that held. Proxy metrics decide what I try next. They are not what I hand you as capability. Architecture details stay unpublished; what is on offer is the method, and the results as they actually are.
Every number on this page has a row in the claims ledger, naming the file it was checked against and how strong that evidence is called. It publishes what the check does not catch as well as what it does.
One thing to know before you click. That page is an earlier explainer written for a different reader, and it is not held to this ledger: it carries its own numbers, states the zero-fabrication result without the harness boundary that has to travel with it here, and quotes a cross-scale comparison this site excludes until it is re-scored on the same scale. Read it for the architecture story. The scorecard is the one above.
Who you would be hiring
I am Shane Pilon, and I did not write code until 2025. Five months in, in August 2025, I filed a provisional patent with 17 claims; provisionals last twelve months, and I am not claiming anything is in force today. The work listed below was built since, on one GPU, with no employees and no co-founders, from Pennsylvania. There is no firm behind this, which is the reason the limits on each engagement above are written out rather than left for you to discover.
A dated record that the instruments exist and have been run
What has been built, when, and what it runs on.
Every entry here has a row in the claims ledger. The ledger was last locked to its sources on 2026-08-08.
July 2026
A 2.5B-parameter language model, training from scratch
One 96GB GPU. Lit 2026-07-05 at 17:55:36 UTC on a budget of 8 billion tokens, finishing in August. One evaluation instrument crashed along the way; the training loop did not. The rig needed two servers only because of one subsystem. I cut that subsystem on June 27th, 2026, and the whole thing has run on one card since. Around the model sit startup checks that refuse to launch on bad disk or a busy GPU, rolling checkpoints, small evaluations that fire every few thousand steps, and an escalation path that raises a flag for review instead of killing the run. The next run is already specified and priced at about forty dollars of compute, and stays parked until this one gives me a reason to spend it.
July 2026
Maestro overview, the earlier explainer
An explainer for the research that gives the method and the results and withholds the recipe. One self-contained HTML page, no framework, deployed on Vercel. It was written for a different reader and is not held to the claims ledger this site runs on; the caveat above the link in the research section says which numbers on it differ and why. maestro-overview.vercel.app
September 2025
A document-navigation layer
Read a 75,000-token document in four to five seconds, and 1.2 million tokens in about twenty minutes. Both figures are my own measurements on my own documents. Neither is a public benchmark.
2025
Context scaffolding, before the tooling existed
A hand-built log file, vision file and goal file, fed to a model through a context window a small fraction of today’s size. There was no compaction and no long-context option to fall back on, so I built the scaffolding myself.
Adjacent work
Shipped, dated, and not evidence about measurement discipline. It is here because it is part of the record, not because it argues for the engagements above.
July 2026
Two narrated sites, a portrait and an explainer
One is a portrait narrated across 62 audio segments, each anchored to the paragraph it describes. The other pairs a walkthrough player to the page: every cue carries a text hint, so when the copy changes the player drops the highlight instead of highlighting the wrong line. Static HTML plus audio, on Vercel.
July 2026
Three pricing and ROI calculators
Built for two operating businesses. Each is one self-contained HTML file with no build step. Two of them autosave scenarios to the browser, and to Postgres when the tool is embedded in an admin dashboard, with the database assigning the owner so a saved scenario cannot be claimed by the wrong account. The product catalog behind one of them was extracted from the supplier’s PDF rather than retyped.
One link above. The other five are live but unlisted, and three are internal tools for businesses I have a stake in. Available on request, except one, which carries a third party’s commercial data.
When teams call
Usually not on a quiet Tuesday. They call before a launch, before a board or customer review, before a funding conversation, or after something shipped green and broke anyway. If a number is about to carry weight it has not carried before, that is the moment to have it attacked.
Send me the number
Tell me what you are building and what you are unsure about. If I am the wrong person for it, I will say so and point you somewhere better.
The Evaluation Teardown: day one runs at my cost, and either side can walk away at the end of it with no fee owed.