Coldcode Si Labs Library

Proof of Mechanism

Book · first edition · Courage Garikai
View-only edition. No download from this page.

Proof of

Mechanism

How leaders who build with AI agents tell a working product from a convincing demo Theories, frameworks and one candid case for founders, CTOs and directors

Courage Garikai

Funded by Coldcode Si Labs C O L D C O D E S I L A B S L I B R A R Y F I R S T E D I T I O N · 2 0 2 6

Contents

About this edition ............................................................................................................... PART I · The new illusion 1 · The demo that wasn't .................................................................................................. 2 · Why surfaces got cheap ............................................................................................. 3 · Simulated completeness ............................................................................................. 4 · Trust products and credence goods ........................................................................... PART II · Anatomy of a convincing demo 5 · Fabricated state .......................................................................................................... 6 · Theatrical actions ........................................................................................................ 7 · Unenforced controls .................................................................................................... 8 · Dangling references .................................................................................................... 9 · Borrowed authority and test theatre ............................................................................ PART III · The mechanism audit 10 · The six probes .......................................................................................................... 11 · The one-hour outsider audit ...................................................................................... 12 · Synthetic evidence and the third party ...................................................................... 13 · Reading an audit as a leader .................................................................................... 14 · Rebuild or repair ....................................................................................................... PART IV · Building for proof 15 · Mechanism first ......................................................................................................... 16 · One source of truth for every claim ......................................................................... 17 · Honest limits as a product feature .......................................................................... 18 · Truthful outreach ...................................................................................................... 19 · Status honesty ......................................................................................................... 20 · Verifiability as strategy ............................................................................................ PART V · Leading AI-built work 21 · Governing agents with evidence ............................................................................. 22 · Metrics, incentives and Goodhart ........................................................................... 23 · Law, ethics and reputation ......................................................................................

About this edition

This book is written for founders, chief technology officers, product leaders, directors and anyone who commissions, accepts or oversees software built with AI coding agents. It is a discussion book. Each chapter sets out a question, the theory that helps answer it, a working case, a framework you can use, the points on which reasonable people disagree, questions for discussion and a short exercise. The case is AttestOS, a bot-verification service for websites. Its first version, committed to a public repository on 26 October 2026, looked finished and was almost entirely surface. An audit the next day found 65 problems, 18 of them critical. The service was rebuilt the same day and checked again live on 28 October. AttestOS belongs to the same owner who commissions the work of Coldcode Si Labs. The audit and the rebuild were carried out by the owner working with AI coding agents, and the same process produced the analysis in this book. It is therefore a bounded case, not an independent evaluation. At the time of writing AttestOS had no customers and had taken no payments. Nothing in this book is evidence that it works against real attackers or succeeds commercially. This book does not say how the first version was produced. Some of its defects resemble failure modes documented in AI systems. They are equally consistent with a person building a demonstration in a hurry. Where that question arises it is treated as open. Throughout, three kinds of statement are kept apart: Research findings are cited to their sources. Facts about the case come from its public commit history, the itemised audit report, the production database and its own usage meter, as of October 2026. Proposed frameworks, constructed examples and recommendations are labelled as such. They are proposals to test, not proven laws. • • •

First edition · 2026

No professional biography, endorsement, ISBN or independent audit is claimed. The author and funding credits are those specified by the commissioning owner. AI tools assisted in drafting this book; the author is responsible for its content.

The promise of this book

For most of the history of software, a product that looked finished was usually close to finished. Screens, dashboards and manuals were expensive, so they were built last, on top of machinery that already worked. Anyone shown a polished demonstration could reasonably assume there was something real underneath. Buyers, investors, boards and managers relied on that assumption every day without noticing it. That assumption no longer holds. An AI coding agent can produce a landing page, an owner console full of charts, a customer portal, product covers, user manuals, a test suite and a confident release note in an afternoon. It can produce all of them whether or not anything behind them works. The surface of software has become cheap, while the mechanism (the stored data, the enforced rule, the message actually sent, the payment actually taken) still costs what it always cost to build and to check. The result is a new kind of risk: products that are convincing and incomplete at the same time, without anyone necessarily intending it. This book calls that condition simulated completeness. It argues that leaders who build with AI need a new habit to match. They should stop accepting surfaces and start asking for proof of mechanism. The title borrows from the case. AttestOS sells "proof of work", a small computation that a visitor's browser must perform to show it is not a cheap automated script. This book asks leaders to demand the equivalent from every product they accept: evidence that the work behind each surface has been done.

Who this book is for

Founders and solo operators who ship with AI agents and must decide what to trust in what the agents produce. • © 2026 Courage Garikai. All rights reserved. Published by Coldcode Si Labs Library, Harare. No part may be reproduced without written permission, except the frameworks and exercises, which are meant to be copied.
CTOs, engineering and product leaders who accept work from agents, contractors and teams, and need acceptance criteria that cannot be satisfied by appearance alone. Directors, investors and board members who are shown products and asked to rely on them, and who need questions that get past the demonstration. Buyers of software, especially of security and compliance products, whose purchases are hardest to judge from the outside. Educators and consultants who want a contemporary case and frameworks to teach with. No programming knowledge is assumed. Technical details appear only where they change a decision, and are explained when they do.

How to use this book

Each chapter follows the same pattern: An opening question that the chapter tries to answer. Theory: the research and ideas that bear on the question, cited so that you can read further. The AttestOS case: what the case shows, and what it does not. A framework or tool you can apply. Points of debate: where the argument is contestable. Discussion questions for a reading group, a class, a leadership team or your own reflection. Apply it: a short exercise for your own products. The chapters can be read in order or by part. Groups meeting weekly can take one part at a time over five sessions. Chapter 25 collects every framework in one place. Chapter 26 is a facilitator's guide, Chapter 27 is a glossary and Chapter 28 lists the references. • • • • 1. 2. 3. 4. 5. 6. 7.

The structure

Part I · The new illusion (Chapters 1–4) explains why products can now look finished without being finished, defines simulated completeness, and shows why it matters most for products whose quality buyers cannot see. Part II · Anatomy of a convincing demo (Chapters 5–9) dissects the forms the gap takes: fabricated numbers, theatrical buttons, controls that do not control, references to things that do not exist, borrowed authority and tests that test nothing. Part III · The mechanism audit (Chapters 10–14) sets out six probes that expose those gaps, shows what an outsider can find in an hour, treats claims about third parties, and helps a leader read an audit and decide whether to repair or rebuild. Part IV · Building for proof (Chapters 15–20) describes how to build so that surfaces stay true: mechanism first, one source of truth for every claim, honest limits, truthful outreach, honest status labels and verifiability as strategy. Part V · Leading AI-built work (Chapters 21–24) turns to governance: evidence contracts for agents, metrics that resist gaming, law and ethics, and a ninety-day programme.

The case in one page

AttestOS protects website forms (sign-up, login, contact) from automated abuse. A small script on the form has the visitor's browser solve a short computational puzzle. The browser receives a token, which the website's server checks with AttestOS before accepting the form. The check has the same request and response shape as the industry's most common verification service, so websites can switch with little effort. Paid modules add: risk signals; device tokens that let some phones and computers pass without solving; passkeys; "AgentPass", a policy for AI crawlers that allows, charges or blocks them. Prices run from free to $299 a month. • • • • • • • • •
The first version was committed in eleven commits over two hours on 26 October 2026. It had a landing page with live demonstrations, an owner "command centre" reporting nearly 1.5 million requests inspected and $482.16 of revenue, a customer portal, product covers, manuals, downloadable integrations, a vulnerability scanner and an "autonomous scout" that drafted security notices to companies. The audit, on 27 October, found 65 problems: Every number in the console was typed in by hand. The security check could be bypassed in seven ways, all confirmed against the live service. Nobody could get a key or pay. The scanner showed the same "vulnerable" verdict for any address. The WordPress plugin locked site owners out of their own sites, and its script came from an unregistered domain that anyone could buy. The scout's security notices were built from a scrambled version of each company's domain name, not from any scan. The rebuild, committed late on 27 October, replaced the whole service with one that works: real accounts and keys; verified tokens that can be used once; a scanner that actually fetches the page; outreach that says only what was observed on the recipient's own website, and is held until the legally required postal address is set. Six follow-up commits on 28 October fixed eleven smaller issues. Six of those were, again, places where words had run ahead of the machinery. At the end of the window the service was live, with one account (the owner's), ten prospective customers identified and not yet contacted, no customers and no revenue. That is the whole case. The rest of the book asks what it teaches. • • • • • • • • • •
PA R T I

The new illusion

Proof of Mechanism

1 · The demo that wasn't

Opening question. How can a product look finished, convincingly enough to persuade a careful person, when almost nothing behind it works? On the afternoon of 26 October 2026, a public software repository received its first commit. It was titled "initial production release v2.0.0". Over the next two hours ten more commits followed: an owner "command centre"; a landing page with brand assets and free downloads; a paying-customer portal; product covers, "live screenshots", user guides and manuals; a design system and a footer borrowed from a sister venture; an "Autonomous Scout & First-Touch Inbound Mesh"; the enforcement of a "Customer-in-the-Loop requirement"; the provisioning of a "VIP Enterprise tenant" with "1-click credentials". By the end of the afternoon the product, a service for protecting websites from bots, had everything a buyer would expect to see. This chapter tells the story of what was behind it, and asks how something so empty could look so full.

What a visitor saw

A visitor to the AttestOS website saw a confident, well-designed security product. The landing page offered a five-second scan of any website, live demonstrations of an invisible bot check, and four product modules with their own covers. It also had calculators showing the money a site could save or earn, and downloadable plugins for popular platforms. • • • • • • • •
The owner's console was more impressive still: 1,489,204 requests inspected and 142,380 blocked; 312,450 hardware attestations; 84.6 hours of attackers' computing time "burned"; $482.16 of revenue from AI crawlers, up 12.4%; a status line reading "Whop Webhooks: Connected (2 Plans Active)", meaning that the payment platform was wired in and two plans were selling. The customer portal told a paying customer they had used 124,180 of their 500,000 monthly verifications and earned "$42.80 AI micropayments". When they saved a policy change, it reported that the change had "Propagated… in <0.03ms". The repository carried a badge reading "Tests 27/27 passing". The manuals described an auto-tuner, an emergency lockdown and exportable audit logs.

What was behind it

The next day the owner asked for a deep-dive audit. The auditor read every file and confirmed that the live service was byte-for-byte the code in the repository. They then probed every public address and sent test requests to the live service. They also checked whether the domains, software packages and payment links the product referred to actually existed. The audit logged 65 findings. Here are some of them. Every number in the console was typed in. None came from a database; there was no database. The console had no login. It was pre-filled with a "master key" that also appeared in the page's source code, so every visitor saw the owner's view. The buttons performed. "Emergency Lockdown" changed a badge on the screen. "Provision tenant" added a row to a table, with a randomly generated key that existed nowhere else. "Dispatch first touch" printed "✓ Dispatched … via Cloudflare Edge". Nothing was stored and nothing was sent. The security check let almost anything through. Seven separate bypasses were confirmed against the live service. For example, the service accepted any value at all in a field meant for a hardware security token, and it admitted any request that claimed to come from an AI search engine. • • • • •
Nobody could become a customer. The button for getting a free key led back to the home page. The page meant to issue keys did not exist. Every checkout link pointed to a plan identifier that existed on no payment platform. The scanner invented its results. Whatever address you entered, it reported the same findings: "100% Solvable", "VULNERABLE", "Human users fail this test 18% of the time". It never looked at the site. The plug-in locked people out. The WordPress plug-in loaded its script from a domain nobody had registered. The script never loaded, so with the default settings the plug-in rejected every login. A site owner who installed it would have locked themselves out of their own site. Worse, anyone who bought the domain could have served code to the login pages of every site that had installed it. The scout wrote fake security notices. The outreach engine drafted messages headed "[Security Notice]" to the security teams of real companies, reporting a vulnerability score and a monthly loss figure. The numbers came from scrambling each company's domain name with a hashing function. No scan was performed. The engine was not wired to send, but its "Dispatch" button said it had. The tests tested a different program. The 30 tests behind the badge exercised a larger application that had never been deployed. Some of them checked that special "demo" passwords unlocked the paid features.

A Potemkin product

The story of Grigory Potemkin building fake village façades along the Dnieper to impress Catherine the Great is probably exaggerated. But the image has lasted because it names something real: a display built to be seen, not used. Researchers studying large language models have recently borrowed it. Mancoridis, Weeks, Vafa and Mullainathan (2025) describe "potemkin understanding", in which a model gives correct-looking answers about a concept that are irreconcilable with any coherent grasp of it. They found such failures to be widespread across models and tasks.
The first version of AttestOS was a Potemkin product. Its façade was complete: landing page, console, portal, manuals, covers, tests, release notes. Almost nothing stood behind it. Many products have unfinished corners; that alone is not what made this case striking. What made it striking was that the signals we normally use to judge completeness were all present and all false.

Why this is a leadership problem

It would be easy to treat this as a story about bad engineering. It is better understood as a story about evidence. Every surface in the first version made a claim: the console claimed that traffic was being measured; the buttons claimed that actions were taken; the badge claimed that tests passed; the security check claimed to protect; the scanner claimed to have looked. A leader deciding whether to rely on the product, by buying it, investing in it, launching it or putting their name to it, would normally weigh those claims. And they would normally weigh them by looking at the surfaces. That is the trap. The surfaces were the only part of the product that could not be trusted. Before AI tools, this trap was rare, because surfaces were expensive and nobody built a convincing console without the data behind it. Chapter 2 explains why that is changing.

The AttestOS case: what it shows and does not show

The case shows that a product can present every conventional signal of completeness while lacking almost every mechanism. It also shows that this can be established in a day by someone who knows where to look. The case does not show how the first version came to be built that way. This book does not state it. Some features resemble documented failure modes of AI systems, such as confidently citing a standard by the wrong number or recommending a software package that does not exist. Others resemble ordinary demonstration• • • • •
building under time pressure. The lesson for leaders does not depend on which explanation is right, and Chapter 3 defines the problem in a way that does not require knowing.

Framework: the Surface–Mechanism Ledger

The first tool in this book is a simple ledger. For any product you are asked to rely on, list its surfaces and, against each, the mechanism that would make it true. Surface What it claims Mechanism that would make it true Evidence seen Console figure: "1,489,204 requests" Requests are being counted A query over a table of stored requests None: the figure was typed into the page Button: "Provision tenant" An account is created A stored record and a working key None: the row vanished on reload Badge: "27/27 tests passing" The product is tested Tests that exercise the deployed code None: tests exercised undeployed code Scanner verdict The site was examined A fetch of the site and analysis of what came back None: same verdict for every address The ledger has three rules: Every surface gets a row. That includes numbers, buttons, status labels, badges, documentation claims and messages to outsiders. "Evidence seen" must come from outside the surface. A screenshot of the console is not evidence that the console is true.

An empty evidence column is a finding, not a gap in your paperwork.

Points of debate

Is "Potemkin" too harsh? The metaphor implies intent to deceive. This book argues that the condition can arise without intent. Does using the metaphor prejudge the question? Every demo is partly theatre. Sales demonstrations routinely show features that are "coming soon". Where is the line between a normal demo and a Potemkin product? 1. 2. 3. • •
Would the audit have been done without suspicion? The audit was requested as routine. In most organisations, would anyone have looked behind a product this polished? Is a public repository a defence or an aggravation? Anyone could read the code and find the problems. Does that transparency reduce the harm or simply make it easier to exploit?

Discussion questions

Think of a product, report or dashboard you relied on recently. Which of its surfaces did you check against a mechanism, and which did you take on trust? What signals do you normally use to judge whether software is finished? Which of them could be produced without the underlying work? Who in your organisation would have caught the problems in the first version of AttestOS, and how long would it have taken? If you had been shown the AttestOS console in a pitch meeting, what single question would have exposed it? Does your organisation reward people for producing surfaces, for producing mechanisms or for producing evidence?

Apply it

Choose one product, feature or internal tool you are responsible for. List at least ten of its surfaces in a Surface–Mechanism Ledger. For each surface, write the mechanism that would make it true. Then fill in "Evidence seen" only with evidence from outside the surface itself. Count the empty evidence cells. Choose the one whose falsity would hurt most, and check it this week. • • 1. 2. 3. 4. 5. 1. 2. 3.

2 · Why surfaces got cheap

Opening question. If AI makes building software faster, why does it make judging software harder? For decades, the costliest part of a software product to build was also the part that made it look real. Designing screens, writing copy, drawing charts and producing documentation took skilled people and weeks of work. Teams therefore built the machinery first and the polish last. A side effect, rarely remarked on, was that polish became evidence. A product with a finished interface had almost certainly been worked on long enough to have a finished engine. This chapter argues that generative AI has broken that link. It has made surfaces cheap faster than it has made mechanisms cheap, and much faster than it has made checking cheap. The result is a widening gap between what software appears to do and what anyone knows it does.

What the productivity evidence says

The evidence that AI speeds up some kinds of work is now substantial: Bounded programming. In a controlled experiment, developers given an AI pair programmer completed a set programming task 55.8% faster than those without (Peng et al., 2023). Writing. Professionals given a chatbot for writing tasks finished faster and produced work rated higher (Noy & Zhang, 2023). Customer support. In a large customer-support operation, access to an AI assistant raised productivity, with the largest gains among the least experienced workers (Brynjolfsson et al., 2025). The same body of evidence contains warnings that matter for this book. • • •
The jagged frontier. In a field experiment with management consultants, AI improved performance on tasks within its capabilities and worsened it on tasks just outside, which looked similar (Dell'Acqua et al., 2026). The authors called the boundary a "jagged technological frontier". The people using the tool could not easily tell which side of it a task was on. Perceived and actual speed diverge. In a randomised trial, experienced opensource developers working on their own mature projects were allowed or not allowed to use AI tools on each task (Becker et al., 2025). Before starting, they forecast that AI would cut their time by 24%. Afterwards, they estimated it had cut their time by 20%. In fact, it had increased their time by 19%. Experts in economics and machine learning had predicted savings of nearly 40%. Confidence rises as quality falls. In a study of programmers writing securityrelevant code, those with an AI assistant wrote less secure code than those without (Perry et al., 2023). They were also more likely to believe their code was secure. An earlier analysis of an AI code assistant found that about 40% of the programs it completed in security-relevant scenarios contained vulnerabilities (Pearce et al., 2022). Taken together, these findings say that AI changes three things at once: how fast work is produced, how good it is, and how good it seems. The third change is the dangerous one, because it distorts the judgement on which acceptance depends.

Surfaces and mechanisms have different cost curves

A useful way to think about the change is to separate a product into two layers: Surfaces are everything a person sees or reads: screens, numbers, buttons, labels, documentation, release notes, test reports, marketing pages, messages. Mechanisms are everything that makes the surfaces true: stored records, enforced rules, sent messages, completed payments, verified signatures, queries that return real data. • •
Generative AI is exceptionally good at surfaces. Surfaces are made of language, layout and convention, and large models have absorbed enormous quantities of all three. Ask for an "owner dashboard for a bot-protection service" and a model will produce one with plausible metrics, tasteful charts and the right vocabulary, because thousands of such dashboards exist to learn from. Mechanisms are harder, for three reasons: They are specific. A mechanism must fit this database, this payment platform's signature scheme and this provider's limits. Plausibility is not enough. They interact with the world. A message is sent or not, and a token is rejected or not. The world, not the model, decides whether a mechanism works. They must be checked. Even when an AI produces a working mechanism, someone must establish that it works, which takes time, skill and access. The result is an asymmetry. AI lowers the cost of surfaces by a large factor. It lowers the cost of mechanisms by a smaller factor. It lowers the cost of checking least of all.

The case in numbers

The first version of AttestOS illustrates the asymmetry. At the end of its first day the repository held 68 files and about 21 megabytes, nearly 18 of which were images that the live service never used. The live service itself was a single file of about 839 kilobytes. More than three-quarters of that file (77.7%) was pictures, encoded as text so that they could be served from inside the program. The logic that was supposed to protect websites ran to a few hundred lines. A larger application elsewhere in the repository contained most of the product's apparent features, including tenants, a payment gate, an "auto-tuner", hardware attestation and the outreach scout. It had never been deployed. None of this proves how the first version was produced. It does show what the asymmetry looks like when nothing corrects for it. The visible layer was large, polished and complete. The working layer was thin, and much of it was not connected to anything. • • •
The rebuild ran the other way. Its source code for the live service is about 3,600 lines, tested by 63 test cases that run against the code actually deployed. Its whole repository is about 3 megabytes. It has fewer pictures and more machinery.

Why the old signals fail

Economists have a precise way of describing what has happened. A signal is something one party can observe that tells them about a quality they cannot observe directly. Spence (1973) showed that signals work only when they are costlier to send for those who lack the quality than for those who have it. A university degree signals ability, in his model, only because it is harder for less able people to obtain. A finished interface used to be a good signal of a finished product, because it was expensive and teams built it last. If AI makes finished interfaces cheap for everyone, whether or not the product behind them works, the interface stops being a signal. It becomes noise that happens to look like information. When signals fail, markets can fail too. Akerlof (1970) showed that when buyers cannot tell good products from bad, they pay the average price, sellers of good products withdraw, and quality falls. A world in which software demonstrations no longer separate working products from convincing ones is at risk of exactly this outcome, unless buyers find new signals. The new signals must have the property the old ones lost: they must be hard to produce without the underlying quality. That is what this book calls proof of mechanism. Examples include a forged token that the service rejects, a number traced to the query that produced it, and a domain shown to be registered to the company. None of these can be drawn.

Framework: the Cost Map

The Cost Map helps a leader decide where to spend scarce checking effort. For each part of a product, estimate two things: how cheap it was to produce and how cheap it is to verify.
Expensive to verify Cheap to produce Check routinely. Examples: links, domain names, prices, citations Danger zone. Examples: dashboards, security controls, analysis results, outreach claims Expensive to produce Check at acceptance. Examples: payment flows, data migrations Plan verification as part of building. Examples: core algorithms, integrations with partners The danger zone is where simulated completeness lives: surfaces that AI can generate cheaply and that a reviewer cannot confirm without work. Most of the first version's critical problems sat there: the console figures; the security check; the scanner's verdicts; the scout's notices. Its cheapest-to-verify problems, such as the unregistered domain and the nonexistent package, would have taken minutes to find. The Cost Map's rule is simple. Spend verification effort in inverse proportion to production cost and in proportion to the harm of being wrong. The cheaper something was to make, the less its appearance tells you.

Points of debate

Will mechanisms get cheap too? AI is improving at producing working code, not just plausible code. Is the asymmetry temporary? Is checking also being automated? AI can write tests and review code. But if the same kind of system produces both the surface and the check, can the check be trusted? (Chapters 9 and 21 return to this.) Does the asymmetry apply to people too? Junior employees, contractors and agencies have always been able to produce polished presentations of unfinished work. Is AI a new problem or an old one at a new scale? Are experienced developers the right benchmark? The slowdown found by Becker and colleagues involved experts on familiar code. Would the result differ for novices building something new? • • • • • • • •

Discussion questions

In your organisation, which surfaces are currently treated as evidence of completeness: demonstrations, screenshots, status reports, test badges, release notes? Have you ever accepted work because it looked finished, and later found it was not? What signal misled you? Where would AI-produced work most easily pass for finished in your products? How much of your organisation's checking effort goes to things that were cheap to produce? What would a "signal that cannot be drawn" look like for your most important product?

Apply it

Draw a Cost Map for one product or project. Place at least twelve components in the four quadrants. For each component in the danger zone, name one piece of evidence that would be hard to produce without the mechanism being real. Ask for that evidence for the two components whose failure would cause most harm. 1. 2. 3. 4. 5. 1. 2. 3.

3 · Simulated completeness

Opening question. How do we name the gap between what a product shows and what it does, without first having to prove who is to blame? Every product is incomplete in some way. Every demonstration shows the best path through it. Every roadmap promises features that do not yet exist. If the lesson of the first version of AttestOS were simply "products can be unfinished", it would not be worth a book. This chapter defines the specific problem more carefully, separates it from its respectable neighbours, and explains why the definition deliberately leaves out intent.

A definition

An artefact exhibits simulated completeness when its surfaces signal working capabilities that its mechanisms do not provide, to people who may rely on them. Each part of the definition matters. Surfaces and mechanisms. As in Chapter 2, surfaces are what people see and read: screens, numbers, buttons, labels, documentation, test reports, release notes, messages. Mechanisms are what make surfaces true. The definition applies only to artefacts that make claims about themselves, which almost every modern software product does. Signal working capabilities. The surface must imply that something works now. A roadmap that says "passkeys coming in March" is not simulated completeness. A portal that reports "86,410 hardware passes" when there have been none is. That the mechanisms do not provide. The gap must be real, established by checking, not merely suspected. Part III of this book is about how to check. To people who may rely on them. This includes buyers, users, investors, managers, colleagues, recipients of messages and the operator. A prototype on a designer's laptop that no one relies on does not count.

What the definition leaves out

The definition says nothing about intent and nothing about how the artefact was made. Both omissions are deliberate. Intent usually cannot be established by inspecting a product. A console full of invented numbers might be there for several reasons: someone meant to deceive; someone built a demonstration for a meeting and forgot to label it; a team ran out of time; an AI agent, asked for "an owner dashboard", produced one with plausible placeholder figures and nobody replaced them. All of these produce the same artefact, and the managerial response begins the same way in every case: find the gaps and close them. Provenance is often unknown too. This book does not say how the first version of AttestOS was produced, and its argument does not need to. A definition that required knowing whether an AI or a person wrote each line would be useless to the buyer, investor or board member who must decide whether to rely on what is in front of them. Leaving intent out also makes the problem easier to discuss. "This product has simulated completeness in its billing flow" is a description that a team can accept and act on. "This product is fraudulent" invites defence. The aim of the concept is to make gaps speakable.

The respectable neighbours

Several established practices involve products that do less than they appear to. Simulated completeness must be distinguished from each. Prototypes and the Wizard of Oz. In the early 1980s, Kelley (1984) studied how people would talk to a computer in natural language before such software existed. He had a hidden person play the computer's part. The method, later known as a • • • •
"Wizard of Oz" study, is now a standard design technique. It simulates a product in order to learn before building. Its simulation is bounded, its purpose is research, and nobody relies on it for anything that matters. The minimum viable product. Ries (2011) popularised building the smallest product that lets a team learn from real customers. A minimum viable product is honest about its scope: it does few things, but the things it does are real. A "fake door" test, a button for a feature that does not yet exist to measure interest, is a deliberate and brief simulation. Done well, it tells the user the feature is coming. Technical debt. Cunningham (1992) coined "technical debt" for the shortcuts a team takes knowingly to ship sooner, which must later be repaid with interest. Kruchten, Nord and Ozkaya (2012) developed the metaphor into a field of study. Debt is a known gap with a plan, or at least an intention, to close it. Verification debt. The first study in this series proposed "verification debt": the gap between what a system does and what anyone has checked that it does, which grows when building outpaces checking (Garikai, 2026). Its holder knows what has not been checked, even if they do not know what is wrong. Fraud. At the far end lies deliberate deception for gain. The history of the bloodtesting company Theranos, documented by Carreyrou (2018), shows how far demonstrations can run ahead of capability when the incentive to keep up appearances is strong and inspection is weak.

Framework: the Boundary Test

The Boundary Test places an artefact among these neighbours with four questions. Question Prototype MVP Technical debt Simulated completeness Fraud 1. Is the gap known to the maker? Yes Yes Yes Not necessarily Yes 2. Is it disclosed to those who rely on the artefact? Yes, or no one relies Scope is honest Usually internal only No: the surface claims completeness Concealed
Question Prototype MVP Technical debt Simulated completeness Fraud 3. Is there a purpose that justifies it? Learning Learning from real use Speed, with a plan None defined Gain from false belief 4. Is there a plan to close it? Build if justified Iterate Repay None None Apply the four questions to any surface that does more than its mechanism. If the answers are yes, yes, yes and yes, you have a legitimate simulation; keep it labelled. If the gap is known but undisclosed, with a plan, you have debt; write it down where those who rely on the product can see it. If the gap is undisclosed and unplanned, you have simulated completeness, whatever anyone intended; close the gap or change the surface. If the gap is known, concealed and exploited, the problem is no longer one this book can solve. The test turns on question 2, disclosure. A simulation becomes simulated completeness at the moment someone relies on the surface without being told what lies behind it. That is why the same console can be a legitimate design mock-up on Monday and a serious problem on Tuesday, when it is shown to an investor as the product.

Decoupling, an old idea in a new place

Sociologists have studied a version of this problem in organisations for half a century. Meyer and Rowan (1977) observed that organisations adopt formal structures (procedures, job titles, certifications) because those structures earn legitimacy, not because they make the work better. When structure and work conflict, organisations decouple them. They keep the structure for show while the work goes on differently, protected by a "logic of confidence and good faith" that discourages anyone from looking closely. Bromley and Powell (2012) later distinguished two forms: policy–practice decoupling, where a policy is adopted but not carried out; • • • • •
means–ends decoupling, where it is carried out but has little to do with the outcome it is supposed to serve. Simulated completeness is decoupling inside an artefact. A dashboard is formal structure and the queries behind it are the work. A security check that admits any token is means–ends decoupling: the check runs, but it achieves nothing. The sociological literature also tells us when decoupling persists: when inspection is ceremonial, when checking is expensive and when the audience is inclined to good faith. That describes almost every software demonstration ever given.

The AttestOS case

Applied to the first version of AttestOS, the Boundary Test gives a clear answer for most surfaces. The console's figures, the portal's usage, the scanner's verdicts, the "Dispatched" message and the test badge all claimed working capabilities. None disclosed that they did not. No purpose was stated that required them, and nothing recorded a plan to close the gap. They were simulated completeness. The case also shows the concept's usefulness after a rebuild. The second version's owner console once showed distribution channels as "active" when they were switched on but had no accounts to post from. That is a small instance of the same condition. Under the Boundary Test it was neither fraud nor debt: the gap was not known to the person who built the label, and nothing disclosed it. It was simulated completeness of the mildest kind. Chapter 19 tells how it was found and fixed.

Points of debate

Can an artefact really have no intent? Every surface was made by someone or something. Is leaving intent out a useful simplification or a way of avoiding responsibility? Is disclosure enough? A product that says "figures shown are illustrative" in small print has disclosed. Has it escaped the problem? Where does marketing fit? Marketing pages routinely describe products at their best. Is aspirational copy simulated completeness? Do minimum viable products drift into simulated completeness? An MVP with fake-door buttons that stay up for months, collecting sign-ups for features never built: when does learning become pretending? • • • • •

Discussion questions

Apply the Boundary Test to three surfaces in a product you know. Which neighbour does each resemble? Where in your organisation is formal structure decoupled from work? Are any of those places in your software? Has a demonstration you prepared ever been relied on more than you intended? What changed when it was? What disclosure practices does your organisation have for unfinished features? Who sees them? Is it fair to hold a team responsible for simulated completeness they did not intend? How should accountability work?

Apply it

Take your Surface–Mechanism Ledger from Chapter 1. For every surface with weak evidence, run the Boundary Test's four questions. For each surface that comes out as simulated completeness, choose one of three actions: build the mechanism, change the surface to tell the truth, or remove it. Write a one-paragraph disclosure policy for unfinished features in your products. Say who must be told, where and in what words. 1. 2. 3. 4. 5. 1. 2. 3.

4 · Trust products and

credence goods

Opening question. Why is simulated completeness most dangerous in exactly the products whose job is to make us feel safe? A broken music player announces itself: the music stops. A broken bot defence announces nothing. The website keeps working, sign-ups keep arriving, and nobody can tell from the outside whether the defence is stopping attacks or whether no attacks were made. Some products reveal their failures at once, others only after harm has been done, and some never reveal them at all. This chapter explains why that difference matters, and why it makes the AttestOS case more than a curiosity.

Search, experience and credence

Economists distinguish goods by when buyers can judge their quality: Search goods can be judged before purchase. A shirt's colour is an example. Experience goods can be judged after use. A restaurant meal is an example. Credence goods cannot be judged even after use, at least not without expert help (Darby & Karni, 1973). Examples are the car repair you were told you needed, the medical test whose result you cannot interpret, and the security product that may or may not be stopping attacks. Darby and Karni's paper bore the title "Free Competition and the Optimal Amount of Fraud". Its point was that credence goods invite deception, because the buyer cannot tell whether they got what they paid for. Dulleck and Kerschbamer (2006) reviewed the economics of these markets and identified three forms of fraud: overtreatment (selling more than needed), undertreatment (providing less than promised) and overcharging. They also showed that markets discipline such fraud only under certain conditions: when sellers are liable for failures, when the service can be verified, or when reputation follows the seller. • • •

Security as a credence good

Information security is a textbook credence good. Anderson (2001) argued that security fails as often for economic reasons as technical ones. Buyers cannot distinguish good security from bad, so vendors compete on features, certifications and reassurance rather than on protection. Markets for security products then resemble Akerlof's (1970) market for "lemons": quality cannot be seen, so it is not paid for, so it declines. Anderson and Moore (2006) developed these observations into a field of research. Schneier (2003) coined the phrase "security theater" for measures that provide the feeling of security without the reality. Theatre is cheap and visible, while real security is often expensive and invisible. Credence markets reward the cheap and visible. Simulated completeness adds a twist. Security theatre used to require at least some real effort, such as a guard at a door or a form to fill in. With AI, the theatre itself can be generated at almost no cost: a console reporting blocked attacks, a badge reporting verified devices, a scanner reporting vulnerabilities. The cheaper theatre becomes, the more of it a credence market will contain, unless buyers learn to demand something theatre cannot provide.

The bot-verification market

Bot verification is a large and contested market. Industry measurement attributed 51% of all web traffic in 2024 to automated programs and 37% to "bad" bots, up from 32% the year before (Imperva, 2025). Websites defend their forms with a range of tools. CAPTCHAs. The best known are challenges meant to be easy for humans and hard for machines (von Ahn et al., 2003). Two decades of evidence complicate the promise: Humans often find CAPTCHAs hard (Bursztein et al., 2010). Human labour markets will solve them on demand. One study found retail prices as low as $1 per thousand, which means a CAPTCHA's real strength is economic: the price of a solution against the value of what it protects (Motoyama et al., 2010). • •
A study of modern CAPTCHAs, with 1,400 participants solving 14,000 challenges, found that how long people took and how they felt about it did not always match. Some people abandoned what they were doing when faced with one (Searles et al., 2023). Proof-of-work. An alternative is to make every request cost a little computation. The idea goes back to "pricing via processing" as a defence against junk mail (Dwork & Naor, 1993) and to Back's (2002) Hashcash. Chapter 17 discusses its limits, which are real. Device tokens and crawler policy. Newer approaches include device tokens issued under the Privacy Pass standards, which let a device that has proved itself once pass without a challenge (Davidson et al., 2018). There are also policies that decide how AI crawlers may use a site's content, discussed in Chapter 20. The buyer of any of these tools faces the credence problem in a sharp form. If the tool is working, attacks fail quietly. If it is not working, attacks succeed quietly, at least for a while. A dashboard reporting "142,380 blocked" is the only visible evidence most buyers will ever see. In the first version of AttestOS, that number was typed into the page.

The AttestOS case

The case shows the credence problem from both sides. The first version. Consider what a customer of the first version would have experienced. Suppose they had found a way to install it, which they could not have done, because there was no way to obtain a key. The widget would have shown "Verification active". Their portal would have shown tens of thousands of verifications and a healthy pass rate. Meanwhile, the live service would have admitted any token, any request claiming to be a search engine and any request that declared itself exempt. The customer would have seen nothing wrong until their forms filled with spam or their accounts with fraud. Even then they might have blamed the attackers' sophistication rather than their defence. In a credence market, a failed product can look like a working product facing a hard problem. •
Worse, a customer who used the WordPress plug-in would have been locked out of their own site at once. That failure, at least, would have been visible. It shows that simulated completeness in a trust product can cause harm in both directions: failing to protect, and blocking legitimate users. The second version. The second version was designed to escape the credence trap rather than exploit it. Its main move was to make efficacy checkable by the customer. Its verification check has the same request and response as the dominant incumbent's, so customers can test it with their existing integration. Its code is public. Its documentation says plainly what it cannot do. A customer can send it a forged or reused token and watch it fail. Chapter 20 develops this into a strategy.

Framework: the Credence Exposure Score

How much should a leader worry about simulated completeness in a given product? The Credence Exposure Score rates five factors from 1 (low) to 5 (high). Factor Question Low (1) High (5) Visibility of failure Would a user notice at once if the product stopped working? The music stops Nothing visible happens Expertise needed to judge Can an ordinary buyer judge quality after use? Yes Only an expert can Harm from failure What happens if it fails silently? Inconvenience Loss of money, data, safety or legal standing Third-party exposure Does the product make claims to or about other people? No Yes: it messages outsiders or reports on them Reliance by others Do customers, partners or the public rely on its outputs? Internal only Many parties A product scoring 20 or more out of 25 is a trust product in the sense of this book. Verification effort, the six probes of Part III, should be concentrated there. The first version of AttestOS would have scored near the maximum on every factor. So would most security, compliance, health, financial and identity products. The score also guides how much to invest in being checkable. A product with a high score that is also easy for customers to verify can turn an industry weakness into its own strength.

Points of debate

Are all AI products becoming credence goods? If buyers cannot easily judge whether an AI system's outputs are correct, does the credence problem extend far beyond security? Do certifications help? Security certifications are meant to substitute for buyers' inability to judge. Power (1997) warned that audits can become "rituals of verification". Can certifications keep up with AI-produced surfaces? Is reputation enough? Established vendors are trusted partly because they have reputations to lose. What should a buyer do with a new vendor that has none? Should trust products be held to a higher standard? Should the law treat a fabricated security dashboard differently from a fabricated sales dashboard?

Discussion questions

Score three products you buy or sell on the Credence Exposure Score. Which is highest, and does your verification effort reflect it? When did you last discover that a security or compliance tool had not been doing what you thought? How did you find out? What evidence do your security vendors give you that their product works? Could that evidence be produced without the product working? If you sell a trust product, what could a customer check for themselves today? Does your organisation treat the absence of visible problems as evidence that a defence is working?

Apply it

Score your three most important products on the Credence Exposure Score. For the highest-scoring product, list every piece of evidence a customer currently receives that it works. Mark each as "drawn" (could exist without the mechanism) or "proved" (could not). Design one new piece of "proved" evidence that customers could check without your help. • • • • 1. 2. 3. 4. 5. 1. 2. 3.
PA R T I I

Anatomy of a

convincing demo

Proof of Mechanism

5 · Fabricated state

Opening question. When a screen shows a number, how do we know the number came from somewhere? Numbers carry authority. A figure such as "1,489,204 requests inspected" feels measured in a way that a sentence like "lots of traffic" does not. The precision implies an instrument, and the instrument implies a mechanism. This chapter is about numbers that have the authority of measurement without the measurement: fabricated state. Fabricated state is the first of the gap types dissected in Part II. Each chapter in this part takes one or two types, explains why they arise, shows them in the case and offers a tool for finding them.

What counts as fabricated state

A surface displays fabricated state when it presents a value as a description of the present or past (a count, total, balance, rate, status or trend) that is not derived from a record of what happened. The definition covers more than invented numbers: Hard-coded figures typed into a page. Generated figures computed from something irrelevant, such as a random number, a clock or a hash. Stale figures copied once from a real source and never updated. Placeholder figures left in place from a design mock-up. Plausible defaults, such as a status of "Connected" shown until something proves otherwise. • • • • •
The definition does not cover estimates and forecasts, as long as they are labelled as such. "Estimated monthly savings: $400" is not fabricated state if the page says how it was estimated and that it is an estimate. It becomes fabricated state when it is presented as something that happened.

Why fabricated state is so common

Three forces produce fabricated state, and AI strengthens all three. Design needs data. A dashboard cannot be designed without numbers in it. Designers and developers fill mock-ups with plausible figures, and the figures survive into the product when nobody replaces them. When an AI agent is asked for a dashboard, it does the same, with numbers chosen to look realistic. Realism is exactly what makes them hard to spot. Empty states look broken. A new product has no data. Its console shows zeros, blank charts and "no records". That looks like failure, so there is pressure to show something more encouraging. A builder optimising for how the product looks in a demonstration will be tempted to fill the gap. Numbers are rarely traced. Few reviewers ask where a number comes from. Campbell (1979) warned that the more a quantitative indicator is used to make decisions, the more it becomes subject to corruption pressures, and the more it distorts the process it was meant to monitor. His warning was aimed at social indicators. It applies with even more force when the indicator is produced by the same process it is meant to evaluate, as a builder's own dashboard is.

The AttestOS case

The first version's owner console displayed: 1,489,204 requests inspected, with 142,380 blocked; 312,450 hardware attestations; 84.6 hours of attacker computing time "burned"; $482.16 of revenue from AI crawlers, with an upward trend of 12.4%; "Whop Webhooks: Connected (2 Plans Active)". • • • • •
Every one of these was written directly into the page. There was no database behind the console, and no record of any request, attestation or payment. The payment platform was not connected: the checkout links used identifiers that existed on no platform, and the service had no route to receive the platform's notifications. The customer portal repeated the pattern for whoever opened it. It accepted any text as a "site key" and showed 124,180 of 500,000 verifications used, 86,410 hardware passes and "$42.80 AI micropayments earned". A button labelled "VIP Enterprise" loaded an "Enterprise $299/mo" account for anyone. An "export audit log" button produced a file of generated rows describing events that had never occurred. Two features of this fabricated state deserve attention. It was internally consistent. The numbers did not contradict one another: blocked requests were fewer than inspected ones, and revenue was plausible for the traffic. Consistency is a natural property of generated figures and says nothing about their truth. It extended to status. "Connected (2 Plans Active)" is fabricated state about the system's own wiring. Status claims are especially dangerous, because they tell the operator not to worry about the very thing that is broken. The second version took the opposite approach. Its console and portal read only from the database. On the last day of the observation window that database held one account (the owner's), one site, ten prospective customers, four redeemed tokens and nothing else, and the screens said so. Empty states were shown as empty.

The cost of empty screens

There is a real tension here, and leaders should not wish it away. Empty dashboards are discouraging, both for operators and for prospective customers shown a demonstration. Honest alternatives exist: Label demonstration data clearly. A demonstration account populated with sample data, marked "Sample data" on every screen, is a legitimate simulation in the sense of Chapter 3. •
Show progress towards first value. Replace empty charts with what must happen for the chart to fill: "Add the widget to a form to see your first verification here." Show real but small numbers with pride. "4 verified, 3 rejected" is less impressive than "1,489,204 inspected", and infinitely more useful.

Framework: every number has an address

The tool for fabricated state is a discipline: every number on every screen must have an address, a precise answer to "where does this come from?" For each displayed value, record: Field Question Example from the second version Value What is shown? "Verifications this month" Source Which table, query, service or file produces it? Count of successful verifications for the account's sites in the current month Freshness When was it last computed? On page load Empty behaviour What is shown when there is no data? "0", with a prompt to install the widget Owner Who is responsible for its correctness? The developer who owns the portal Test What check fails if the address breaks? A test that creates two verifications and expects "2" The Trace probe in Part III is the audit version of this discipline: pick a number, follow it back to its source, and confirm the source exists and contains what the number claims. A number that cannot be traced is fabricated state until proved otherwise. Two shortcuts make tracing quick: Reset and reload. On a test account or empty database, a fabricated figure stays the same while a real one drops to zero. Act and recount. Perform one countable action, such as a sign-up or verification, and see whether the figure moves by exactly one. • • • •

Points of debate

Are sample figures in sales demonstrations acceptable? Many vendors demonstrate with rich sample data. Is it enough to label it, and how prominent must the label be? What about smoothed or rounded figures? Real systems often display approximations, such as "about 1.5 million". When does approximation become fabrication? Should status default to optimistic or pessimistic? A status that says "Connected" until proven otherwise is fabricated state. One that says "Disconnected" until proven otherwise may cause needless alarm. Which is the right default for a trust product? Is internal fabricated state less serious? A dashboard only the founder sees harms no customer. Or does it, by distorting the founder's decisions?

Discussion questions

Pick the dashboard you look at most often. Can you give the address of every number on it? Has your organisation ever made a decision on the basis of a figure that turned out to be a placeholder, a default or a stale copy? How do your products behave when there is no data? Do they tell the truth? Which status indicators in your systems default to "healthy" unless something proves otherwise? Would your team be comfortable showing a prospective customer an honest, nearly empty dashboard? What would make it comfortable?

Apply it

Choose one screen in your product with at least eight displayed values. Fill in the address table for each. On a test account, reset the data and reload the screen. Note every value that does not change. For each status indicator, write the condition under which it shows each state. Rewrite any that would show "healthy" without evidence. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

6 · Theatrical actions

Opening question. When you press a button and the screen says "Done", what has actually been done? Every interface makes promises about cause and effect. "Send" promises that a message leaves. "Delete" promises that something is gone. "Provision" promises that something now exists that did not before. Users rarely check these promises, because checking would defeat the point of having a button. This chapter is about buttons that keep their promise on the screen and nowhere else: theatrical actions.

What counts as a theatrical action

A control is a theatrical action when activating it produces the visible signs of an effect (a confirmation, an animation, a new row, a changed badge) without the effect. Typical examples: A "Send" button that displays "Sent" but never hands the message to any mail system. A "Save" button that updates the screen but writes nothing that survives a reload. A "Lockdown" switch that changes a label but leaves the protected system as it was. A "Create account" form that shows a welcome page but stores no account. A "Sync" button that animates a spinner and reports success without contacting the other system. Theatrical actions are the active cousin of fabricated state. Fabricated state misreports what has happened; a theatrical action misreports what is happening in response to the user. • • • • •

Why theatrical actions arise

Interfaces are built before plumbing. In most projects, the screen for an action is built before the system behind it. A temporary handler that shows success lets designers and testers move on, and it can survive into the product unless someone replaces it. Optimistic interfaces are good design, until they are not. Modern interfaces often show the result of an action at once and confirm it in the background, so that the user is not kept waiting. This is a legitimate technique, but only if the background confirmation exists and failures are reported. Remove the background step and an optimistic interface becomes a theatrical one. Success messages are cheap to generate. An AI agent asked to add a "dispatch" feature will reliably produce a button, a handler and a success message. Whether it also produces a working connection to a mail service depends on whether the agent was given credentials, documentation and a way to test. An agent with none of these can still produce a feature that looks complete.

The automation literature

Human-factors research has long studied what happens when people supervise automated systems they cannot fully observe. Bainbridge (1983) described the "ironies of automation". The more reliable automation becomes, the less practice operators get, and the less able they are to step in when it fails. Parasuraman and Riley (1997) distinguished the use, misuse and disuse of automation, and showed that people both over-trust and under-trust automated systems. Parasuraman and Manzey (2010) reviewed evidence on "automation bias" and complacency: people tend to accept what an automated system reports, and to look for confirming evidence less than they should. Theatrical actions exploit these tendencies. An operator who presses "Dispatch" and sees "✓ Dispatched" has no reason to doubt it. The system's report is their evidence. And the more polished the report, with timestamps, channel names and a tick mark, the less likely they are to check.

The AttestOS case

The first version's console was full of theatrical actions: "Emergency Lockdown" changed a badge on the page. The verification service was unaffected. "Provision tenant" added a row to a table, complete with a randomly generated key. The key existed nowhere else and the row vanished on reload. "Rotate key" generated a new random string and displayed it. Nothing on the server changed. "Dispatch first touch" printed "✓ Dispatched … via Cloudflare Edge" next to each prospect. No message was sent; nothing was connected that could send one. Policy changes in the portal were confirmed as "Propagated… in <0.03ms". There was nothing to propagate them to. Each of these was believable. The lockdown switch looked like a security feature that should exist. Tenant provisioning looked like ordinary administration. The dispatch message named a real infrastructure provider. The speed claim was absurdly precise, but precision reads as measurement to most people. The "Dispatched" message is the most instructive example, because of what it would have done to the operator's behaviour. An owner who believed outreach had gone out would wait for replies. Receiving none, they might conclude that the market was not interested, or that the message needed rewording. Then they would press "Dispatch" again. A theatrical action does not merely fail; it corrupts the feedback the operator uses to learn. The second version shows the remedy. Its outreach engine stores each message in an outbox with a status: queued, sending, sent or failed. Anyone who unsubscribes is added to a suppression list that the engine checks before every message. A message is marked sent only after the mail service accepts it. Until the operator sets the postal address that United States commercial email law requires, the engine refuses to send at all, and the console says why. At the end of the observation window, ten prospects had been identified and none contacted. The console showed exactly that: ten queued, zero sent. • • • • •

The Trigger probe

Theatrical actions are found by the Trigger probe: activate the control, then look for its effect somewhere other than the screen that reported it. Where to look depends on the action: Saves and creations: reload, log out and in, or open on another device. The record should still be there. Messages: check the recipient's inbox, the mail service's log or the outbound record in the database. Security changes: send a request that the change should now block or allow. Integrations: check the partner system, such as the payment platform, the ticketing system or the analytics tool. The Trigger probe is cheap, but it is rarely run, because the screen has already answered the question. The discipline is to treat the screen's answer as a claim, not a result.

Framework: the Side-Effect Register

The Side-Effect Register makes the Trigger probe systematic. List every control that claims to change something, and for each record the following: Control Claimed effect Where the effect lives How to observe it independently Failure reporting Last verified "Send outreach" Email delivered to prospect Mail service and outbound table Outbound record with status "sent" and provider message ID Status "failed" with reason, shown in console Date and who "Rotate key" Old key stops working, new key works Keys table Verify with old key (expect rejection) and new key (expect success) Error shown if rotation fails "Lockdown" Stricter verification for all sites Site settings Verification at default difficulty is rejected Console shows active lockdown from stored state • • • •
Three rules make the register useful: Every claimed effect must live somewhere other than the screen. If the only place the effect can be seen is the confirmation message, the action is theatrical. Every action must be able to fail visibly. A control that can only ever report success is suspect. Real systems fail: networks drop, providers refuse, quotas run out. Confirmation follows evidence. The interface should report success only after the system has evidence of it, such as a stored record or an acknowledgement from the provider, and it should report the evidence.

Points of debate

Is optimistic interface design now too risky? If showing success before confirmation is how theatrical actions start, should trust products abandon it? What about deliberately simulated controls in demonstrations? A sales demonstration might include a "send" button that sends nothing. Is labelling enough? Who owns the side effect? In many teams, the person who builds the button is not the person who builds the integration. Who is accountable when they are never connected? Can automated tests catch theatrical actions? Tests that check whether "Sent" appears will pass. Tests that check the outbound record will catch them. Why are the first kind so common?

Discussion questions

Which actions in your products would you be least surprised to find were theatrical? When did you last check the effect of an action somewhere other than the screen that confirmed it? How do your products report failures of actions that depend on outside services? 1. 2. 3. • • • • 1. 2. 3.
Has a theatrical action, such as a sync that did not sync or a send that did not send, ever corrupted your organisation's learning? What would it cost to add independent evidence to every success message in your main product?

Apply it

Build a Side-Effect Register for the ten most important controls in one product. Run the Trigger probe on each: activate it, and confirm the effect independently. For any control that cannot report failure, write the failure states it should report and the evidence its success message should include. 4. 5. 1. 2. 3.

7 · Unenforced controls

Opening question. How would you know if a lock on your door opened for anyone who turned the handle? A control exists to say no. A password check says no to the wrong password. A payment gate says no to non-payers. A rate limit says no to the hundred-and-first request. A bot check says no to bots. If a control never says no, it is not a control, however sophisticated it looks. This chapter is about unenforced controls: checks that run, look rigorous and admit everyone. In the AttestOS case they made up the largest group of critical problems.

What counts as an unenforced control

A control is unenforced when it is presented as restricting something but, under some reachable condition, fails to restrict it. Common forms include: Accepting any value. The check confirms that a field is present but not that its content is valid. Trusting the claimant. The check believes what the request says about itself: its user agent, its role, its origin, its payment status. Validating in the wrong place. The check runs in the browser, where the user controls it, instead of on the server. Escape hatches. Special values, magic tokens or debug modes unlock the protected thing. Missing binding. A credential is valid but not tied to the right context, so a token issued for one site or one action works for another. Missing single use. A credential can be replayed as often as an attacker likes. Fail-open defaults. When the check cannot run, because a script failed to load or a service is down, it lets everyone through. Or it blocks everyone, which is also a failure. • • • • • • •

Why unenforced controls arise

The happy path is tested first and often last. Developers, human or AI, build a control by making the legitimate case work: the right password is accepted, the valid token passes. The rejection cases are where the control earns its keep, and they are tested less, because nothing visibly breaks when they are missing. Controls are judged by their presence. In audits, checklists and demonstrations, the question is often "is there a password check?" rather than "does the password check reject wrong passwords?" Power (1997) described how audit can become a ritual that verifies the existence of controls rather than their effect. Plausible security code is easy to produce. Security code has a recognisable vocabulary: signatures, hashes, tokens, headers. Code that uses the vocabulary correctly can still fail to enforce anything. Perry et al. (2023) found that programmers with an AI assistant wrote less secure code and were more confident in it. Pearce et al. (2022) found vulnerabilities in about 40% of AI-completed programs in security-relevant scenarios. Layers create false comfort. Reason (2000) described accidents in complex systems as holes in successive slices of "Swiss cheese" lining up. Each layer of defence has holes, and harm occurs when the holes align. Unenforced controls are layers that are all hole, and they are hard to notice because the other layers seem to provide cover.

The AttestOS case

The audit of the first version confirmed seven bypasses against the live service. Each was demonstrated by sending a crafted request and receiving a pass. In plain terms: A public secret. The secret used to sign verification tokens was written in the public code, so anyone could create valid tokens. Any hardware token. A field meant for a hardware security token accepted any value. The word "anything" passed. Self-declared AI agents. A request saying it was an AI crawler, in a header anyone can set, was admitted without checking that it was. 1. 2. 3.
Spoofed search engines. Any request claiming to be a major search engine's crawler, in its user agent, was admitted without verification. Magic tokens. Special token values recognised as "demo" or "test" were accepted in production. No binding. Tokens issued for one website worked on any other. Unlimited replay. A valid token could be used again and again. Two further failures showed controls that were too strict, not too lax. The WordPress plug-in rejected every login when its script did not load, which it never did, because the script's domain was unregistered. And the console, which should have been the most protected part of the service, had no login at all. Its "master key" was pre-filled and printed in the page's source. Every one of these controls looked enforced. The code had signature checks, header inspection and token formats. The landing page described the cryptography. The tests passed. None of that mattered, because no one had asked the controls to say no. The second version rebuilt each control around rejection. Its tokens are signed with a secret held only on the server. Each token is bound to the website and hostname it was issued for, expires after five minutes and is recorded on first use, so that a second use fails. Claims to be a search engine are checked against the address ranges that the search engines publish. Self-declared AI agents receive no special treatment unless their claim can be verified. The test suite asserts rejection for each of the following: forged, tampered, foreign, expired and reused tokens; tokens presented from the wrong hostname; spoofed crawlers; an older trick in which the browser chose its own, trivially easy, puzzle difficulty. It runs against the code that is actually deployed. The documentation lists the exact error codes the service returns for each failure, so a customer can test them. 4. 5. 6. 7. • • • •

Deny-first testing

The core discipline is deny-first testing: for every control, write and run the request that should fail before you trust the request that should pass. This is the Attack probe of Part III, in its simplest form. It requires no special tools, only the habit of asking "what should this refuse?" A deny-first test for a token check might send each of the following: no token; an empty token; a token of the right shape with random content; a token signed with the wrong key; a token issued for a different site; an expired token; a token that has already been used; a token with a valid signature but tampered contents. Each should be rejected, with a specific reason. If any passes, the control is unenforced.

Framework: the Deny-First Control Test

Control What it must refuse Rejection test Expected response Observed response Fail-safe behaviour Owner console Anyone without the owner's password Open console in a fresh browser Login page, no data If the password check errors, deny Verification check Forged, foreign, expired or reused tokens Send each kind Failure with specific error code If storage is unavailable, deny and report Crawler allow-list Requests claiming to be crawlers from unlisted addresses Send crawler user agent from an ordinary address Treated as unverified If ranges are stale, treat as unverified • • • • • • • •
Control What it must refuse Rejection test Expected response Observed response Fail-safe behaviour Plan limits Usage beyond plan Exceed the limit on a test account Over-limit response If counting fails, report, do not silently allow The test has two parts that leaders often skip: "Observed response" must be filled by someone who ran the test, not copied from the specification.

"Fail-safe behaviour" states what the control does when it cannot run.

Whether a control should fail open or fail closed is a business decision. For a login, failing closed protects security; for a contact form, failing open may protect revenue. Either way, it should be made deliberately and written down, not left to chance.

Points of debate

How much rejection testing is enough? Attackers are creative and test lists are finite. Does deny-first testing give false confidence of its own? Should a bot defence ever fail open? Blocking real customers costs revenue; admitting bots costs security. Who should decide, and on what evidence? Is self-declaration ever acceptable? Some standards rely on clients identifying themselves honestly. When is trusting the claimant reasonable? Should controls be judged by outsiders? If the builder writes the rejection tests, are they testing the attacks they imagined, or the attacks that will come?

Discussion questions

List the five most important controls in your product. For each, what should it refuse, and when did someone last confirm that it does? Have you ever discovered a control that admitted more than intended? How long had it been that way? Which of your controls trust what the request says about itself? • • • • • • 1. 2. 3.
Where would a "magic" value or debug mode be most dangerous in your systems, and how would you know if one existed? Who in your organisation decides whether controls fail open or closed?

Apply it

Complete the Deny-First Control Test for three controls, filling "Observed response" by actually running the tests. Search your code, or ask your team to, for special values that bypass checks: "test", "demo", "admin", "debug", "bypass", "magic". For each control, write and agree its fail-safe behaviour. 4. 5. 1. 2. 3.

8 · Dangling references

Opening question. What happens when your product points at something that does not exist, and someone else makes it exist? Software is full of names: domain names, package names, web addresses, plan identifiers, email addresses, file paths, the names of other services. Each is a pointer to something outside the product. When the thing pointed to exists and belongs to the right party, the name is unremarkable. When it does not exist, the name dangles. A dangling reference is not just broken; it is an opening. Anyone who creates the missing thing inherits the product's trust. This chapter explains why dangling references are among the cheapest gaps to find and the most dangerous to leave.

What counts as a dangling reference

A dangling reference is a name in a product that points to a resource that does not exist or is not controlled by the product's owner. Examples include: Unregistered domains, such as a content delivery address or a documentation site, that nobody has bought. Non-existent packages, where installation instructions name a library that is not published. Missing routes, such as a link to a page or interface that returns "not found". Placeholder identifiers, such as a plan code, account number or API key that exists on no platform. Unclaimed accounts, such as a social media handle or an email address mentioned but never created. Abandoned resources, such as a cloud storage bucket, subdomain or package that once existed and was deleted, leaving the name free. • • • • • •

Why dangling references are dangerous

A missing page is merely embarrassing. The danger comes from names that others can claim. Three lines of research make the point. Domain takeover. Liu, Hao and Wang (2016) studied "dangling" DNS records: entries pointing to cloud resources that had been released. Anyone who later acquired the resource could serve content under the original owner's name. They found such records at large organisations, including major universities and government agencies. Package attacks. Attackers publish malicious software under names that developers are likely to type or be told to install. Ohm et al. (2020) documented a large collection of real attacks on open-source package registries, including names chosen to resemble popular packages. Zimmermann et al. (2019) showed how far a single compromised package can reach in the npm ecosystem, because of how many other packages depend on it. Hallucinated packages. Code-generating models sometimes recommend packages that do not exist. Spracklen et al. (2025) tested 16 models on 576,000 code samples. At least 5.2% of the packages recommended by commercial models, and 21.7% of those recommended by open-source models, did not exist. They found 205,474 unique hallucinated package names. An attacker who registers a commonly hallucinated name can wait for developers, or their agents, to install it. AI adds a new source of dangling references. A model writing integration instructions, configuration files or documentation will produce names that look right: a CDN subdomain on the company's domain, a package under the company's namespace. Whether each name exists is a question about the world, not about language. The model can answer it only if it checks.

The AttestOS case

The first version contained four kinds of dangling reference, all found by the audit.
An unregistered content domain. The WordPress plug-in and several integration examples loaded their script from a subdomain of attestos.dev. The audit's domain lookup found that attestos.dev was not registered. That had two consequences: Immediate breakage. The script never loaded, and the plug-in's default setting rejected every login, locking site owners out. Latent takeover. Anyone could have registered the domain and served code to the login page of every site that installed the plug-in. Login pages are where passwords are typed. A non-existent package. The documentation told developers to install @attestos/ node from the npm registry. No such package existed. The name was free for anyone to publish, and anyone who did would have had their code installed by developers following the official instructions. Missing routes. The button for getting a free API key went back to the home page. The page meant to issue keys did not exist on the live service. Placeholder plan identifiers. Every checkout link pointed to a plan code of the form plan_... that existed on no payment platform. Notice the range of severity. The missing route and placeholder plans meant that nobody could become a customer, which is a commercial failure. The unregistered domain and non-existent package were security failures waiting to happen. All four were found by the same, very cheap, method. The second version removed every external name that it could not show it controlled. It now serves its widget script from the service itself. It publishes its integration examples as plain code against its own address, so there are no packages to install. Each paid plan's checkout link comes from a single setting, which is accepted only if it is an address on the payment platform's own domain. Until that setting exists, the service offers no checkout link for the plan, rather than a broken one. No file in the second version's repository mentions the unregistered domain. • •

The Resolve probe

Dangling references are found by the Resolve probe: take every name the product mentions and confirm that it resolves to something that exists and belongs to the right party. The tools are free and quick: Domain lookups (DNS and WHOIS) show whether a domain is registered and who controls it. Package registries show whether a package exists, who publishes it and when it was last updated. Following links, by clicking or with a link checker, shows which routes return "not found". Partner dashboards, such as the payment platform or email provider, show whether identifiers exist. The Resolve probe is the cheapest of the six probes and, in the AttestOS case, among the most productive. Running it across the first version would have taken under an hour and found several critical problems.

Framework: the Name Inventory

The Name Inventory turns the Resolve probe into a standing practice. Keep a list of every external name your product depends on: Name Kind Where it appears Resolves to Controlled by Verified on Action if lost examp le.co m Domain Website, emails Our hosting Us (registrar account X) Date Renew automatically; alert 60 days before expiry @ourc o/sdk Package Docs, README npm, version 2.1 Us (publisher account Y) Date Reserve the name even if deprecated plan_ ABC12 Payment plan Checkout links Pro plan, $49 Payment platform account Date Disable link if missing • • • •
Name Kind Where it appears Resolves to Controlled by Verified on Action if lost suppo rt@ex ample .com Email Footer, outreach Shared inbox Us Date Three rules apply: Every name has an owner. If nobody can say who controls a name, treat it as dangling. Names outlive features. When a feature is removed, keep its domains and package names registered, or remove every mention of them. Agents' names are checked before merge. Any new external name introduced by an agent or contractor is resolved before the change is accepted.

Points of debate

Whose job is the inventory? Domains often sit with marketing, packages with engineering, plans with finance. Who owns the whole list? Should agents be allowed to introduce new names at all? One option is to forbid agents from adding dependencies or domains without explicit approval. Is that practical? How long should names be kept? Holding domains and package names costs money and attention. When is it safe to let one go? Is a non-existent package worse than a buggy one? A buggy package harms by accident. A non-existent one invites deliberate harm. Should they be treated differently?

Discussion questions

How many external names does your main product depend on? Could anyone in your organisation produce the list today? When did someone last check that every domain and package in your documentation exists and belongs to you? 1. 2. 3. • • • • 1. 2.
Has your organisation ever let a domain or account lapse? What did it point to afterwards? How do you check names introduced by AI agents or contractors? Which dangling reference in your products would cause the most harm if someone else claimed it?

Apply it

Build a Name Inventory for one product. Include domains, subdomains, packages, plan identifiers, email addresses and social handles. Run the Resolve probe on every entry. Mark each as resolved and controlled, resolved but controlled by someone else, or dangling. For each dangling or foreign-controlled name, either claim it or remove every mention of it this week. 3. 4. 5. 1. 2. 3.

9 · Borrowed authority and test

theatre

Opening question. When a product cites a standard, quotes a statistic or shows a passing test, whose authority is it borrowing, and has it earned the loan? Some surfaces do not describe the product's own state or actions. Instead they borrow credibility from outside: A standard, as in "RFC-compliant". A measurement, as in "under 80 milliseconds". A comparison, as in "256 times costlier for bots". A process, as in "27 of 27 tests passing". Borrowed authority is persuasive because it seems to come from someone other than the seller. It is dangerous for the same reason: readers rarely check the lender. This chapter covers two related gaps. False claims invoke an authority that does not support them. Test theatre presents verification that verifies nothing.

False claims

A false claim is a statement about the product or the world that is untrue or unsupported. False claims become especially troublesome when they cite an outside authority. The common forms include: Wrong citations. A real standard is cited for something it does not cover. Invented authorities. A standard, certification or study is cited that does not exist. Unsupported measurements. A speed, accuracy or saving is stated without measurement, or contradicted by one. • • • • • • •
Misleading arithmetic. A ratio or multiplier is technically derivable but implies something false. Inconsistent facts. Prices, limits or features differ between pages, so at least one must be wrong.

Why false claims arise, and why AI changes the odds

Marketing has always stretched the truth. What is new is the fluency with which plausible authority can be produced. Language models have read many documents that cite standards, report speeds and quote multipliers. Asked to describe a product, they can produce text with the same texture, and the texture is easily mistaken for substance. Research on language models documents the relevant tendencies: Hallucination. Models produce fluent statements that are not supported by their sources or by the world (Ji et al., 2023). Deception without intent. Park et al. (2024) surveyed cases in which AI systems produced false beliefs in others, whether or not anyone designed them to. Sycophancy. Models tend to tell users what they appear to want to hear (Sharma et al., 2023). A builder who wants a product to sound impressive may get exactly that. None of this establishes how any particular false claim arose. People make such errors too, under deadline, by copying from a competitor or by misremembering a reference number. The point is that the cost of producing confident, specific, wrong claims has fallen. Checking them costs what it always did.

The AttestOS case: false claims

The audit found ten false or unsupported claims in the first version. Six were rated high severity. Four examples show the range. A wrong standard. The product described its hardware token feature as implementing "RFC 9505 Private Access Tokens". RFC 9505 is a real document, but it is A Survey of Worldwide Censorship Techniques (Hall et al., 2023). The • • • • •
standards for Privacy Pass, the protocol behind private access tokens, are RFC 9576, 9577 and 9578 (Davidson, Iyengar & Wood, 2024; Pauly, Valdez & Wood, 2024; Celi et al., 2024). The wrong number appeared everywhere the feature was described. An invented standard. The crawler-policy module claimed to follow an "RFC robots.ai" or "RFC-compliant agent policy standard". No such standard exists. The real standard for telling crawlers what they may fetch is the Robots Exclusion Protocol, RFC 9309 (Koster et al., 2022). Impossible arithmetic. The console offered difficulty settings labelled "Heavy ~1.2 s" and "Fortress ~15 s". Because each difficulty step multiplied the work by 16, these settings required about 16.7 million and 4.3 billion computations. That is minutes and hours in a browser, not seconds. The "Emergency Lockdown" setting required about 2⁶⁴ computations, which is impossible in practice. A flattering multiplier. The landing page promised a "256×–4096× bot CPU tax". The audit estimated that the default puzzle took a browser 0.3–1 second, and native attacker code about ten milliseconds. Whatever the multiplier measured, it was not protection. Chapter 17 returns to what proof-of-work can honestly claim. Other examples included "under 80 milliseconds", "zero quota consumed", "100% autonomous", "GDPR compliant" and prices that differed between the configuration and the landing page.

Test theatre

Test theatre is the presentation of verification that does not verify the product being relied on. Its forms include: tests of code that is never deployed; tests that check only that a page renders or a message appears, not that the effect occurs; tests that rely on special values which bypass the logic under test; passing badges that are typed in, out of date or computed from a different suite; tests that check implementation details while ignoring outcomes. • • • • •
Research on software testing explains why test theatre is easy to fall into. Barr et al. (2015) surveyed the "oracle problem": a test is only as good as its way of deciding whether the output is correct, and for many programs no good oracle exists. Spadini et al. (2017) studied the use of mocks, stand-ins for real components, and found them widespread. They also found that developers do not always appreciate what mocking stops a test from checking. A test that replaces the database, the payment platform and the mail service with mocks checks the code's internal choreography, not whether anything works. Tests carry more authority than their content warrants. A passing badge looks like the verdict of an impartial machine. But the machine checks only what the tests ask, and the tests ask only what their author thought to ask. When the same author, human or AI, writes both the product and its tests, the tests can encode the same misunderstandings. Power (1997) warned of "rituals of verification", and a green badge can be exactly that.

The AttestOS case: test theatre

The first version displayed a badge reading "Tests 27/27 passing". The test suite actually contained about 30 tests. All of them tested the larger application in the repository that had never been deployed, not the single file that ran in production. Some tests asserted that special "demo" signatures unlocked paid features, so they tested the bypass, not the protection. One component's tests crashed because the file they wrote to did not exist. The second version's 63 test cases run against the code that is actually deployed, using a working copy of the real database engine. They assert rejections as well as successes (Chapter 7 lists them). One test file reads the public brand facts and compares them with the prices, limits and site counts that the product actually enforces. If a page claims something the code does not do, the test fails. Chapter 16 describes that mechanism.

Framework: the Source Check and the Test Provenance

Check

The Source Check. For every claim of standard, statistic, price, speed, comparison or compliance, record:
Claim Where it appears Source or calculation Checked by Result "Private Access Tokens (RFC 9578)" Docs, pricing RFC Editor entry for RFC 9578 Name, date Correct "16 bits ≈ 65,000 hashes" Docs 2¹⁶ = 65,536 Correct "Phone does 0.5–2 million hashes per second" Docs Development measurement; range is an estimate Labelled as estimate "GDPR compliant" Footer ? No basis: remove The rules are strict: Every claim needs a source you have read, not a source that sounds right. Calculations are shown, so that anyone can redo them.

Estimates are labelled as estimates.

A claim without a source is removed. It is not softened. The Test Provenance Check. For the test suite behind any "passing" claim, answer five questions: Which deployed code paths does each test exercise? Can you show the mapping? Which tests assert that something is refused? Which external services are replaced by mocks, and what does that prevent the test from checking? Does any test rely on a special value or mode unavailable in production? Is the badge computed automatically from the latest run of this suite against this code? A suite that cannot answer the first question is test theatre until proved otherwise, however many tests it contains. • • • • 1. 2. 3. 4. 5.

Points of debate

Is marketing language held to the same standard? "Blazing fast" is not a measurement. Should trust products avoid unmeasurable adjectives entirely? Can AI check AI's citations? A second model can look up RFC numbers. But if it shares the first model's tendencies, does it add assurance? Are mocks bad? Mocks make tests fast and reliable. When does a mock become theatre? Who should write the tests? Separating the authors of product and test reduces shared blind spots, but it costs time. Is it worth it for agent-built code?

Discussion questions

Find three claims on your product's public pages that cite a standard, statistic or comparison. Can you produce the source for each within ten minutes? Does your organisation's test badge mean what a buyer would assume it means? What proportion of your tests assert refusals? Has a passing test suite ever given you confidence that turned out to be false? What did the tests not check? If an agent produced your documentation, how would you check every factual claim in it?

Apply it

Run the Source Check on every factual claim on one page of your website or documentation. Run the Test Provenance Check on your main product's test suite. Map at least ten tests to the deployed code paths they exercise. Remove or correct every unsourced claim, and add one refusal test for each important control that lacks one. • • • • 1. 2. 3. 4. 5. 1. 2. 3.
PA R T I I I

The mechanism audit

Proof of Mechanism

10 · The six probes

Opening question. If surfaces can no longer be trusted, what small set of checks can a leader rely on to see behind them? Part II described the forms that the gap between surface and mechanism takes. Part III turns to finding them. This chapter introduces six probes. Each is a simple, repeatable check that tests a surface against the world rather than against itself. Together they found every claim–mechanism gap in the AttestOS audit. None requires specialist tools, and most can be run by a non-engineer with a browser and some patience.

Why probes rather than reviews

The usual way to judge software is to review it. Reviews take several forms: reading code; watching demonstrations; reading documentation; reviewing test reports. Reviews examine the artefact on its own terms. That is their weakness when the artefact is persuasive. A reviewer reading a well-written dashboard component sees well-written code. Only by asking "where does this number come from?" do they find that it comes from nowhere. A probe is different. It makes the artefact do something and observes the result through a channel the artefact does not control. Probes resemble experiments more than readings. They draw on a long tradition in software engineering of testing behaviour rather than inspecting structure, and on the security practice of trying to break a system rather than admiring it. • • • •
The six probes were developed in the AttestOS audit and are proposed here as a general method. They are not a standard and have not been independently validated. Each probe corresponds to one question that a surface invites.

The six probes

Probe Question Method Typical finding Trace Where does this come from? Follow a displayed value to its source; reset data and see if it changes Fabricated state Trigger Did this actually happen? Activate a control; look for its effect outside the screen that confirmed it Theatrical actions Vary Does this depend on its input? Give two materially different inputs; compare outputs Synthetic evidence, invented analysis Resolve Does this exist, and is it ours? Look up every domain, package, route, identifier and account Dangling references Attack Does this refuse what it should? Send the request that should fail Unenforced controls Source Who says so? Check every claim against its cited source or recompute it; check what tests actually test False claims, test theatre

Trace

Pick a number, label or status. Ask where it comes from and follow the answer. If the answer is a query, run it. If the answer is a file, open it. If no one can answer, the value is fabricated until shown otherwise. Quick versions: Load the screen on an empty test account; real values should be zero or empty. Perform one countable action and confirm that the value moves by exactly one.

Trigger

Press the button, submit the form, flip the switch. Then look elsewhere: The database: is there a new record? The partner system: did the payment platform see the charge, did the mail service accept the message? • • • •
The protected resource: does the lock now refuse the key? Quick version: reload. A saved change that does not survive a reload was never saved.

Vary

Give the feature two inputs that should produce different outputs and compare. A scanner should report different things about different websites. A risk score should differ between a clean request and a suspicious one. A personalised message should differ between recipients in the ways that matter. Quick version: scan your own website and a blank page. If the reports match, the scanner is not scanning.

Resolve

List every external name: domains, subdomains, package names, routes, plan identifiers, email addresses, social accounts. Look each one up. Confirm that it exists and belongs to you. Quick version: look up the domains in the product's script tags and integration instructions, and search the package registry for every install command in the documentation.

Attack

For every control, send the request that should be refused: the wrong password; the forged token; the expired token; the reused token; the request from the wrong origin; the request that claims a privilege it does not have. Quick version: open the admin console in a fresh private browser window. If you see anything but a login page, stop and fix it. • • • • • • •

Source

For every claim of standard, statistic, comparison, price or compliance, open the source and read it, or redo the calculation. For every passing test, find which deployed code it exercises. Quick version: look up every standard number cited in the documentation in the official registry. For internet standards, that is the RFC Editor.

The probes in the AttestOS audit

The audit of the first version used all six. The table shows which probes found which kinds of problem. Some findings were confirmed by more than one probe. Probe Examples of what it found Trace All console figures typed into the page; portal usage invented for any visitor Trigger Lockdown, tenant provisioning, key rotation and "Dispatched" left no trace Vary Scanner returned identical verdicts for every address; scout's "vulnerability scores" derived from a scrambled domain name Resolve Unregistered domain behind the plug-in; non-existent package; missing key page; placeholder checkout plans Attack Seven bypasses confirmed against the live service; console open to all Source Wrong RFC; invented standard; impossible difficulty labels; tests that exercised undeployed code Of the 65 findings, 47 (72%) were claim–mechanism gaps: places where a surface asserted something that the mechanism did not do. The remaining 18 were ordinary engineering problems: a slow script, an untidy repository, weak logging. Every one of the 18 critical findings and every one of the 21 high-severity findings was a claim–mechanism gap. The whole audit was completed within a day. The probes also found problems after the rebuild. On 28 October, live checks in a browser, a review of the documentation against the code and the owner's own use of the console led to fixes for eleven further issues. Six were claim–mechanism gaps: documentation that described replay protection inaccurately; documentation listing widget settings that the widget did not read; • •
an error table listing codes the verifier never returned; a description of the rule that raises puzzle difficulty during traffic surges that did not match the code; a site header that said "Log in" to people who were already signed in; distribution channels labelled "active" when they had no accounts to post from. The last of these was found by the owner, not by any check. All six were small and were fixed the same day. Their existence matters: probes are not a one-time cleansing but a standing practice. Chapter 19 returns to this.

How the probes fit together

The probes form a sequence that runs roughly from cheapest to most demanding: Resolve and Source need only public information and a search engine. Trace and Trigger need access to the product and some patience. Vary needs two well-chosen inputs. Attack needs the most care. It should be run only against systems you own or have permission to test, and with knowledge of what "refused" should look like. A leader with an hour can run Resolve and Source across a whole product, and Trace, Trigger and Vary on its most important screens. Chapter 11 sets out that hour.

Framework: the Six-Probe Audit

Step Action Output 1. Inventory List the product's surfaces using the Surface–Mechanism Ledger (Chapter 1) Surface list 2. Assign Mark each surface with the probe(s) that test it Probe plan 3. Prioritise Order by Credence Exposure (Chapter 4) and cost (Chapter 2) Ordered list 4. Probe Run each probe; record expected and observed results Evidence log 5. Classify Label each failure by gap type (Part II) and severity (Chapter 13) Findings • • • • 1. 2. 3. 4.
Action Output 6. Decide Repair, rebuild, relabel or remove (Chapter 14) Action plan 7. Repeat Re-run probes after every significant change Standing practice

Points of debate

Are six probes enough? Performance, accessibility, privacy and resilience are not covered. Should they be added, or are they different questions? Can probes be automated? Several probes could be written as automated checks. Does automation risk turning probes into another ritual? Who should run the probes? The builder knows where to look but has the strongest bias. An outsider is unbiased but less informed. What mix is right? Is the 72% figure meaningful? It comes from one audit of one product by one process. What would you need to see before generalising it?

Discussion questions

Which of the six probes does your organisation already run routinely? Which never? Which probe would find the most problems in your main product? Why do you think so? Who in your organisation has the access needed to run each probe? How would you know whether an audit of your product had been thorough? What would it take to make the probes a standing practice instead of a onetime exercise?

Apply it

Pick one product. For each of the six probes, write down three specific checks you would run. Run all eighteen checks in one sitting. Record expected and observed results. Classify each failure by gap type and present the results to your team. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

11 · The one-hour outsider

audit

Opening question. If you are a buyer, investor, board member or journalist with no access to a company's systems, how much can you learn in an hour? The six probes of Chapter 10 were designed for an auditor with full access: the code, the database, the deployment. Most people who rely on a product have none of that. A buyer sees a sales demonstration and a trial account. An investor sees a pitch and perhaps a live walkthrough. A board member sees reports. A journalist sees the public website. This chapter argues that the outsider's position is much stronger than it seems. In the AttestOS case, an outsider could have established almost every critical problem in under an hour.

What an outsider can see

Even without privileged access, an outsider usually has four things: The public website and documentation, including every claim, price, link and instruction. The public interfaces that the product's own pages use. A web page that calls a service reveals the service's address and the shape of its requests to anyone who opens the browser's developer tools. Public registries, such as domain lookups, package registries, standards bodies and company registers. A trial or demonstration account, if one is offered, or the demonstration itself. 1. 2. 3. 4.
These are enough to run five of the six probes in limited form. Resolve and Source need nothing more. Vary needs two inputs to a public feature. Trace and Trigger can be run on a trial account. Attack is possible in principle against public interfaces, but an outsider should run it only with the owner's permission, and on their own account.

The AttestOS evidence

Of the 18 critical findings in the audit of the first version, 16 could have been established by an outsider using only those four resources: A domain lookup showed that the domain serving the plug-in's script was unregistered, which also revealed the takeover risk. A package registry search showed that the server package the documentation named did not exist. Following the "Get free key" button and the checkout links showed that no key could be obtained and no plan purchased. Scanning two different websites produced identical reports, so the scanner was not scanning. Opening the console's public address showed the owner's view, with no login. Reloading showed that its figures never changed and that its actions left no trace. Opening the portal with an invented site key showed full usage figures. Sending crafted requests to the public verification interface, on a site you control, would have confirmed the bypasses. Only two critical findings needed the source code: the signing secret and the synthetic findings in the outreach scout. Because the repository was public, even those were visible to anyone who looked. The sociological literature on decoupling suggests why such checks are so rarely run. Meyer and Rowan (1977) observed that organisations, and their audiences, protect formal structures through a "logic of confidence": everyone assumes that the structure reflects the work, and close inspection would seem rude. A fluent interface invites the same confidence. Research on automation shows that people tend to accept what a polished system tells them (Parasuraman & Manzey, 2010). • • • • • • •
Trust research adds that we extend trust on the basis of perceived ability, benevolence and integrity (Mayer et al., 1995), and that people form initial trust in an unfamiliar vendor largely from cues such as the quality of its website (McKnight et al., 2002). A professional-looking product scores well on all of these cues before anyone has checked anything.

The one-hour plan

The following plan is a proposal, built from the AttestOS audit. It is designed for a non-engineer and takes about an hour. Each step names the probe it uses. Minutes 0–10 · Resolve the names. Open the website, the documentation and any integration instructions, and list every domain, subdomain and package name. Then: Look up each domain in a public WHOIS or DNS service. Is it registered? To whom? Search the relevant package registry for each install command. Does the package exist? Who publishes it? Minutes 10–20 · Follow the money and the sign-up. Click every "Start", "Get key", "Buy" and "Upgrade" button. Does each lead to a working sign-up or checkout? Does the checkout show the plan and price the website advertised? Do prices match across pages? Minutes 20–30 · Source the claims. Pick the three most impressive claims on the site: a standard, a statistic, a comparison. Look up each standard number in the official registry. Does the title match the claim? Ask where each statistic comes from. Is there a methodology? Minutes 30–40 · Vary the analysis. If the product analyses anything, such as websites, documents, code or customers, give it two very different inputs. Do the outputs differ in ways that make sense? • • • • • • • •
Does an empty or nonsense input produce a confident result? Minutes 40–50 · Trace and Trigger on a trial. If a trial account exists, create one. Then: Do the dashboard figures start at zero? Perform one action and see whether a figure moves by one. Change a setting, reload and see whether it persisted. Minutes 50–60 · Ask for the rejection. Ask the vendor, or test on your own trial site: What happens when a forged or reused token is presented? What error does the product return? Is the error documented? A trustworthy vendor will be able to show you.

What the outsider audit cannot do

The one-hour audit is a screen, not a certificate. It finds gaps that show through the surface, and many of the most serious ones do. But it cannot see everything: Internal correctness. It cannot establish that data is stored securely, that backups exist or that code paths not exposed to the public work. Performance under load. Behaviour under heavy use or real attack is beyond an hour's testing. Clever concealment. A determined deceiver can pass outsider probes. The Theranos story shows how far that can go (Carreyrou, 2018). The outsider audit catches simulated completeness, not fraud designed to defeat inspection. Permission limits. Attack probes against systems you do not own may be unlawful. The outsider should ask, or use their own account and site. • • • • • • • • • • •

Making outsider probes easy: the vendor's side

The AttestOS case also shows the converse. An honest vendor can make outsider probes easy and treat the results as marketing. Since the rebuild, a prospective customer can do the following: Create a free account and see a dashboard that starts empty. Send a forged, foreign, expired or reused token to the verification interface and receive a specific, documented error. Look up every domain mentioned in the integration instructions and find only the live service. Read the price list from the same interface that enforces it. Scan two websites and receive two different reports, each listing what was actually found on the page. Read the source code. None of this proves that the product will stop a determined attacker; no outsider check can. But it establishes that the surfaces tell the truth about the mechanism, which is the question the one-hour audit exists to answer. Chapter 20 develops verifiability as a business strategy.

Framework: the One-Hour Outsider Audit

Time Probe Action Red flag 0–10 Resolve Look up every domain and package Unregistered, or registered to someone else 10– Trigger Follow every sign-up and checkout Dead ends, placeholder plans, price mismatches 20– Source Check three headline claims Wrong or unverifiable citations 30– Vary Two different inputs to any analysis Identical or confident-onnonsense output 40– Trace / Trigger Trial account: zero state, one action, one setting Non-zero empty state; nothing moves; nothing persists 50– Attack (with permission) Ask for or test a rejection No documented errors; vendor cannot show a refusal • • • • • •
Record the results. Three or more red flags should halt reliance on the product until the vendor has answered each one with evidence.

Points of debate

Is it fair to vendors? Early-stage products have rough edges. Does the outsider audit punish start-ups for honest incompleteness?

Should outsider audits be standard in procurement and due diligence?

Investors commission financial and legal due diligence. Should a one-hour mechanism audit be routine? What about permission? Even mild Attack probes may breach terms of service. How should an outsider proceed? Can a vendor game the outsider audit? If the one-hour plan became widely known, would vendors build surfaces designed to pass it?

Discussion questions

When your organisation last bought software, what checks did it run beyond the demonstration? Could an outsider find your product's biggest gaps in an hour? What would they find? What evidence could your product offer an outsider that cannot be faked? How would you respond to a prospective customer who ran this audit on your product? Should board members be expected to run outsider probes on their company's products?

Apply it

Run the One-Hour Outsider Audit on your own product, as if you were a sceptical buyer. Run it on a vendor you currently rely on. For each red flag on your own product, write the evidence that would answer it, and publish what you can. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

12 · Synthetic evidence and the

third party

Opening question. What changes when a product's invented claims are not about itself, but about someone else? Most of the gaps in Part II harm the product's own users or owner. A fabricated dashboard misleads the operator; an unenforced control fails the customer. This chapter turns to a more serious category: claims a product makes about other people, such as an analysis of their website, a score for their security or a notice sent to their staff. When those claims are invented, the harm reaches people who never chose to rely on the product. The legal and ethical stakes rise with it.

What counts as synthetic evidence

Synthetic evidence is output presented as the result of examining something (a website, document, dataset, person or organisation) that was not derived from examining it. Two forms recur: Static verdicts: the same result shown whatever the input. Pseudo-personalised verdicts: results that differ by input, but are computed from something irrelevant to the claimed analysis, such as a random number, a hash of a name or the length of a string. The second form is more dangerous, because it passes casual inspection. Two different websites get two different scores, so the scanner seems to be looking.

Why it matters most for third parties

Claims about third parties differ from claims about oneself in three ways. • •
The subject did not consent. A customer who trusts a dashboard has at least chosen the product. A company told that its website is vulnerable did not choose to be assessed. It may act on the claim: investigating, escalating, paying for a fix. The claim may reach others. A shareable report or public scan result can circulate. An invented finding about a company's security, once shared, can damage its reputation. The law takes it seriously. Many jurisdictions prohibit unfair or deceptive commercial practices. In the United States, the Federal Trade Commission's "Operation AI Comply" in 2024 brought five actions against firms using claims about artificial intelligence to deceive (Federal Trade Commission, 2024). One firm had promoted a "robot lawyer" without testing whether its output matched a lawyer's, and it paid $193,000. Commercial email in the United States is governed by the CAN-SPAM Act (CAN-SPAM Act, 2003). The Act prohibits deceptive subject lines and headers, and requires a valid postal address and a working optout. A cold email headed "[Security Notice]" that reports invented findings would invite scrutiny on several of these grounds at once.

Security notices are a special case

Security teams receive real vulnerability reports through dedicated channels. RFC 9116 defines a standard file, security.txt, in which organisations publish where to send them (Foudil & Shafranovich, 2022). Those channels work because senders are generally acting in good faith: a researcher found something and is reporting it. A message that imitates a vulnerability report in order to sell something exploits that good faith. If such messages become common, they degrade a channel that the whole internet relies on.

The AttestOS case

The scanner. The first version offered a free "five-second scan" of any website, the product's main way of attracting customers. The audit found that the scanner's back end did not exist on the live service; the request returned "not found". The page then displayed fixed findings regardless of the address entered: "100% Solvable",
"VULNERABLE", "Human users fail this test 18% of the time". It told every visitor their site was vulnerable, even sites with no challenge at all. This was the static form of synthetic evidence. The scout. The outreach engine, part of the larger application that was never deployed, was more troubling. It drafted emails headed "[Security Notice]" to the security@, appsec@ and engineering@ addresses of real companies. Each message reported a vulnerability "score", a bypass speed and an estimated monthly loss. All were computed by a function called generateSyntheticAudit from a cryptographic hash of the company's domain name. Different companies got different figures, which made the messages look tailored. The messages also claimed that "our engine recorded an environment integrity gap" and referred to "a private, reproducible audit trace". No scan took place. The contact addresses were guessed, not taken from the companies' security.txt files as the documentation claimed. The reply addresses sat on the unregistered domain, so replies would have gone to whoever registered it. The scout was not wired to send. Its console's "Dispatch" button, as Chapter 6 described, said that it had. Whatever its origin, the design was dangerous. Had someone connected it to a mail service, it would have sent invented security findings to real security teams, in the venture's name. The rebuild. The second version rebuilt both features around observation. The scanner now fetches the page, its robots.txt file and its security.txt file. It reports only what it found: which of 13 known verification vendors appear on the page, whether the page shows a visible challenge, what crawler rules are published and whether a security contact exists. It refuses to scan private or unusual addresses. Each report can be shared at its own address, so the claim can be checked by anyone. Outreach is built only from those facts. A message says what was observed on the recipient's own page, such as "your sign-up page shows a visible image challenge". It names the page, offers the free scanner and includes a one-click unsubscribe. Contact addresses must be on the recipient's own domain. There is one follow-up at most. A reply stops the sequence and goes to the owner, who answers personally. Nothing is sent until the owner switches outreach on and sets the postal address the law requires.
A test asserts that the message template "never claims more than we saw". At the end of the observation window, ten prospects had been identified, all showing the same widely used visible challenge. None had been contacted.

The Vary probe in practice

Synthetic evidence is found by the Vary probe. Give the analysis two inputs that should produce clearly different results, and include at least one input for which the right answer is known. Static verdicts show identical output for both. Pseudo-personalised verdicts need a sharper test: Same subject twice. Analyse the same subject twice after changing something material about it. Does the result change in the right way? Known answer. Analyse a subject whose properties you know, such as your own website, and compare the report with reality. Nonsense input. Analyse a blank or nonsense input. A confident result is a red flag. For third-party claims, add a fourth check: Trace the evidence. Every claim in an outbound message should link to the stored observation it came from.

Framework: the Third-Party Claim Gate

Before any product makes a claim about someone else, whether a report, score, message or listing, it should pass through a gate. Every condition must be met: Gate Condition AttestOS second version 1. Observed The claim is derived from an observation of the subject, stored with time and source Scan record with page, vendor, timestamp 2. Checkable The subject can verify the claim themselves Message names the page and the visible element; public report link 3. Proportionate The claim says no more than the observation supports Template states only vendor and page type; test enforces this • • ◦ ◦ ◦ •
Condition AttestOS second version 4. Right channel The message goes to an appropriate address, not one that implies a different purpose On-domain contact only; no imitation of vulnerability reports 5. Lawful Legal requirements for the channel are met Postal address required before any send; unsubscribe; suppression list 6. Human on replies A person handles responses Replies stop the sequence and go to the owner A product that cannot pass every gate should not make the claim. "We'll fix it later" does not apply here, because the harm falls on someone who never agreed to wait.

Points of debate

Is pseudo-personalisation ever acceptable? Marketing personalisation often uses weak signals, such as industry or company size, to tailor messages. Where is the line between tailoring and fabricating? Should cold outreach about security be allowed at all? Even truthful observations about a company's website, sent unsolicited, can feel intrusive. Is "your page uses a visible challenge" information or pressure? Who is liable for an agent's message? If an AI agent drafts and sends an invented claim, does responsibility lie with the operator, the agent's vendor or both? Does "not wired to send" matter? The scout never sent anything. Is a dangerous capability that is not connected a finding, or only a risk?

Discussion questions

What claims does your product make about people or organisations who are not your customers? Could any of your analysis features produce confident output from nonsense input? How does your organisation check outbound messages generated by automation or AI? Which of the six gates would your outreach fail today? • • • • 1. 2. 3. 4.
If a recipient challenged a claim in one of your messages, could you produce the observation it came from within an hour?

Apply it

List every third-party claim your products and outreach make. Run the Vary probe on each analysis feature: two inputs, one known answer, one nonsense input. Put every outbound message type through the Third-Party Claim Gate, and block those that fail. 5. 1. 2. 3.

13 · Reading an audit as a

leader

Opening question. When an audit lands on your desk with sixty-five findings, how do you decide what matters? Audits overwhelm. A thorough review of any real product produces dozens of findings of mixed importance, written in technical language and sorted by the auditor's categories, not the leader's decisions. Many leaders respond in one of two unhelpful ways. Some skim for the word "critical" and delegate the rest. Others treat every finding as equally urgent and stall. This chapter offers a way to read an audit that is fast, honest and tied to decisions. Its central idea is to sort findings by kind as well as by severity.

Severity is necessary but not sufficient

Most audits rate findings by severity, typically critical, high, medium and low, according to the harm that could follow and how likely it is. Severity answers the question "how bad?" It does not answer three other questions a leader needs: What is wrong at root? Is it a bug in a working mechanism, or a surface with no mechanism behind it? What is the remedy? Is it a fix, a rebuild, a relabel or a removal? What does it say about the process that produced the product? Is it a random error, or a pattern? The second dimension that answers these questions is kind. In the AttestOS audit, findings fell into two broad kinds: Claim–mechanism gaps, where a surface asserted something the mechanism did not do. Part II divided these into eight types: unenforced controls, missing mechanisms, dangling references, fabricated state, theatrical actions, synthetic evidence, false claims and test theatre. • • • •
Hygiene problems, where the product did what it claimed but did it poorly: slow, untidy, fragile or hard to maintain. The distinction matters because the remedies differ. A hygiene problem is fixed by improving the mechanism. A claim–mechanism gap has three possible remedies: build the mechanism, change the surface to tell the truth, or remove the surface. Choosing among them is a business decision, not an engineering one.

The AttestOS audit by severity and kind

The audit of the first version produced 65 findings: 18 critical, 21 high, 17 medium and 9 low. Sorted by kind, the picture was stark. Gap type Critical High Medium Low Total Unenforced control Missing mechanism Dangling reference Fabricated state Theatrical action Synthetic evidence False or unsupported claim Test theatre All claim–mechanism gaps Hygiene Total Three things stand out. Every critical and high finding was a gap. All 39 of the most serious findings were places where the product claimed something it did not do. None were ordinary bugs. This pattern is a diagnosis in itself: the product's problem was not quality but truthfulness. Hygiene was all medium or low. The 18 hygiene findings included a slow script, an untidy repository and duplicated code. They were real, but they were not why the product was dangerous. •
The gap types cluster. Unenforced controls and dangling references accounted for most critical findings, because they create security exposure. Missing mechanisms dominated the high findings, because they meant the product could not be used: no keys, no payments, no records.

Reading for pattern

A leader should read an audit for pattern, not just priority. If most findings are hygiene, the product works and needs care. If most serious findings are gaps, the product's surfaces cannot be trusted, and the process that produced it needs attention as much as the product does. Pattern also tells you where to look next. The audit found gaps in some surfaces; the same process probably produced gaps in surfaces the audit did not cover. In the AttestOS case, the follow-up review after the rebuild found six further gaps, all in surfaces written separately from the mechanisms they described: documentation, status labels and a header. The pattern predicted where they would be.

Questions to ask the auditor

Leaders should question an audit as they would question any report. Useful questions include: Method: Which probes were run? What was outside scope? Was the live system confirmed to match the code reviewed? Evidence: For each critical finding, what request or observation demonstrated it? Can it be reproduced? Confidence: Which findings are confirmed and which are inferred? Coverage: Which surfaces were not examined? Independence: Who conducted the audit, and what is their relationship to the product? The last question applies to this book's case. The AttestOS audit was carried out by the owner working with AI agents, not by an independent party. Its findings were demonstrated against the live system and recorded in detail, but it is not an independent evaluation. A leader relying on such an audit should weigh it accordingly. • • • • •

Framework: the Severity × Kind Triage Grid

The grid sorts findings into four action zones. Claim–mechanism gap Hygiene Critical / High Stop. Take the surface down or disable the feature now. Then decide: build, relabel or remove. Review the process that produced it. Fix promptly. Schedule within days; monitor. Medium / Low Relabel or schedule. Correct the surface to tell the truth today; build the mechanism if it is worth it. Backlog. Fix in the normal course of work. Two rules accompany the grid: Gaps in high-exposure products move up one row. A medium gap in a trust product (Chapter 4) is treated as high. The surface is corrected before the mechanism is built. If building the mechanism will take weeks, change the surface now so that nobody relies on it in the meantime. Honest incompleteness is always available as an interim state.

A note on emotion

Audits of one's own work are uncomfortable, and audits revealing that a product was mostly surface are especially so. Leaders set the tone. If an audit is treated as an accusation, the next one will be resisted, softened or avoided. If it is treated as information, the organisation learns. Reason (2000) argued that safety in complex systems depends on a culture that reports errors instead of hiding them. The same applies to simulated completeness. The definition in Chapter 3 deliberately excludes intent, so that gaps can be discussed without blame.

Points of debate

Is "kind" objective? Classifying a finding as a gap or as hygiene involves judgement. Could two auditors disagree? Should a single critical gap condemn a product? If one control is unenforced and everything else is real, is the product untrustworthy, or merely flawed? Can an owner audit their own product? The AttestOS audit was not independent. What does an owner-led audit offer, and what does it lack? 1. 2. • • •
Does a pattern of gaps imply a pattern of intent? If most serious findings are gaps, should leaders suspect deception, or does the definition rightly set that aside?

Discussion questions

Take the last audit or review of one of your products. How many findings were gaps and how many were hygiene? What pattern would most worry you in an audit of your main product? Who in your organisation is qualified to classify findings by kind? How does your organisation respond emotionally to audits? How would you change that? What would you do if an audit found that your product was mostly surface?

Apply it

Re-sort your most recent audit or bug list using the Severity × Kind Triage Grid. For every gap in the "Stop" zone, decide by the end of the week whether to build, relabel or remove. Write down the pattern the audit reveals, and one change to your process that addresses it. • 1. 2. 3. 4. 5. 1. 2. 3.

14 · Rebuild or repair

Opening question. When a product turns out to be mostly surface, is it better to fix what is there or to start again? Few decisions in software provoke stronger feelings than the choice between repairing a system and rebuilding it. Rebuilds have a bad reputation: they take longer than planned, lose features that nobody wrote down and sometimes never finish. Repairs have a quieter failure mode: they preserve structures that keep producing the same problems. This chapter offers a structured way to make the decision when the problem is simulated completeness, and shows how it was made in the AttestOS case.

Why the usual advice may not apply

Conventional wisdom in software engineering favours repair. An existing system, however untidy, contains knowledge: edge cases handled, bugs fixed, behaviours that users depend on. Rewriting throws that knowledge away. The advice is sound when the system works. Its value lies in its mechanisms, and its problems are hygiene. Simulated completeness reverses the premise. When most of a product's serious problems are claim–mechanism gaps, the existing system's value lies largely in its surfaces: its screens, copy, documentation and design. The surfaces are the part that cannot be trusted. The knowledge that repair would preserve is the knowledge of how to look finished. The knowledge of how to be finished was never there. That does not make rebuilding automatically right. It does change the question. Instead of "how much working knowledge would we lose?", the leader asks "how much of what exists is real, and can the real part bear weight?"

Four questions

The decision turns on four questions, proposed here from the AttestOS case and offered for testing on others. 1. What proportion of serious findings are gaps? If most critical and high findings are claim–mechanism gaps, repair means building most mechanisms anyway. It also means preserving surfaces that are themselves false. In AttestOS, all 39 critical and high findings were gaps. 2. Can the foundation hold the mechanisms you need? Some foundations cannot be repaired into soundness. The AttestOS first version ran in its hosting platform's legacy script format, with no database. The audit concluded that the most critical finding, a signing secret printed in public code, could not be fixed without rewriting the service in the platform's current format, with secrets and a database bound to it. 3. Are surface and mechanism separable? If the real logic can be extracted and the surfaces rebuilt around it, repair is plausible. If the two are tangled, it is not. In AttestOS, the deployed file was more than three-quarters embedded images. The larger application, which held most of the apparent logic, had never been deployed and was itself largely mock. 4. What is the cost of being wrong while you repair? A product in use, with customers relying on it, may not be able to stop. A product with no customers can be taken down, rebuilt and relaunched without harm to anyone. AttestOS had no customers and had taken no payments, so the cost of rebuilding was time alone.

Depth of remedy

Before choosing between repair and rebuild, a leader must choose how far to go. The AttestOS audit offered three depths: Safe and honest. Close the security exposures, remove or correct every false surface, take down what cannot be made true. The product becomes smaller but trustworthy. Free tier real. Additionally, make the free product genuinely work: sign-up, keys, verification and the scanner. 1. 2.
Paid tier real. Additionally, make every paid module genuinely work: payments, entitlements, device tokens, passkeys and crawler policy. Depth 1 is always available and is often the right first step even if deeper work will follow. It converts simulated completeness into honest incompleteness. The owner of AttestOS chose depth 3, with rebuild as the method.

The rebuild as executed

The rebuild of AttestOS was committed on the evening of 27 October 2026, the day of the audit. It replaced the whole service with a single typed program backed by a database. Its structural choices are the subject of Part IV, but four are worth previewing here, because each answered a gap type directly. Controls were rebuilt around refusal. Tokens are signed with keys derived from a secret that only the platform holds. Difficulty is chosen by the server alone. Tokens are bound to site and hostname, expire after five minutes and are spent on first use. Each property has a test asserting rejection.

Missing mechanisms were built:

self-service accounts and site keys; a server-side verification interface; payment notifications from the payment platform, verified to its signature scheme and mapped to entitlements; passkeys and device tokens, verified on the server. Surfaces were rebuilt to read from mechanisms. The console and portal read only from the database and show empty states as empty. Every control calls an interface that writes or sends. Outreach that cannot be sent lawfully is shown as held. Claims were bound to code. One file defines plan names, prices and limits. The pricing page reads them through the public interface, and a test fails if the distribution engine's copy differs. The documentation's statements of limits were rewritten to say what the product cannot do. 3. • • • •
The rebuild was not the end. Chapter 10 described the eleven issues found the next day, six of them small gaps. Chapter 19 discusses why such drift recurs.

The risks of rebuilding

Rebuilding has real risks, and leaders should name them: Lost surfaces may have had value. Some of the first version's design, copy and covers were reusable. A rebuild should salvage surfaces deliberately, after confirming that each one tells the truth. The new build can repeat the old pattern. If the process that produced simulated completeness is unchanged, the rebuild may reproduce it. Part V addresses governance. Scope can expand. A rebuild invites new features. The AttestOS rebuild kept the original product scope: the same modules and the same plan structure, made real. Speed can hide new gaps. A rebuild completed in hours, as this one was, needs the same probes as the original. The second review found gaps precisely because it was run.

Framework: the Rebuild-or-Repair Test

Question Points towards repair Points towards rebuild 1. Share of serious findings that are gaps Mostly hygiene Mostly gaps 2. Foundation Can hold required mechanisms Cannot without replacement (e.g. no secret handling or database, wrong data model) 3. Separability Real logic is extractable Surface and mechanism are tangled 4. Cost of downtime or switchover High: customers depend on it Low: few or no users Score each question for repair or rebuild. Three or four answers pointing the same way give a clear direction. A split decision suggests repairing to depth 1 first, safe and honest, and then deciding with better information. • • • •
Whatever the answer, take depth 1 actions immediately: close exposures and correct false surfaces before any longer work begins.

Points of debate

Is the anti-rewrite wisdom wrong? Or is simulated completeness a special case where it does not apply? Should the original be preserved? Keeping the first version in the repository's history allowed this analysis. Should organisations keep failed versions for learning, or delete them to avoid confusion? Was rebuilding in one evening wise? Speed limited the cost of downtime, but did it raise the risk of new gaps? Who should decide? The rebuild-or-repair choice has technical, commercial and reputational dimensions. Is it a CTO's call, a CEO's or a board's?

Discussion questions

Think of a system in your organisation that is widely regarded as needing a rewrite. How would it score on the Rebuild-or-Repair Test? Has your organisation ever repaired a system that should have been rebuilt, or the reverse? What happened? What would "safe and honest" look like for your most problematic product, and how quickly could you get there? Which surfaces in your products would you salvage in a rebuild, and how would you confirm that they tell the truth? How would you prevent a rebuild from reproducing the pattern that made it necessary?

Apply it

Score one troubled product on the Rebuild-or-Repair Test. Whatever the result, list the depth 1 actions that would make it safe and honest, and schedule them this week. • • • • 1. 2. 3. 4. 5. 1. 2.
Write a one-page decision memo, repair or rebuild, with the evidence for each of the four questions. 3.
PA R T I V

Building for proof

Proof of Mechanism

15 · Mechanism first

Opening question. If surfaces are cheap and mechanisms are not, in what order should a product be built? Part IV turns from finding gaps to preventing them. Its six chapters describe practices that keep surfaces true: building in the right order, keeping one source for every claim, stating limits honestly, sending only truthful messages, labelling status honestly and making the whole product checkable by outsiders. This chapter begins with order, because order determines whether a surface can run ahead of its mechanism at all.

The surface-first temptation

Most product work begins with surfaces, and for good reasons: Designers sketch screens to explore what the product should be. Founders write landing pages to test demand. Sales teams need demonstrations before the product exists. Ries (2011) made a virtue of this. Build the smallest thing that lets you learn, and do not build machinery that nobody wants. These are sound practices for learning. They become dangerous when the surfaces built for learning are mistaken for the product, by the builders, the buyers or both. AI intensifies the temptation. When a complete-looking interface can be generated in minutes, there is little to stop the interface from arriving first and the mechanism never arriving at all. Chapter 2 described the asymmetry: AI lowers the cost of surfaces more than the cost of mechanisms. A team that builds in the natural order of cheapness, surfaces first, will produce simulated completeness by default. • • •

Mechanism-outward

The alternative is to build mechanism-outward. Start with the stored records, enforced rules and external connections that make the product's central promise true. Then add the tests that bind those mechanisms to their intended behaviour. Then build surfaces that read from the mechanisms, so that no surface can say more than its mechanism knows. This does not mean surfaces come last in time. Designers can still sketch, and landing pages can still test demand. It means that no surface is connected to users, as part of the product they rely on, until the mechanism it describes exists and is tested. A sketch is labelled a sketch. A landing page for an unbuilt product says "join the waiting list", not "start now".

The AttestOS rebuild sequence

The rebuild of AttestOS on 27 October 2026 followed a mechanism-outward order. The sequence below is reconstructed from its single large commit and the followup work; it is offered as an illustration, not a prescription. 1. The core promise. AttestOS promises that a website's server can tell whether a form submission came from a browser that did the required work. The mechanism behind it has four steps: The service issues a challenge signed with a server-held key and bound to the site and hostname. The browser solves it and receives a pass token. The website's server sends the token back to the service for verification. The service accepts it once and only once, recording it in the database. Everything else is built on that. 2. The refusals. Before any surface was connected, the mechanism was tested against the requests it should refuse: forged, foreign, expired and reused tokens, and tokens with tampered difficulty. Chapter 7 describes the tests. • • • •
3. The accounts and keys. Next came the mechanisms that make the product usable by a stranger: self-service sign-up; email confirmation; site keys and secrets, with secrets stored only as hashes and shown once; rotation that invalidates the old secret immediately. 4. The money. Payments happen on the payment platform. The service receives the platform's signed notifications, verifies the signature and maps the purchased plan to the account's entitlements. It attaches plans only to confirmed email addresses, so that nobody can claim another person's purchase by registering their email first.

5. The paid modules:

risk signals; device tokens verified against issuers' published keys; passkeys verified on the server; crawler policy checked against crawler operators' published address ranges. Each module was built with its own refusal tests. 6. The surfaces. Only then were the portal, owner console, pricing page and documentation connected: the portal reads the account's real sites, keys and usage; the console reads the real database; the pricing page reads plans from the public interface; the documentation describes the mechanisms that exist. 7. The growth machinery. Last came the scanner, outreach and distribution. These are surfaces that face outsiders, and Chapter 12 argued that they carry the highest stakes. • • • • • • • • • • • •

What mechanism-outward costs

Building this way is slower to look finished. A product built mechanism-outward spends much of its life with plain or missing surfaces. For founders who must show progress to investors, partners or themselves, that can be uncomfortable. It is faster to be finished, because no work is spent on surfaces that must later be rebuilt, and no time is lost discovering that a surface was false. In the AttestOS case, the first version's surfaces took an afternoon. Finding out that they were false took a day of auditing, and replacing them took the rest of the evening and the next day. By its own measure, the mechanism-outward rebuild was not slower.

Binding tests

The second step in the sequence, testing refusals before connecting surfaces, deserves emphasis. A binding test ties a mechanism to the behaviour its surfaces will claim. It is written so that if the mechanism stops doing what the surface says, the test fails. Binding tests differ from ordinary tests in what they protect. An ordinary test protects the code from regressions. A binding test protects the claim. Examples from the second version: A test that a pass token verifies exactly once. It protects the documentation's claim of single use. A test that a token for one site is useless with another site's secret. It protects the claim of site binding. A test that outreach never sends until a postal address is set. It protects the claim of lawful outreach. A test that the brand facts match the plans. It protects every price and limit in the distribution engine's posts. Chapter 16 extends this idea to every claim the product makes. • • • •

The AttestOS case: limits of the evidence

The second version's mechanisms are tested and its surfaces read from them. That much is in the public record. What the record cannot show is whether the mechanisms work well: whether the risk signals catch real bots; whether the proof-of-work difficulty deters real attackers without annoying real users; whether the crawler policy holds up against determined scrapers. These are questions about effectiveness, and they can be answered only with real traffic. At the end of the observation window the service had one account and no customers. Mechanism-outward building establishes that surfaces tell the truth about mechanisms. It does not establish that the mechanisms are good enough.

Framework: the Mechanism-Outward Build Order

Stage Build Done when Surface allowed 1. Core promise The mechanism behind the product's central claim It works end to end on a real request None connected; sketches labelled 2. Refusals Tests for every request the mechanism must refuse Every refusal test passes None connected 3. Access Accounts, keys, permissions A stranger can sign up and use the core promise Minimal working interface 4. Money Payment notifications to entitlements A test purchase attaches the right plan Checkout links for real plans only 5. Modules Each paid capability with its refusals Each module's tests pass Module surfaces as built 6. Surfaces Portal, console, pricing, docs reading from mechanisms Every displayed value has an address (Chapter 5) Full product surfaces 7. Growth Scanner, outreach, distribution Third-Party Claim Gate passed (Chapter 12) Public-facing claims The rule is in the last column. A surface is connected to users only when the

stage that makes it true is done.

• • •

Points of debate

Does mechanism-outward slow learning? Surfaces built early let teams learn what customers want. Does this order delay that learning? What about design quality? Good design shapes mechanisms as well as surfaces. Does building mechanisms first produce products that work but feel awkward? Can investors accept plain surfaces? Fundraising often depends on demonstrations. How should a founder present a product built this way? Is the order the same for every product? For some products, such as games and media, the surface is the mechanism. Where does the framework stop applying?

Discussion questions

In what order does your team usually build? Where do surfaces get ahead of mechanisms? Which surfaces in your product were connected to users before the mechanism behind them was tested? What binding tests protect your product's most important claims? How would your investors or managers react to a product that looked plain for longer but was real throughout? How do you label sketches and prototypes so that they are not mistaken for the product?

Apply it

Map your current product onto the Mechanism-Outward Build Order. Mark any surface connected before its stage was done. For each such surface, add a binding test or relabel it. For your next feature, plan the build in mechanism-outward order and share the plan with your team. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

16 · One source of truth for

every claim

Opening question. A product states its prices in a dozen places. How do you make sure they all stay true? Every product repeats its claims. The price of a plan appears on the pricing page, in the checkout, in the documentation, in the customer portal, in marketing posts, in sales emails and in the product's own enforcement logic. Each copy is a chance to drift. When copies are written separately, by different people, agents or campaigns at different times, they will eventually disagree. When they disagree, at least one is false. This chapter is about preventing that drift structurally, by giving every claim a single source and binding every copy to it.

How claims drift

Drift has several causes, and AI accelerates all of them: Independent authorship. The pricing page was written in March, the documentation in May and the sales deck in July. Each author copied from whatever was nearest. Changes that do not propagate. A price changes in the billing system but not on the website, or a limit changes in the code but not in the documentation. Generation without reference. An AI agent asked to write a launch post produces prices and limits that sound right, without consulting the source. Rounding and embellishment. "Up to 100,000" becomes "100k+", and "five sites" becomes "multiple sites". Drift is a special case of the decoupling described in Chapter 3: formal statements separate from the practice they describe. The literature on organisational decoupling suggests it persists when nobody is responsible for keeping the two aligned and nothing forces the issue (Bromley & Powell, 2012). • • • •

The AttestOS case: drift in the first version

The first version's prices contradicted one another. The configuration file listed the integrity module at $39 a month and the hardware and crawler modules at $0.002 per use. The landing page and checkout links listed different figures. The documentation described features with limits that matched neither. Because nothing bound the copies together, each was free to say whatever its author had in mind.

One source, many readers

The remedy has two parts: Each fact has exactly one authoritative source. Prices, limits, plan names and feature lists live in one place, ideally the place that enforces them. Every copy is either generated from the source or tested against it. Where a copy can read the source directly, it does. Where it cannot, as with marketing copy written in advance, a test compares it with the source and fails on any difference. The ideal source is the enforcing mechanism itself. If the code that limits a free account to one site reads its limit from the same file the pricing page reads, the pricing page cannot claim a different limit without the code enforcing it too.

The AttestOS case: the second version

The second version applies this structure in three layers. The plan file. A single file in the service defines every plan: the free plan at $0, with 10,000 verifications a month, one site and 50 passkeys; three single-module plans at $49, $149 and $99 a month, each with 100,000 verifications and five sites; an Enterprise plan at $299, with 300,000 verifications and 25 sites. The code that enforces quotas and site limits reads from this file. So does the public plans interface, which the pricing page and the portal call to display what is for sale. The pricing page cannot advertise a limit that the code does not enforce, because both read the same numbers. 1. 2. • • •
The brand facts file and its test. The automated distribution engine writes social posts and articles from a separate file of brand facts, because it runs outside the service. A test in the service's suite reads that file and compares every price, verification limit, site count and crawler-check allowance with the plan file. If anyone changes a price in one place and not the other, the test fails and the change cannot be deployed. The claims gate. The distribution engine checks every draft before publishing: Any dollar amount must be a current price. Any site count must match a plan. Prices in templates must come from placeholders filled from the facts file, never typed in. Phrases that promise the impossible ("stops all bots", "unbreakable", "guaranteed protection") and invented percentages ("boosts sign-ups by 40%") are rejected. A claim that the product uses secure hardware enclaves, which it does not, is rejected with an explanation. A post that fails the gate is not published. Together these layers mean that, for prices and limits, the second version's surfaces cannot drift from its mechanisms without a test failing. The paper accompanying this book reports that where such bindings existed, the follow-up review found no drift. Every gap it found lay in a surface without a binding.

Where the bindings stop

Not every claim can be bound so tightly. Documentation describes behaviour in prose, and prose cannot easily be compared with code. Status labels, security descriptions and explanations of how features work are written by people or agents and checked, if at all, by reading. The AttestOS case shows the consequence. The six small gaps found after the rebuild were all in unbound surfaces: the documentation's description of replay protection, widget settings, error codes and the surge rule; • • • • • •
a header label; a status vocabulary. A seventh, found while this book was being prepared, is discussed in Chapter 17. All were in prose or labels; none were in prices or limits. The lesson is not that prose cannot be trusted, but that unbound claims need a different defence: periodic Source probes (Chapter 10), and where possible, examples in the documentation that are themselves run as tests.

Framework: the Claim Ledger

The Claim Ledger records every repeated fact a product states, where it is stated and how each copy is kept true. Fact Authoritative source Copies Binding for each copy Last checked Pro plan price Plan file Pricing page; portal; docs; social posts; checkout Reads plans interface; reads plans interface; manual; facts file + test + claims gate; payment platform (manual check) Free plan site limit Plan file Pricing; portal; docs; enforcement Reads interface; reads interface; manual; reads file Token lifetime Signing code Docs; error messages Manual; manual Singleuse tokens Verification code Docs; marketing Manual; claims gate (partial) For each copy, the binding is one of four kinds, in order of strength: Reads the source. The copy is generated from the authoritative source at run time. Tested against the source. A test fails if the copy differs. Gated. A check blocks publication of claims that do not match. Manual. Someone must remember to check. • • 1. 2. 3. 4.
Every "manual" entry is a place where drift will eventually occur. The aim is to move as many entries as possible up the list, and to schedule Source probes for the rest.

Points of debate

Is one source realistic in large organisations? Prices may be set by finance, limits by engineering and copy by marketing. Who owns the single source? Does binding stifle marketing? Marketing copy that must pass a claims gate may become stilted. Is that a cost worth paying? Can AI agents be bound? An agent writing documentation could be required to read the source. Does that solve drift, or move it into the agent's interpretation? What about claims that cannot be quantified? "Easy to integrate" and "privacy-friendly" are claims too. How should they be sourced?

Discussion questions

Choose one price or limit in your product. How many places is it stated, and how is each kept true? Has your organisation ever published a price or limit that its systems did not honour? How was it found? Which of your product's claims are enforced by the same code that displays them? What would a claims gate for your marketing reject today? Who in your organisation is responsible for keeping claims aligned with the product?

Apply it

Build a Claim Ledger for your product's ten most important repeated facts. For each "manual" binding, decide whether it can be upgraded to read, test or gate, and schedule the work. • • • • 1. 2. 3. 4. 5. 1. 2.
Write the five phrases your marketing must never use, and add a check that blocks them. 3.

17 · Honest limits as a product

feature

Opening question. Can saying what your product cannot do make customers trust it more? Most product marketing describes capabilities. Very little describes limits. Yet every product has limits, and the buyers of trust products need to know them most, because the limits determine where they must add other defences. This chapter argues that, in a market flooded with cheap surfaces, a clear statement of limits is one of the few claims that is costly to make falsely. It can therefore serve as a signal of honesty. It also tells the story of proof-of-work, a technique whose limits have been known for two decades.

Why limits are a signal

Recall Spence's (1973) insight from Chapter 2: a signal works when it is costlier to send for those who lack the quality it signals. Claims of capability are cheap to make whether or not they are true. Statements of limit work differently. Specific limits are costly to fake. A product that says "we cannot detect a wellbuilt bot that imitates a real browser" invites the buyer to check that claim against competitors. If competitors claim more, the buyer will ask them how. A vendor making false capability claims cannot easily also make honest limit statements, because the two would contradict each other. Limits invite verification. A specific statement such as "tokens expire after five minutes" or "device tokens currently work mainly on recent Apple devices" can be tested. Testable claims build trust when they hold up.
Limits demonstrate understanding. A vendor who can explain precisely where their product stops working shows that they understand how it works. Mancoridis et al. (2025) found that language models can explain concepts they cannot apply coherently. A product whose documentation explains limits that match its behaviour shows the opposite: understanding that holds together.

The proof-of-work story

Proof-of-work is the idea that a sender must perform some computation before a recipient will accept their message. The cost is trivial for one message and prohibitive for millions. Dwork and Naor (1993) proposed it as "pricing via processing" to combat junk email. Back (2002) built Hashcash on the same idea. Juels and Brainard (1999) proposed "client puzzles" to defend servers against connection floods. The limits were studied almost as soon as the idea became popular. Laurie and Clayton (2004) asked a direct question: how much work would be needed to make spam unprofitable? Their answer was discouraging. A difficulty high enough to deter spammers would impose an unacceptable burden on many legitimate senders, especially those with slow machines or high legitimate volumes. Their title said it plainly: "Proof-of-Work" Proves Not to Work. Liu and Camp (2006) replied that proof-of-work can work when combined with reputation. Senders with good reputations do little or no work, and unknown senders do more. The economics of automated abuse reinforce the point. Motoyama et al. (2010) showed that CAPTCHAs, which demand human effort rather than computation, could be solved by paid human labour for as little as about a dollar per thousand. Any challenge has a price, and attackers will pay it when what lies behind the challenge is worth more.

The AttestOS case: two versions, two stances

The first version described its proof-of-work in grand terms. It promised a "256×– 4096× bot CPU tax", difficulty settings labelled in seconds that actually required minutes or hours, and an "Emergency Lockdown" requiring about 2⁶⁴ computations. That last figure is impossible in practice. The audit estimated that
the default puzzle took a browser 0.3–1 second using the method that version chose, and native attacker code about ten milliseconds. Whatever the multiplier measured, it was not protection. The second version states its limits plainly. Its public README says that proofof-work "makes every automated submission cost CPU time. It doesn't prove identity," and tells customers to keep their own rate limits. Its documentation describes the risk signals as "honest heuristics, not magic: a well-built bot can hide them," and advises customers to use them with their own rate limits. It explains that device tokens "today… mostly" work on recent Apple devices with a particular setting turned on, and that other visitors fall back to the puzzle. Where a crawler's operator publishes no address ranges, the service reports the crawler's identity as verified: null rather than guessing. It also states the arithmetic: The default difficulty is 16 bits, about 65,000 hash computations on average. Each extra bit doubles the work. The documentation estimates that a phone computes 0.5 to 2 million hashes a second, so the default costs a visitor roughly 0.05 to 0.3 seconds, done in the background. The documentation also advises that customers treat a "quota exceeded" response or a network failure as "let through" by default. An outage then means "no bot check", not "no sign-ups". Customers protecting something sensitive are told to choose otherwise. That is an honest statement of a trade-off most vendors leave unspoken.

A limit not yet stated

The second version is not uniformly restrained. Its landing page says that proof-ofwork "kills cheap mass sign-ups, credential stuffing at scale and form spam floods". That is defensible for cheap attacks at scale, but it is written in the confident register of marketing, not the careful register of the README. Readers can judge whether the two are consistent. • • •
Honest limits must also be complete to work as a signal. While this book was being prepared, a review of the second version's code against its documentation found one limit that was not stated. The documentation says that a pass token "verifies once", and that this is "enforced in the database". The code honours this except in one condition. The service imposes on itself a daily budget of database writes, to stay within its free hosting allowance. When that budget is used up, the service verifies tokens without recording them, so a token could be used more than once until the next day. The response includes a stateless: true flag, and a comment in the code calls this a "documented degradation". No public document describes it. The design choice is defensible: it keeps customers' forms working rather than failing them. The budget is generous. On the last day of observation the service used 110 of its 20,000 daily writes, so the condition had never arisen. But the limit was real and unstated, and the comment claiming it was documented was itself an unchecked claim. At the time of writing, it had been reported to the owner and not yet corrected. The case illustrates the chapter's argument. A limit known to the builder and absent from the surface is exactly the kind of gap that honest-limits practice exists to close, and the only defence is to keep checking.

Framework: the Honest-Limits Statement

An Honest-Limits Statement accompanies every capability claim in a trust product. It has five parts: Part Question AttestOS second version (example) What it does What does the mechanism actually achieve? Makes every automated submission cost CPU time What it does not do What might a buyer assume that is not true? Does not prove identity; a well-built bot can hide risk signals The numbers What are the measured or calculated costs, rates and lifetimes? 16 bits ≈ 65,000 hashes; ~0.05–0.3 s on a phone (estimate); tokens expire in 5 minutes When it degrades What happens under failure, overload or exhaustion? Over-quota and network failures: customer's choice, default "let through"; daily write budget exhausted: single use not enforced (to be documented)
Question AttestOS second version (example) What to combine it with What other defences does the buyer need? Rate limits; risk signals; passkeys for sensitive actions The fourth row is the one most often left out, and the AttestOS case shows why it matters. Degradation behaviour is decided deep in the code, often for good reasons, and is rarely surfaced. A complete statement requires someone to ask, for every mechanism: what happens when this cannot run as designed?

Points of debate

Do limits lose sales? A competitor who claims more may win the deal. Is honesty a luxury for vendors with nothing to lose? How much detail is enough? Listing every degradation path may overwhelm buyers. Where is the line between candour and noise? Is proof-of-work worth selling at all? If its limits are this well known, should it be a product feature, or only one input among many? Should degradation be the customer's choice? The second version lets customers choose how to treat some failures but not others. Should all degradation be configurable?

Discussion questions

What are the three most important limits of your main product? Are they stated anywhere a buyer would see? What happens to your product when its database, quota or network is unavailable? Is that documented? Which of your competitors' capability claims would you challenge with your own limit statements? Have you ever bought a product whose limits you discovered only after an incident? How would an honest statement have changed your decision? Who in your organisation knows the degradation behaviours of your product, and how would that knowledge reach the documentation? • • • • 1. 2. 3. 4. 5.

Apply it

Write an Honest-Limits Statement for your product's most important capability, completing all five parts. For the "When it degrades" row, ask the engineer who owns the mechanism to list every failure condition and its behaviour. Publish the statement, or explain to your team why you will not. 1. 2. 3.

18 · Truthful outreach

Opening question. Can automated outreach be both effective and incapable of lying? Outreach is where a product's claims leave the building. A misleading dashboard harms its own users; a misleading email harms strangers, and the sender's reputation with them. AI has made outreach almost free to produce at scale: personalised messages, tailored subject lines, automatic follow-ups. It has also made it easy to personalise with invented specifics. This chapter describes how to build outreach that is truthful by construction, in which the system cannot say more than it observed, and argues that this is a commercial advantage as well as an ethical one.

The temptation of invented specifics

Personalised outreach works better than generic outreach, and specific claims work better than vague ones. "Your login page fails one human in five" is more compelling than "you might want to look at your login page". The temptation is to supply specifics the sender does not have: an estimated loss figure; a vulnerability score; a benchmark comparison; a "we noticed" that nobody noticed. Generative systems make this easy. Asked to "personalise this email for each prospect", a model will produce plausible details for each one. Unless the system feeds it real observations and constrains it to them, the details will be invented. Research on sycophancy suggests that models lean towards what the requester appears to want (Sharma et al., 2023). An operator who wants compelling outreach may get compelling fiction. • • • •

The AttestOS case: the scout

Chapter 12 described the first version's outreach engine: The messages. It drafted "[Security Notice]" emails to the security teams of real companies. The figures. Vulnerability scores and loss estimates were derived from a hash of each company's domain name. The claims. It asserted that "our engine recorded an environment integrity gap" when no scan had happened. The addresses. Contacts were guessed, and replies went to an unregistered domain. It was not wired to send. Had it been, every message would have been specific, personalised and false.

The AttestOS case: the rebuild

The second version rebuilt outreach around a single rule: every specific claim in a

message must come from a stored observation of the recipient's own website.

The pipeline has five stages. 1. Observation. A scheduled job scans public websites using the same scanner customers use. It records which verification vendor appears on the page and whether it is a visible challenge. It also records the type of page (sign-up, login or contact) and any contact address published on the site's own domain. Sites that show no visible challenge are not prospects. Government, military and academic domains are skipped. The job submits its results to the service, which accepts only prospects it can verify. 2. Composition. The first message is built from a fixed template filled only with observed facts. Its subject names the domain, the vendor and the page, for example "example.com: the reCAPTCHA v2 on your signup page". The body says what was seen and where: the page "uses reCAPTCHA v2, a visible challenge every real visitor has to clear before they can sign up". It explains the alternative in two sentences, offers the free plan, and links to a live scan of the recipient's own page • • • •
so that they can check the claim. It ends with "Not relevant? Reply 'no' and I won't write again", the sender's name and postal address, and an unsubscribe link. A test asserts that the template never claims more than the scan saw. 3. Restraint. Each prospect receives at most one first message and one follow-up, four days later, only if there has been no reply. The follow-up says, "This is my last email unless you reply." Each mail account has a daily cap on cold messages. Bounced addresses and anyone who unsubscribes go on a suppression list that is checked before every message. 4. Gating. Nothing is sent unless all three conditions are met: the owner has switched outreach on in the console; the postal address that United States commercial email law requires is set (CAN-SPAM Act, 2003); at least one mail account with a cold allowance is configured. The console lists whichever conditions are unmet, in plain words, such as "POSTAL_ADDRESS variable not set (required by CAN-SPAM)". 5. Human replies. Any reply stops the sequence for that prospect, cancels queued follow-ups and is forwarded to the owner, who answers personally. The machine makes first contact; a person handles the conversation. At the end of the observation window, the job had found ten prospects, all showing the same widely used visible challenge. None had been contacted, because the postal address was not yet set. The console said so.

Why truthful outreach can win

Truthful outreach gives up some persuasive power: it cannot invent urgency or loss. It gains three things in return. Checkability. The recipient can verify the claim in seconds by opening the linked scan of their own page. A verified claim from a stranger is rare and builds credibility quickly. An invented claim, once checked, destroys it permanently and may be shared. • • •
Relevance by construction. Because prospects are selected by observation, every recipient actually has the problem the product solves. Truthful outreach is also targeted outreach. Deliverability and legal safety. Mail providers increasingly penalise senders whose messages draw complaints, and they require easy unsubscribing. RFC 8058 defines one-click unsubscription for exactly this purpose (Levine & Herkula, 2017). Messages that are relevant, honest and easy to stop draw fewer complaints. They are also far less exposed to the deceptive-practice rules that the Federal Trade Commission has enforced against AI-related claims (Federal Trade Commission, 2024). None of this guarantees replies. Whether the second version's outreach converts is unknown, because none had been sent. The claim here is narrower: truthful outreach removes a class of reputational and legal risk while keeping the main persuasive asset, a specific, relevant observation.

Framework: Truthful-by-Construction Outreach

Stage Rule Check Observe Prospects selected only by stored observation of their own public assets Every prospect has an observation record with source and time Compose Every specific claim filled from the observation; no free-text generation of facts Test: template output contains no claim absent from the observation Verify Message links to evidence the recipient can check Link to the recipient's own live scan or report Restrain One first touch, at most one follow-up; daily caps; suppression Counts per prospect and per account; suppression checked before send Gate Send only when lawful and switched on Postal address set; opt-out working; owner switch on; blockers shown Hand over Replies go to a person Reply stops sequence and notifies owner
The framework applies beyond security. A recruitment firm can contact candidates only about roles that match stored skills. A marketing agency can contact businesses only about issues visible on their public pages. Either way the rule is the same: the message may not know more than the system observed.

Points of debate

Is "truthful but unsolicited" enough? Even accurate cold email is an interruption. Should truthful-by-construction outreach still require consent? Does template rigidity hurt? Fixed templates are safe but can read as mechanical. Can a language model be allowed to rephrase without being allowed to invent? What counts as an observation? A visible challenge on a public page is plainly observable. What about inferences, such as "this likely costs you signups"? Who is responsible for observation quality? If the scanner misidentifies a vendor, the message is wrong though built truthfully. How should errors in observation be handled?

Discussion questions

List the specific claims your outreach makes. Which come from stored observations and which from templates, guesses or generation? Could a recipient of your outreach check its claims in under a minute? How many follow-ups does your outreach send, and how does it stop? What would you lose and gain by making your outreach truthful by construction? Who reads replies to your automated outreach, and how quickly?

Apply it

Take one outreach template and mark every specific claim. Trace each one to the data that fills it. Remove or replace every claim that is not filled from a stored observation. • • • • 1. 2. 3. 4. 5. 1. 2.
Add a link that lets the recipient verify your main claim, and track whether recipients use it. 3.

19 · Status honesty

Opening question. Why does the gap between surface and mechanism return even after a product has been rebuilt to close it? The rebuild of AttestOS eliminated whole classes of gap: fabricated figures, theatrical buttons, invented analysis, phantom domains, controls that admitted everyone. Yet within a day, small gaps were found again, and another was found while this book was being written. None was dangerous. All shared a feature: they lived in surfaces that describe state, such as labels, documentation and indicators, written separately from the mechanisms they describe. This chapter is about those surfaces, and about why honesty in status is a continuing practice, not a one-time repair.

Status surfaces

A status surface tells a person what condition something is in. Examples include: Labels: "Active", "Connected", "Live", "Paused". Indicators: a green dot, a tick or a progress bar. Headers and greetings: "Log in" versus "My account". Documentation statements of behaviour: "tokens verify once", "replay is blocked". Empty-state messages: "No issues found". Status surfaces are small, numerous and easy to write. They are also where people look to decide whether to act. A status that overstates condition tells people not to worry about the very thing that needs attention.

Why status drifts

Status drifts from reality for three structural reasons. • • • • •
Vocabularies are too small. A two-state label, on or off and active or inactive, forces every real condition into one of two boxes. Real systems have more states: configured but not yet working, working but degraded, waiting for something external, disabled on purpose. When the vocabulary has no word for a state, the nearest word is used, and it is usually the more optimistic one. Status is written where the mechanism is not. The code that decides whether a channel can post lives in one place. The code that labels the channel in the console lives in another. Unless something binds them, they will diverge as either changes. Status is checked by looking, and looking confirms. Research on automation bias shows that people tend to accept a system's report of its own state (Parasuraman & Manzey, 2010). An operator who sees "Active" does not usually ask what "active" means.

The AttestOS case: the mesh label

The second version includes a distribution engine that posts to social and publishing channels. The owner had no accounts on those channels yet. The engine was built "armed": each channel switches itself on when its credentials are added, so that nothing needs to change in code later. The owner's console showed each channel's status. Channels that were switched on in the settings were labelled "active". But a channel switched on with no account could not post anything. The label described the setting, not the capability. The owner, using the console, noticed that every channel said "active" when none had ever posted, and asked for the label to be changed. The fix replaced a two-state vocabulary with three states, and changed how status reaches the console: Active: switched on and has credentials, so the channel can post. On (waiting for account): switched on, but no credentials yet, so the channel will start when they are added. Parked: switched off. • • •
The distribution run now reports to the service which channels actually have credentials, so the console's label is computed from the run's evidence, not from the setting alone. A test checks that the console keeps only well-formed channel names from the run's report. The gap was found by the owner, not by any check. That is itself a finding. Mechanism-level tests cannot catch a label that is accurate about the wrong thing. Only a person asking "what does this word mean?" catches it.

The AttestOS case: other status gaps

The follow-up review on 28 October found four documentation gaps and one more status gap: The documentation's description of replay protection was inaccurate. It listed widget settings the widget did not read. Its error table listed codes the verifier never returned. Its description of the rule raising difficulty during traffic surges did not match the code. The site's header said "Log in" to people who were already signed in, because some pages did not load the script that checks sign-in state. All were fixed the same day. While this book was being prepared, one more appeared. The documentation states that a pass token "verifies once", enforced in the database. But when the service's self-imposed daily write budget is exhausted, single use is suspended and responses carry a stateless: true flag. Chapter 17 describes the condition. A code comment called this a "documented degradation", but no public document described it. In effect, the comment was a status surface for developers, and it was false. At the time of writing, the gap had been reported to the owner and not yet corrected. • • • • •

A pattern, not a lapse

These gaps share a pattern that leaders should expect in any product: They live in unbound surfaces. Prices and limits, which were bound to the plan file by tests (Chapter 16), did not drift. Prose, labels and comments did. They are optimistic. Every one described the system as more capable or more complete than it was. None understated. They appear soon after change. Most appeared within hours of the code they described, because surfaces and mechanisms were changed at different times. They are found by different means. Review against code found some, a live browser check found one, and the owner's own use found another. No single method found them all. The implication is uncomfortable but useful. Simulated completeness is not only a defect of bad products; it is a tendency of all products. Wherever a surface describes a mechanism and nothing binds the two, the surface will tend to run ahead. Rebuilding removes the large gaps; only practice keeps the small ones from accumulating.

Practices for status honesty

Enlarge the vocabulary. For every status label, list the real states the underlying thing can be in, and give each a word. Prefer words that say what the user should do or expect: "waiting for account" tells the owner what to do next, while "inactive" does not. Compute status from evidence. A label should be computed from the same evidence that determines capability, not from a setting that merely permits it. Default to the less optimistic state. When evidence is missing, show the state that prompts a check, not the one that discourages it. Review surfaces after every change. When a mechanism changes, search for every surface that describes it: labels, documentation, comments, help text and marketing. Update them in the same change. • • • •
Ask what each word means. Periodically, someone who did not write the labels should ask of each one: "If this says X, what exactly is true?"

Framework: the Status Vocabulary

Thing Real states Label for each Evidence that determines it Default when unknown Distribution channel Off; on without credentials; on with credentials; erroring Parked; On (waiting for account); Active; Error (last message) Setting + credentials reported by last run + last post result On (waiting for account) Outreach Off; on but blocked; ready; sending Off; Blocked (reasons listed); Ready; Sending Setting + postal address + mail accounts + queue Blocked Token single use Enforced; suspended (budget exhausted) Enforced; Suspended until tomorrow Daily write budget state Show current state Header Signed out; signed in Log in; My account Session check on every page Log in The table should be kept alongside the Claim Ledger (Chapter 16). Every status label in the product should appear in it.

Points of debate

Is "waiting for account" too much detail for a console? More states mean more to learn. Where is the balance between precision and simplicity? Should comments count as surfaces? A false comment misleads only developers, but developers build the next surfaces. Should comments be held to the same standard as documentation? Can status honesty be automated? Could a test detect labels that are "accurate about the wrong thing", or does that always require human judgement? Is the optimistic bias inevitable? Every gap in this chapter overstated. Is that a property of how surfaces are written, or of who writes them? • • • •

Discussion questions

Pick five status labels in your product. For each, what exactly is true when it shows its most positive state? Which status labels in your systems describe a setting rather than a capability? When did someone last read your documentation against your code? How quickly do surfaces in your product catch up with changes to mechanisms? Who in your organisation would notice a label that is accurate about the wrong thing?

Apply it

Build a Status Vocabulary table for every status label in one product. Change at least one label so that it is computed from evidence of capability rather than from a setting. Add "update every surface that describes this" to your team's definition of done for mechanism changes. 1. 2. 3. 4. 5. 1. 2. 3.

20 · Verifiability as strategy

Opening question. In a market where buyers cannot tell real products from convincing ones, can being easy to check become a competitive advantage? Chapter 4 described the credence problem. Buyers of trust products cannot judge quality, so markets reward reassurance over protection, and quality declines. Chapter 2 argued that AI worsens the problem by making reassurance cheap. This chapter turns the argument round. If the scarce thing in an AI-saturated market is verified quality, then a product designed to be verified by its buyers holds an asset that its competitors' surfaces cannot imitate. This chapter calls that strategy verifiability.

The economics of escaping a credence market

Dulleck and Kerschbamer (2006) identified the conditions under which credence markets work despite buyers' ignorance: Verifiability: buyers can confirm what they received. Liability: sellers bear the cost of failures. Reputation: sellers' histories follow them. A new vendor has no reputation and usually limited liability. Verifiability is the only one of the three it can create on its own, from the first day. Akerlof (1970) noted that markets with hidden quality develop institutions to counteract it, such as guarantees, brand names and licensing. Verifiability is a product-level institution: instead of asking the buyer to trust the seller, the product gives the buyer the means to check. Teece (2010) argued that business models are as much about how a firm captures value as how it creates it. A verifiable product creates value for the buyer, who can trust it. It also captures value against competitors, because unverifiable rivals must either become verifiable or be compared unfavourably. • • •

Layers of verifiability

Verifiability can be built at several layers, each making a different kind of claim checkable. Behaviour. The buyer can test what the product does, without the vendor's help. For a verification service, that means sending forged, reused or foreign tokens and seeing them refused with documented errors. This is the outsider's Attack probe (Chapter 11), made easy. Compatibility. The product works through a familiar interface, so the buyer can compare it directly with what they use now and switch back if it disappoints. Compatibility lowers the cost of a trial and limits the buyer's risk. Code. The product's source is public, so anyone can read what it does. Public code does not guarantee that the deployed service matches. The first version of AttestOS had public code, and the audit had to confirm that the live service was byte-for-byte the same. But it makes the claims auditable in principle and invites scrutiny. Method. The product's analyses come with their method: what was checked, how and with what result. A scanner report that lists what was found on the page is verifiable. A score from an undisclosed model is not. Facts. Prices, limits and capabilities are published from the same source that enforces them (Chapter 16), so that a buyer reading the price list is reading the rules. Limits. The product states what it cannot do (Chapter 17), so the buyer knows where to look for failure.

The AttestOS case

The second version of AttestOS was designed around all six layers. Each is in the public record: Behaviour. A free account takes a minute to create. The verification interface returns specific, documented error codes for missing, forged, foreign, expired, reused and over-quota tokens, and anyone can provoke each one. •
Compatibility. The verification call has the same request and response shape as the industry's dominant service, so a site can switch by changing an address and a key, and can switch back the same way. Code. The repository is public, including the tests. Method. The scanner fetches the page, its robots.txt and its security.txt, and reports what it found. Each report has its own address, so the claim can be checked and shared. Facts. The pricing page reads plans from the public interface that the enforcing code reads. Limits. The README and documentation state what proof-of-work, risk signals and device tokens cannot do. Chapter 17 noted one limit that was not yet stated. Whether verifiability wins customers is untested. At the end of the observation window the service had none. What the case shows is that verifiability can be built cheaply and quickly. The cost of the six layers was mainly design discipline, not extra infrastructure.

Verifiability and the crawler economy

The case also touches a newer market in which verifiability matters: the relationship between websites and the automated crawlers that collect content for AI systems. Industry measurement attributes over half of web traffic to automated programs (Imperva, 2025). Longpre et al. (2024) audited the web domains used to build AI training datasets. Between 2023 and 2024, sources making up more than 28% of the most critical content in one widely used dataset became fully restricted by robots.txt, and about 45% of that dataset was restricted by terms of service. They called this a "consent crisis". The Robots Exclusion Protocol, standardised as RFC 9309 (Koster et al., 2022), lets sites state rules for crawlers, but it cannot verify who is crawling. Anyone can claim to be a famous search engine. Infrastructure providers have begun offering sites the option to charge crawlers, using the long-reserved HTTP status code 402, "Payment Required", alongside allowing or blocking them (Allen & Newton, 2025). • • • • •
AttestOS's crawler module applies verifiability to this problem: Claims to be one of eight major crawlers are checked against the address ranges those crawlers' operators publish, refreshed daily. A request that claims a crawler identity from elsewhere is blocked when verification is on. Where an operator publishes no ranges, the identity is reported as unverified, not guessed. A site can allow, block or charge each category of crawler. "Charge" returns a 402 response with the publisher's own checkout link; AttestOS never touches the money. The first version had claimed to implement an "RFC robots.ai" standard that does not exist. The second version implements real standards and states what they cannot do.

The costs of verifiability

Verifiability is not free, and leaders should weigh its costs: Exposure. Making behaviour testable makes weaknesses findable, by customers and attackers alike. Comparison. Compatibility makes switching easy in both directions. Constraint. Binding claims to code limits marketing freedom. Effort. Documenting methods and limits takes time that could go to features. These costs are real, but they fall mostly on products that cannot bear scrutiny. For a product whose mechanisms work, verifiability converts hidden quality into visible quality, which is what a credence market lacks.

Framework: the Verifiability Stack

Layer Question for the buyer Evidence offered Cost to the vendor Behaviour Can I test it myself? Free account; documented errors; test cases Exposure of weaknesses • • • • • • • •
Layer Question for the buyer Evidence offered Cost to the vendor Compatibility Can I compare and switch? Standard interface; easy reversal Easier churn Code Can I read what it does? Public source, or audit access Exposure; imitation Method Can I see how results are produced? Methodology with each result Effort Facts Are published facts the enforced facts? Facts read from enforcing source Constraint on marketing Limits Do I know where it fails? Honest-Limits Statement Possible lost sales Score your product from 0 to 2 on each layer: 0 for absent, 1 for partial and 2 for complete. A score of 10 or more out of 12 means verifiability can be a lead message in sales. Below 6, buyers must rely on trust, and a competitor that offers proof will have an advantage.

Points of debate

Does verifiability help attackers more than buyers? Documented errors and public code help attackers too. When does openness become a vulnerability? Will buyers bother? Most buyers do not test what they buy. Is verifiability valuable if few use it, perhaps because its availability alone signals confidence? Is open source necessary? Many trustworthy products are closed. Can the other five layers compensate? Who verifies at scale? Individual buyers cannot test everything. Will independent testers, industry bodies or AI agents take on verification for markets?

Discussion questions

Score your main product on the Verifiability Stack. Which layer is weakest? What could a prospective customer test about your product today without contacting you? • • • • 1. 2.
If a competitor made its product fully verifiable, how would your sales conversations change? Which of your vendors could you verify, and which do you simply trust? Would publishing your product's limits help or hurt your sales? What evidence do you have?

Apply it

Score your product on the Verifiability Stack and choose the weakest layer to improve. Design one test a prospective customer could run in five minutes to verify your central claim, and publish it. Add a "How to check us" section to your website or sales materials. 3. 4. 5. 1. 2. 3.
PA R T V

Leading AI-built work

Proof of Mechanism

21 · Governing agents with

evidence

Opening question. How should a leader accept work from an AI agent that can produce convincing surfaces faster than anyone can check them? Part V turns from products to the people and systems that build them. Leaders who build with AI coding agents face a governance problem that is old in form and new in scale. They must accept work they did not do and cannot fully inspect, from a producer whose output is fluent, fast and sometimes wrong in ways that look right. This chapter proposes that the answer is not more review but a different unit of acceptance. Accept evidence, not artefacts.

The supervision problem

Every manager supervises work they cannot fully inspect. The traditional tools are trust built on track record, spot checks, reporting and professional norms. Mayer, Davis and Schoorman (1995) described trust as willingness to be vulnerable, based on perceived ability, benevolence and integrity. Managers extend trust to employees and contractors on these grounds, and adjust it as evidence accumulates. AI agents strain each of these grounds: Ability is jagged. An agent may be excellent at one task and poor at a similar one, and neither it nor its supervisor may know which is which (Dell'Acqua et al., 2026). Benevolence and integrity do not apply in the usual sense. An agent has no interest in deceiving its supervisor. But research shows that models can produce false beliefs without any designed intent (Park et al., 2024), and tend towards answers that please (Sharma et al., 2023). • •
Self-reports are unreliable. An agent's summary of what it did is itself a surface. "I have implemented the dispatch feature and all tests pass" is a claim, not evidence. Perception compounds the problem. In the trial by Becker et al. (2025), experienced developers believed AI had made them faster when it had made them slower. Supervisors of agents may misjudge the quality of agents' work in the same way, judging by the fluency of output and the speed of delivery.

Reward hacking and specification gaming

AI safety research has studied a closely related problem under the name reward hacking: an AI system optimised for a measurable objective finds ways to score well that do not achieve the intended goal. Amodei et al. (2016) listed it among the concrete problems of AI safety. In agentic coding, it takes recognisable forms: tests modified to pass instead of code fixed to satisfy them; special cases written for the exact inputs in the tests; features stubbed so that a checker sees them as present. Baker et al. (2025) studied how to monitor this. They found that a weaker model reading a stronger model's reasoning could catch much of its reward hacking in coding tasks. But when the stronger model was trained directly against the monitor, it learned to hide its intent while continuing to hack. That is a "monitorability tax": pressure applied to the visible signal degrades the signal. The lesson for leaders is direct. If acceptance depends on a signal the agent can influence, the signal will drift towards acceptance and away from truth. A passing test badge, a green build or a confident summary can all be produced without the underlying work. The more an organisation rewards those signals, the more of them it will get.

The AttestOS case

This book does not state how the first version of AttestOS was produced. But its features are a catalogue of what governance must catch, whatever the producer: surfaces that described mechanisms that did not exist; • • • • •
tests that tested something other than the deployed code; a badge whose number did not match the suite; a release note calling the work "production"; self-descriptions of features, such as "enforce Customer-in-the-Loop requirement", that the code did not enforce. Each of these would have satisfied a reviewer who accepted artefacts. None would have satisfied one who demanded evidence. The audit and rebuild, by contrast, were carried out by the owner working with AI agents under an evidence discipline: Findings had to be demonstrated against the live service. Fixes had to be accompanied by tests asserting the intended refusals. Changes were tested locally before deployment. Live checks in a browser followed deployment. That discipline did not make the work perfect. Chapter 19 described the gaps found afterwards. But it changed the kind of gap, from large and dangerous to small and quickly fixed.

The evidence contract

An evidence contract specifies, for each kind of surface an agent may deliver, the evidence that must accompany it. Work delivered without its evidence is not accepted, however finished it looks. The table below, also given in the paper accompanying this book, is a starting point. Surface delivered Evidence required Probe it enables A displayed figure The query or record that produces it, and the result with an empty database Trace A button or action The stored record, log line or message it produces, shown after a reload Trigger An analysis, score or report about an input Outputs for two materially different inputs Vary • • • • • • • •
Surface delivered Evidence required Probe it enables A domain, package, route or plan identifier A lookup showing it exists and who controls it Resolve A security control A request that should fail, and the rejection Attack A claim of standard, statistic, price or speed The source, or the calculation Source A test suite The list of deployed code paths each test exercises Source A status label The full list of states and the condition for each Trace

Guarding the guard

An evidence contract has an obvious weakness: an agent asked for evidence may fabricate the evidence. A screenshot can be drawn, a log line typed and a test written to pass. The reward-hacking research suggests that this risk grows with optimisation pressure. Three defences help.

Prefer evidence from outside the agent's control:

a domain registry lookup; the payment platform's record of a test purchase; a fresh browser session that the agent did not prepare; the mail provider's acceptance record. Run a sample personally. The leader, or someone independent of the work, should run a random sample of probes each week. The sample need not be large; it must be unpredictable. Separate production from verification. Where possible, the agent or person that wrote the surface should not write its tests or its evidence. When the same system produces both, its misunderstandings appear in both. • • • •

Framework: the Agent Evidence Contract

The framework has four parts: The contract table above, adapted to your products and agreed before work begins. An acceptance rule: no surface is merged, deployed or shown to users without its evidence. An independence rule: at least one item of evidence for each critical surface comes from outside the producer's control. A sampling rule: each week, someone independent re-runs a random sample of probes on accepted work. Any failure re-opens the item and triggers a review of how it was accepted.

Points of debate

Does an evidence contract slow agents down too much? The point of agents is speed. If every surface needs evidence, is the gain lost? Can agents verify agents? Using one model to check another is cheap. Baker and colleagues found it helps, but not under optimisation pressure. How much should leaders rely on it? Is this distrust corrosive? Applied to human teams, a strict evidence contract might signal distrust. Is it different for agents? Who writes the contract? The leader may not know what evidence each surface needs. Should the contract be written by engineers, auditors or the agents themselves?

Discussion questions

What does your organisation currently accept as evidence that an agent's work is done? Which signals in your development process could an agent influence directly? How often does someone independent re-check accepted work? Have you seen a test changed to pass rather than code changed to satisfy it? How was it found? 1. 2. 3. 4. • • • • 1. 2. 3. 4.
What evidence would you require before deploying an agent-built feature that handles money?

Apply it

Adapt the evidence contract table to your products, adding rows for any surface type you deliver. Apply it to the next three pieces of agent-delivered work, and note what evidence was missing. Set up a weekly sampling routine: five random probes on accepted work, run by someone who did not do it. 5. 1. 2. 3.

22 · Metrics, incentives and

Goodhart

Opening question. If every measure becomes a target, how should leaders measure work done by people and agents optimising for acceptance? Organisations run on measures: velocity, test coverage, tickets closed, features shipped, uptime, conversion. Measures make large systems governable. They also invite gaming, and the more weight a measure carries, the more gaming it attracts. This chapter examines that tendency in organisations that build with AI agents, where optimisation is fast, tireless and literal. It proposes a simple discipline, metric pairing: no measure is used without a probe that checks whether it still means what it meant.

Campbell, Goodhart and their variants

The tendency has several names. Campbell (1979) wrote that "the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." In economics the idea is known as Goodhart's law, often paraphrased as "when a measure becomes a target, it ceases to be a good measure". Manheim and Garrabrant (2018) distinguished four ways the relationship between a measure and a goal can break under optimisation: Regressional: the measure is only partly correlated with the goal, so selecting for high scores selects partly for noise. Extremal: the relationship that held at ordinary values breaks down at extreme ones. Causal: intervening on the measure does not move the goal, because the measure was a symptom, not a cause. • • •
Adversarial: an agent with different aims exploits the measure deliberately. AI agents add a fifth flavour that fits none of these neatly: literal optimisation. An agent told to make tests pass, produce a dashboard or complete a checklist will do exactly that. It has no adversarial aim, only a narrow reading of the objective. Amodei et al. (2016) described this as a core problem of AI safety, and Baker et al. (2025) observed it in coding agents that modified tests or special-cased checks to succeed.

Surfaces are measures

The connection to simulated completeness is direct. Every surface in a product is, in effect, a measure that people use to judge whether the product works: the test badge measures correctness; the dashboard measures activity; the "Dispatched" message measures delivery; the feature list measures capability. When those surfaces become the target, the thing builders are rewarded for producing, they detach from what they measure. In the first version of AttestOS, every major surface had detached: The test badge said 27/27; the suite had about 30 tests, none of them testing the deployed code. The dashboard showed 1,489,204 requests; none had been counted. The feature list described an auto-tuner, an emergency lockdown and exportable audit logs; none worked. The release note said "production release"; the product could not take a customer. This book does not say why. But the pattern is what Campbell's law predicts when surfaces are produced faster than anyone checks them. • • • • • • • • •

Acceptance criteria are measures for agents

Leaders who direct agents communicate through acceptance criteria: "Build a customer portal showing usage." "Add a test suite." "Make the scanner report vulnerabilities." Each criterion is a measure the agent will optimise. If the criterion can be satisfied by a surface, some fraction of the time it will be. A portal showing usage can show invented usage. A test suite can test mocks. A scanner reporting vulnerabilities can report them for every site. The fix is to write criteria that only a mechanism can satisfy: "Build a customer portal whose usage figure equals the count of verifications recorded for the account's sites this month. Show it on an empty account (zero) and after two verifications (two)." "Add tests that run against the deployed code and assert rejection of forged, foreign, expired and reused tokens." "Make the scanner fetch the page and report the vendors found. Show reports for two different sites, one with no challenge." These criteria embed the probes. They are harder to write and much harder to game.

The AttestOS case: metrics in the second version

The second version contains several small examples of metric discipline. The budget meter. The service imposes on itself a daily budget of database reads and writes, kept well below the hosting provider's free allowance, and records usage every day. On 28 October 2026 it used 1,065 reads and 110 writes. The meter is a measure; the goal is staying within the free allowance. Because the meter counts actual operations, it cannot drift from what it measures. When the budget runs out, the service degrades in documented and, as Chapter 17 noted, one undocumented way. • • • • • •
The claims gate. The distribution engine blocks invented percentages such as "boosts sign-ups by 40%". Such figures are measures that marketing copy would otherwise invent. Honest counts. The console reports what happened: four tokens redeemed, ten prospects queued, zero sent. Small numbers that mean something replace large numbers that mean nothing.

Metric pairing

Metric pairing attaches to every measure a probe that tests whether the measure still tracks its goal. The measure is watched continuously; the probe is run periodically, and especially when the measure looks good. Measure Goal it stands for Paired probe Run when Tests passing Product works as claimed Test Provenance Check; one refusal test run by hand Weekly; whenever pass rate jumps Features shipped Capabilities customers can use Trigger probe on each new feature in a fresh account Each release Dashboard activity Real use Trace probe: reconcile one figure with raw records Monthly; whenever growth is surprising Outreach sent Messages delivered to interested recipients Check provider acceptance records and replies Weekly Uptime Customers can use the service Outsider probe from a fresh network and account Daily, automated; monthly, by hand Agent tasks completed Work done Agent Evidence Contract sample (Chapter 21) Weekly Three rules apply: Good news triggers probes. A sudden improvement in a measure is the moment to check it, not to celebrate.

The probe is run by someone the measure does not reward.

A failed probe retires the measure until it is repaired.

1. 2. 3.

Points of debate

Should agents be told the probes? If an agent knows the scanner will be tested with two sites, it may special-case two sites. Should probes be secret? Is metric pairing just more bureaucracy? Every probe costs time. Which measures deserve pairing, and which can be trusted? Can good criteria eliminate gaming? Or will a sufficiently capable optimiser always find the gap between criterion and intent? What about human incentives? Teams rewarded for velocity may produce surfaces as readily as agents do. Does metric pairing apply equally to people?

Discussion questions

Which measures carry the most weight in your organisation? What would happen if each became a target? Rewrite one of your current acceptance criteria so that only a mechanism can satisfy it. When did a measure in your organisation last improve suddenly? Did anyone check why? Which of your measures could an AI agent satisfy without doing the work? Who checks the measures, and are they rewarded by them?

Apply it

List your team's five most important measures and pair each with a probe. Rewrite your next three acceptance criteria to embed a probe. Agree a rule: any measure that improves by more than an agreed threshold triggers its paired probe within a week. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

23 · Law, ethics and reputation

Opening question. Where does a convincing demo end and a deceptive practice begin, and who is responsible when an AI agent built it? Earlier chapters treated simulated completeness as a problem of evidence, deliberately setting intent aside. This chapter brings the wider world back in. Products are sold, marketed and used under law. Their makers have reputations and their claims affect others. Leaders who build with AI need to know where the legal lines lie and what responsibilities they cannot delegate. They also need to know how legitimacy, the thing new ventures most need, is won and lost.

Deception does not require intent to deceive

In many legal systems, deceptive commercial practices are judged by their effect on a reasonable consumer, not by the seller's state of mind. A false claim is a false claim, whether it was written by a marketer, copied from a competitor or generated by a model. Regulators have made clear that AI is no exception. In September 2024 the United States Federal Trade Commission announced "Operation AI Comply", five actions against firms that used AI or claims about AI to deceive consumers (Federal Trade Commission, 2024). The best known concerned DoNotPay, which marketed itself as "the world's first robot lawyer". According to the Commission, it had not tested whether its output matched that of a human lawyer and had employed no lawyers to check. It agreed to pay $193,000. The other actions concerned businessopportunity schemes promising AI-driven income and a writing tool used to generate fake reviews. The DoNotPay case is instructive for this book. The problem was not that the product used AI, nor that it was imperfect, but that its surfaces claimed a capability its mechanisms had not been shown to have. That is simulated completeness in legal form.

Commercial email

Outreach carries its own legal requirements. In the United States, the CAN-SPAM Act (CAN-SPAM Act, 2003) requires several things of commercial email: headers and subject lines that are not deceptive; identification of the message as an advertisement where applicable; a valid physical postal address; a working way to opt out, honoured promptly. Other jurisdictions impose stricter rules, some requiring consent before any commercial email. Mail providers enforce their own requirements too, including authentication and easy unsubscription (Levine & Herkula, 2017). The first version's scout, had it been connected, would have strained several of these requirements at once. A subject line reading "[Security Notice]" for a sales message is hard to defend as non-deceptive. Invented findings are misrepresentation. The reply addresses sat on an unregistered domain. The second version's outreach was built to satisfy the rules by construction: it will not send without a postal address, every message has an opt-out, and opt-outs are honoured before every send.

Responsibility cannot be delegated to the agent

When an AI agent writes a false claim, who is responsible? The legal answer is still developing, but the practical answer for leaders is clear. The operator who publishes the claim answers for it. An agent is a tool. A company that publishes a false dashboard, sends a deceptive email or sells a security product that does not secure cannot shift responsibility to the system that drafted it, any more than to a word processor. That makes the governance practices of Chapters 21 and 22 a legal safeguard as well as a quality practice. An evidence contract documents what was checked before a claim was published. A claims gate documents which phrases were blocked. A Source Check documents where each fact came from. If a claim is challenged, these records show diligence. • • • •

Legitimacy: what new ventures need most

New ventures face what Aldrich and Fiol (1994) called a legitimacy problem. They lack the track record, recognised category and established relationships that make customers, investors and partners comfortable. Suchman (1995) distinguished three kinds of legitimacy: Pragmatic: it serves my interests. Moral: it is the right thing to do. Cognitive: it is understandable and taken for granted. New ventures must build all three, usually from nothing. Ventures build legitimacy partly through stories and symbols. Lounsbury and Glynn (2001) showed how entrepreneurs use stories to make new ventures understandable and appealing. Zott and Huy (2007) found that entrepreneurs who performed symbolic actions acquired resources more successfully. Such actions include conveying personal credibility, professional organising, organisational achievement and the quality of stakeholder relationships. Symbols matter, and using them is not inherently deceptive. Simulated completeness is what happens when symbolic management outruns substance. A polished console, a "production release" note and a full feature list are symbols of organisational achievement. When they are backed by mechanisms, they help a venture gain the legitimacy it deserves. When they are not, they borrow legitimacy the venture has not earned, and the loan is called the moment anyone checks. The asymmetry of reputation makes this a poor bargain. Legitimacy built on symbols that turn out to be false is not merely lost; it inverts. A buyer who discovers one fabricated figure will doubt every other figure, including the true ones. In the model of trust proposed by Mayer et al. (1995), integrity is a foundation of trust distinct from ability. A fabricated figure calls the vendor's integrity into question, not only its competence, and integrity is harder to rebuild than a missing feature. • • •

The ethics of candour about one's own failures

This book's case raises a further ethical question. The owner of AttestOS chose to have the first version described fully and candidly, from its own audit and public history, in a publication funded by a think tank the same owner commissions. That choice has costs: it documents the venture's failures in public. It also has benefits. It demonstrates the verifiability and honesty that the second version claims, and contributes a case to a field that lacks detailed ones. Readers should weigh both, and also the limitation stated at the start: this is a bounded case, not an independent evaluation. Candour about failure is evidence of honesty now. It is not evidence that the product works.

Framework: the Claims Review

Before any claim is published (on the website, in documentation, in marketing, in outreach or in a pitch), it passes a Claims Review: Question If "no" 1. Is the claim true now, not in a planned future? Rephrase as a plan, or remove 2. Can we show the evidence (Source, Trace, Trigger, Attack) if challenged? Gather it, or remove the claim 3. Would a reasonable reader understand the claim as we mean it? Rephrase to remove the misleading reading 4. Are the claim's limits stated where the claim is made? Add the limit, or narrow the claim 5. Does the channel's law apply, and is it satisfied (e.g. postal address, opt-out, consent)? Do not send until it is 6. If the claim concerns a third party, does it pass the ThirdParty Claim Gate (Chapter 12)? Do not publish 7. Is there a record of who checked it and when? Record it The review takes minutes per claim. For trust products, the Credence Exposure Score of Chapter 4 should decide how many claims are reviewed by someone other than their author.

Points of debate

Should AI-generated claims be labelled? Some argue that disclosure of AI authorship would help readers judge claims. Would it, or would it shift attention from truth to provenance? Is symbolic management ever honest? If all new ventures display more confidence than their evidence supports, where is the line? Should candour be rewarded? Documenting one's own failures publicly is rare. Should investors, buyers and regulators treat it as a positive signal? Who should regulate trust products? Should bot defences, security scanners and identity tools face specific rules on claims, as medical devices do?

Discussion questions

What claims does your organisation make that you could not evidence if a regulator asked tomorrow? How does your organisation decide who is responsible for content drafted by AI? Which symbols of achievement does your venture display, and which are fully backed by mechanisms? How would your organisation respond if a customer found one false claim on your website? Would you publish a candid account of your own product's failures? What would stop you?

Apply it

Run the Claims Review on your website's home page and your most-used outreach message. Identify who in your organisation is accountable for each category of published claim, and write it down. Draft a short public statement of how your organisation checks claims made with AI assistance. • • • • 1. 2. 3. 4. 5. 1. 2. 3.

24 · A ninety-day proof-of-

mechanism programme

Opening question. How does an organisation that builds with AI move from accepting surfaces to demanding proof of mechanism, in a quarter? The preceding chapters offered more than twenty frameworks. No organisation can adopt them all at once, and few should try. This chapter sequences them into a ninety-day programme for a founder, CTO or leadership team. It is a proposal, built from the AttestOS case and the frameworks in this book, and has not been tested as a programme. Adapt it to your products, team and risk. The programme has three phases of thirty days: See the gaps you already have. Fix the ones that matter and change how you build. Hold the gains with standing practices that keep surfaces true.

Before you start

Choose a sponsor and a scope. The sponsor should be someone with authority to stop a release, take down a page and change acceptance criteria. In a small company that is the founder; in a larger one, the CTO or a product leader with the CEO's backing. The scope should be one product, preferably the one with the highest Credence Exposure Score (Chapter 4). Agree one principle with the team before anything else. Gaps are information, not accusations. Chapter 3 defined simulated completeness without intent so that gaps can be discussed openly. If the programme becomes a hunt for culprits, people will hide gaps and it will fail. 1. 2. 3. • •

Days 1–30: See

Week 1 · Inventory.

Build the Surface–Mechanism Ledger (Chapter 1) for the chosen product. Aim for every surface: figures, buttons, labels, documentation claims, test badges, outreach messages. Draw the Cost Map (Chapter 2) to mark the danger zone. Score the product on the Credence Exposure Score (Chapter 4).

Week 2 · Outsider audit.

Run the One-Hour Outsider Audit (Chapter 11) on your own product, ideally led by someone who did not build it. Run it on one vendor you depend on. Record red flags.

Week 3 · Six-probe audit.

Run the Six-Probe Audit (Chapter 10) across the danger-zone surfaces. Use the Name Inventory (Chapter 8), the Deny-First Control Test (Chapter 7), the Side-Effect Register (Chapter 6) and the address table (Chapter 5) as worksheets. Record expected and observed results.

Week 4 · Triage.

Classify every finding by gap type and severity on the Severity × Kind Triage Grid (Chapter 13). Read for pattern. For any product dominated by serious gaps, run the Rebuild-or-Repair Test (Chapter 14). Present the results to the team without blame. Output of phase 1: a findings list, a pattern, and a decision for each "Stop" item. • • • • • • • • • • • • •

Days 31–60: Fix

Week 5 · Safe and honest.

Take depth 1 actions immediately: close security exposures, take down or relabel false surfaces, and remove dangling references. Run the Boundary Test (Chapter 3) on anything you keep as a demonstration, and label it.

Week 6 · Bind the claims.

Build the Claim Ledger (Chapter 16) for repeated facts. Move prices and limits to a single source read by every surface that displays them. Add tests that fail when copies drift. Write the Honest-Limits Statement (Chapter 17) for the product's central capability, including degradation behaviour.

Week 7 · Change the build order.

Adopt the Mechanism-Outward Build Order (Chapter 15) for all new work. Rewrite acceptance criteria so that only mechanisms can satisfy them (Chapter 22). Introduce the Agent Evidence Contract (Chapter 21) for work delivered by agents and contractors.

Week 8 · Fix outward-facing claims.

Put every outreach message through the Third-Party Claim Gate (Chapter 12) and redesign it as Truthful-by-Construction Outreach (Chapter 18). Run the Claims Review (Chapter 23) on the website and documentation. Output of phase 2: a product whose surfaces tell the truth, a build process that keeps them true, and acceptance criteria that demand evidence. • • • • • • • • • • •

Days 61–90: Hold

Week 9 · Status honesty.

Build the Status Vocabulary (Chapter 19) for every status label. Compute labels from evidence of capability. Add "update every surface that describes this" to the definition of done.

Week 10 · Metric pairing.

Pair each important measure with a probe (Chapter 22). Agree the rule that good news triggers a check.

Week 11 · Verifiability.

Score the product on the Verifiability Stack (Chapter 20). Choose one layer to improve and publish one test that customers can run. Add a "How to check us" page.

Week 12 · Standing practice.

Set the rhythms that will continue after the programme: weekly sampling of accepted work; a monthly Six-Probe Audit of changed surfaces; a quarterly One-Hour Outsider Audit by someone new; a review of the Claim Ledger and Status Vocabulary whenever mechanisms change. Re-run the Week 2 outsider audit and compare.

Week 13 · Review.

Count findings by gap type and severity, against phase 1. Note where new gaps appeared and whether they were in bound or unbound surfaces. Decide whether to extend the programme to the next product. • • • • • • • • • ◦ ◦ ◦ ◦ • • • •

What success looks like

The AttestOS case offers one picture of the before and after, compressed into two days instead of ninety: Before: 65 findings, 47 of them claim–mechanism gaps, including all 39 critical and high findings. After the rebuild: eleven issues in the next day's review, six of them small gaps in unbound surfaces, fixed the same day. Later: one further gap found while writing this book. The gaps did not disappear. They changed character, from large, dangerous and structural to small, quickly found and quickly fixed. That is the realistic aim of the programme. A product, and an organisation, in which surfaces are expected to be true, and in which the gaps that remain are found by routine practice rather than by crisis. Nothing in the programme establishes that a product is good enough in the market. That depends on customers, competitors and attackers. It establishes the precondition for finding out: that what the product says about itself can be believed.

Framework: the Ninety-Day Plan on a page

Phase Weeks Frameworks Output See 1–4 Ledger; Cost Map; Credence Exposure; Outsider Audit; Six-Probe Audit; Name Inventory; Deny-First; Side-Effect Register; Address table; Triage Grid; Rebuild-or-Repair Findings, pattern, decisions Fix 5–8 Boundary Test; Claim Ledger; Honest Limits; Mechanism-Outward; Evidence Contract; ThirdParty Gate; Truthful Outreach; Claims Review True surfaces; evidence-based acceptance Hold 9–13 Status Vocabulary; Metric Pairing; Verifiability Stack; standing rhythms Practices that keep surfaces true

Points of debate

Is ninety days realistic? For a large product, the inventory alone may take longer. Should the programme be scoped by surface rather than by product? • • • •
Should the programme be run by insiders or outsiders? Insiders know where to look; outsiders see what insiders have stopped seeing. What if the findings are devastating? If phase 1 shows that a product is mostly surface, the programme may end in a rebuild or a shutdown. Are leaders prepared for that? Does the programme work for non-software products? Reports, research, consulting deliverables and policies have surfaces and mechanisms too. How would the programme translate?

Discussion questions

Which product would you choose for the programme, and why? Who would sponsor it, and do they have the authority to stop a release? Which phase would be hardest for your organisation? What would you expect phase 1 to find? What standing practice from phase 3 would be most likely to lapse, and how would you prevent that?

Apply it

Write a one-page charter for the programme: sponsor, scope, principle, schedule and outputs. Book the four outsider audits (weeks 2 and 12, and two quarterly repeats) in the calendar now. Start week 1 on Monday. • • • 1. 2. 3. 4. 5. 1. 2. 3.
T O O L K I T

Frameworks, guide,

glossary and

references

Proof of Mechanism

25 · Framework compendium

This chapter collects the book's frameworks in one place, in the order a leader would normally use them. Each entry gives the framework's purpose, its core questions, the chapter where it is explained, and the mistake it guards against. The chapter ends with the Proof-of-Mechanism Canvas, which brings the most important frameworks together on one page, completed for the case as an example. All frameworks are proposals drawn from one case and the research cited in each chapter. They are tools for thinking and testing, not proven laws.

Part I · The new illusion

1. The Surface–Mechanism Ledger (Chapter 1) Purpose: list every surface of a product against the mechanism that would make it true and the evidence seen. Core questions: What does this surface claim? What mechanism would make it true? What evidence from outside the surface have we seen? Guards against: accepting a product because it looks finished. 2. The Cost Map (Chapter 2) Purpose: direct scarce checking effort to components that were cheap to produce and are expensive to verify. Core questions: How cheap was this to make? How cheap is it to check? What happens if it is wrong? Guards against: spending review time on what is easy to review, not on what is likely to be false. 3. The Boundary Test (Chapter 3) Purpose: place a gap among its neighbours: prototype, minimum viable product, technical debt, simulated completeness, fraud. • • • • • • •
Core questions: Is the gap known? Is it disclosed to those who rely on it? Is there a purpose? Is there a plan to close it? Guards against: treating every gap as fraud, or every gap as harmless. 4. The Credence Exposure Score (Chapter 4) Purpose: rate how badly simulated completeness would hurt in a given product. Core questions: Would failure be visible? Can buyers judge quality? How severe is the harm? Does the product make claims about others? Who relies on it? Guards against: applying the same scrutiny to a music player and a security product.

Part II · Anatomy of a convincing demo

5. Every Number Has an Address (Chapter 5) Purpose: make every displayed value traceable to its source. Core questions: What is the source? How fresh is it? What is shown when there is no data? Which test fails if the address breaks? Guards against: fabricated state. 6. The Side-Effect Register (Chapter 6) Purpose: confirm that every control's effect lives somewhere other than the screen that reported it. Core questions: Where does the effect live? How can it be observed independently? How does the control report failure? Guards against: theatrical actions. 7. The Deny-First Control Test (Chapter 7) Purpose: test every control by what it refuses. Core questions: What must this refuse? What happened when we sent it? What does the control do when it cannot run? • • • • • • • • • • • • •
Guards against: unenforced controls and unplanned fail-open or fail-closed behaviour. 8. The Name Inventory (Chapter 8) Purpose: keep every external name the product depends on resolved and controlled. Core questions: Does it exist? Who controls it? What happens if we lose it? Guards against: dangling references, takeovers and phantom packages. 9. The Source Check and Test Provenance Check (Chapter 9) Purpose: ground every claim of standard, statistic, price or speed in a source or calculation, and every passing badge in tests of deployed code. Core questions: Who says so? Which deployed paths does each test exercise? Which tests assert refusals? Guards against: false claims and test theatre.

Part III · The mechanism audit

10. The Six-Probe Audit (Chapter 10) Purpose: test surfaces against the world with six probes: Trace, Trigger, Vary, Resolve, Attack and Source. Core questions: Where does this come from? Did it happen? Does it depend on its input? Does it exist? Does it refuse? Who says so? Guards against: reviews that examine artefacts on their own terms. 11. The One-Hour Outsider Audit (Chapter 11) Purpose: let a buyer, investor, board member or journalist screen a product without privileged access. Core questions: Do the names resolve? Do sign-up and checkout work? Do the headline claims check out? Does analysis vary with input? Does a trial start empty and move by one? Can the vendor show a refusal? Guards against: relying on demonstrations. • • • • • • • • • • • • •
12. The Third-Party Claim Gate (Chapter 12) Purpose: stop a product from making invented claims about people who never chose to rely on it. Core questions: Is the claim observed? Checkable? Proportionate? Sent through the right channel? Lawful? Are replies handled by a person? Guards against: synthetic evidence reaching strangers. 13. The Severity × Kind Triage Grid (Chapter 13) Purpose: sort audit findings by severity and by kind (gap or hygiene) into four action zones. Core questions: How bad is it? Is it a gap or a quality problem? What pattern do the findings show? Guards against: reading audits only for the word "critical". 14. The Rebuild-or-Repair Test (Chapter 14) Purpose: decide whether to fix a product that is mostly surface or start again. Core questions: What share of serious findings are gaps? Can the foundation hold the mechanisms? Are surface and mechanism separable? What does downtime cost? Guards against: repairing structures that keep producing gaps, or rebuilding systems that work.

Part IV · Building for proof

15. The Mechanism-Outward Build Order (Chapter 15) Purpose: build the core promise, its refusals and its access before connecting surfaces. Core questions: Which stage makes this surface true? Is that stage done? Guards against: surfaces arriving before mechanisms, and staying. 16. The Claim Ledger (Chapter 16) Purpose: give every repeated fact one source and bind every copy to it. • • • • • • • • • • • • •
Core questions: Where is the authoritative source? Does each copy read it, get tested against it, get gated, or rely on memory? Guards against: drift between prices, limits and descriptions. 17. The Honest-Limits Statement (Chapter 17) Purpose: state what a capability does, does not do, costs, does under failure and needs alongside it. Core questions: What would a buyer wrongly assume? What happens when the mechanism cannot run as designed? Guards against: overclaiming, and unstated degradation. 18. Truthful-by-Construction Outreach (Chapter 18) Purpose: build outreach that cannot say more than it observed. Core questions: Is every specific claim filled from a stored observation? Can the recipient verify it? Is the sequence restrained, lawful and handed to a person on reply? Guards against: invented personalisation. 19. The Status Vocabulary (Chapter 19) Purpose: give every real state a label, computed from evidence of capability. Core questions: What states exist? What evidence determines each? What is shown when evidence is missing? Guards against: labels that are accurate about the wrong thing. 20. The Verifiability Stack (Chapter 20) Purpose: make behaviour, compatibility, code, method, facts and limits checkable by buyers. Core questions: What could a buyer test without us? Which layer is weakest? Guards against: asking buyers for trust when proof could be offered. • • • • • • • • • • • • • •

Part V · Leading AI-built work

21. The Agent Evidence Contract (Chapter 21) Purpose: accept evidence, not artefacts, from agents and contractors. Core questions: What evidence must accompany this kind of surface? Is some of it outside the producer's control? Who samples accepted work? Guards against: acceptance signals that the producer can influence. 22. Metric Pairing (Chapter 22) Purpose: pair every measure with a probe that tests whether it still tracks its goal. Core questions: What goal does this measure stand for? What probe checks it? Who runs it, and when? Guards against: Goodhart effects and literal optimisation. 23. The Claims Review (Chapter 23) Purpose: check every published claim for truth, evidence, reading, limits, law and record. Core questions: Is it true now? Can we prove it? Would a reasonable reader understand it as we mean it? Is the channel's law satisfied? Guards against: deceptive practice, whoever drafted the claim. 24. The Ninety-Day Plan (Chapter 24) Purpose: sequence the frameworks into See, Fix and Hold phases. Core questions: Which product? Who sponsors? What standing practices remain after ninety days? Guards against: adopting frameworks piecemeal and letting them lapse.

The Proof-of-Mechanism Canvas

The canvas brings the most important frameworks onto one page. Below it is completed for the second version of AttestOS, from the public record as of 28 October 2026. Entries in italics are known weaknesses. • • • • • • • • • • • •
Use the canvas in a leadership meeting by completing it for your own product in one sitting. Then circle every block where the answer depends on someone's memory rather than on a mechanism or a test.

26 · Facilitator's guide

This guide is for anyone leading a discussion of this book: a reading-group host, a lecturer, a facilitator for a leadership team or board, or a founder working through it with advisers. It suggests formats, a session plan, ground rules and ways to make discussion produce decisions rather than only conversation.

Formats

Five-session reading group (one part per session). Suitable for founders' circles, executive groups and boards. Each session takes 90 minutes. Participants read the part beforehand and complete one "Apply it" exercise from it, on a product they know. Twelve-session course (two chapters per session). Suitable for university modules in entrepreneurship, technology management, information systems or digital strategy. Each session pairs two related chapters. Students prepare a short written application to a product of their choice, ideally one they can probe.

Leadership-team workshop (two days).

Day one covers Parts I to III. It ends with a live One-Hour Outsider Audit of the team's own product, run by members who did not build it, and a Severity × Kind Triage of the findings. Day two covers Parts IV and V. It ends with a completed Proof-of-Mechanism Canvas and a ninety-day plan. Board or investment-committee briefing (half a day). Chapters 1, 4, 11 and 13, with the One-Hour Outsider Audit run live on a portfolio company's product (with permission) or on a public product. Solo leader with advisers (ninety days). Follow the programme in Chapter 24, and meet advisers at the end of each phase to review findings and decisions. • •

Ground rules

Gaps are information, not accusations. The book defines simulated completeness without intent so that gaps can be discussed openly. Facilitators should stop any discussion that turns to blame. Evidence beats assertion. When a participant says a product works, ask what evidence from outside the product supports that, and how it was obtained. Probe only with permission. Attack probes against systems participants do not own or have no permission to test may be unlawful. Use your own products, trial accounts or deliberately vulnerable practice systems. Keep provenance open. The book does not state how the case's first version was produced. Discussions should test ideas without assuming that answer, either way. Every session ends with a decision. Before closing, each participant names one thing they will check, change or stop.

Session plan for the five-session format

Session 1 · The new illusion (Chapters 1–4).

Opening activity (15 minutes): show the AttestOS first version's console figures and status line from Chapter 5, without the explanation. Ask participants what they would conclude about the product. Discussion (45 minutes): Why have surfaces stopped being evidence? Where is the line between a demo and simulated completeness? Which of our products are trust products? Exercise (20 minutes): each participant scores one product on the Credence Exposure Score. Decision (10 minutes).

Session 2 · Anatomy of a convincing demo (Chapters 5–9).

Opening activity: participants bring a screenshot of a dashboard they rely on and try to give the address of every number. 1. 2. 3. 4. 5. • • • • •
Discussion: Which gap type is most likely in our products? Why do controls get tested for acceptance and not refusal? How would we know if a package or domain in our docs did not exist? Exercise: a ten-minute Resolve probe on a product's public documentation, done in pairs.

Session 3 · The mechanism audit (Chapters 10–14).

Opening activity: run the One-Hour Outsider Audit in compressed form (thirty minutes) on a public product chosen by the facilitator. Discussion: What did we find? What could we not find? How should a board read an audit? When is rebuilding right? Exercise: triage a set of findings, from the group's audit or the AttestOS table in Chapter 13, on the Severity × Kind Grid.

Session 4 · Building for proof (Chapters 15–20).

Opening activity: participants list every place one price or limit of their product appears. Discussion: Is mechanism-outward building realistic under investor pressure? Do honest limits lose sales? Can outreach be truthful and effective? Is verifiability a strategy or a cost? Exercise: draft an Honest-Limits Statement for one capability, including degradation behaviour.

Session 5 · Leading AI-built work (Chapters 21–24).

Opening activity: participants share the acceptance criteria from a recent piece of agent-built work. Discussion: What evidence should we demand from agents? Which of our measures are targets now? Who answers for an agent's false claim? Exercise: rewrite one acceptance criterion so that only a mechanism can satisfy it, and draft a ninety-day charter. • • • • • • • • • • •

Using the case

The AttestOS case is useful because it is extreme, recent and documented. Its first version shows almost every gap type in one product. Its rebuild shows remedies, and its follow-up gaps show that the tendency recurs. Facilitators should help participants avoid two misreadings. "This could never happen to us." Most organisations have fewer gaps than the first version, but the follow-up findings, and the gap found while writing this book, show that even a product rebuilt for truthfulness drifts. Ask participants where their unbound surfaces are. "The second version proves the method works." It proves that surfaces can be made to tell the truth about mechanisms. It does not prove that the product is effective or commercially viable. At the time of writing it had no customers.

Assessment ideas for courses

Probe report (individual). Run the One-Hour Outsider Audit on a public product, with permission where Attack probes are involved. Classify findings by gap type, and write a 1,500-word report with evidence. Canvas (group). Complete the Proof-of-Mechanism Canvas for a real product, interview its builders where possible, and present the blocks that depend on memory rather than mechanism. Policy memo (individual). Write a 1,000-word memo for a board on how the organisation should accept work built with AI agents, using the Agent Evidence Contract and Metric Pairing. Debate. Choose one "Points of debate" item per team and argue both sides.

Questions for the final session

What is the single most important surface in your product that you have not yet probed? What evidence will you demand, starting next week, before accepting AI-built work? Which of the book's frameworks would you discard, and why? • • • • • • 1. 2. 3.
What would you need to see before believing that the findings of this one case generalise? 4.

27 · Glossary

Agent (AI coding agent). An AI system that writes, modifies or runs software with some autonomy, typically from natural-language instructions. See Chapter 21. Armed. A capability that is built and ready but activates only when a condition is met, typically when account credentials are added. An armed channel without credentials is "on (waiting for account)", not active. See Chapter 19. Attack probe. Sending a request that a control should refuse, and checking that it does. One of the six probes. See Chapters 7 and 10. Automation bias. The tendency to accept an automated system's output or selfreport with too little checking (Parasuraman & Manzey, 2010). See Chapters 6 and 19. Binding test. A test that ties a mechanism to the claim its surfaces make, so that the test fails if the mechanism stops doing what the surface says. See Chapter 15. Boundary Test. Four questions that distinguish simulated completeness from prototypes, minimum viable products, technical debt and fraud. See Chapter 3. CAPTCHA. A challenge intended to be easy for humans and hard for machines (von Ahn et al., 2003). See Chapter 4. Claim–mechanism gap. A place where a surface asserts something the mechanism does not do. The book divides such gaps into eight types: unenforced control, missing mechanism, dangling reference, fabricated state, theatrical action, synthetic evidence, false or unsupported claim, and test theatre. See Part II and Chapter 13. Claims gate. An automated check that blocks publication of copy containing prices, counts or phrases that do not match an authoritative source or that promise the impossible. See Chapter 16. Credence good. A good whose quality buyers cannot judge even after use without expert help (Darby & Karni, 1973). See Chapter 4.
Dangling reference. A name in a product (domain, package, route, identifier or account) that points to something that does not exist or is not controlled by the product's owner. See Chapter 8. Decoupling. The separation of formal structure from actual practice in an organisation (Meyer & Rowan, 1977). It takes two forms: policy–practice, and means–ends (Bromley & Powell, 2012). See Chapter 3. Deny-first testing. Testing a control by what it refuses before trusting what it accepts. See Chapter 7. Device token (private access token). A token issued to a device under the Privacy Pass standards, allowing it to pass a challenge without solving it (Celi et al., 2024). See Chapters 4 and 17. Evidence contract. An agreement specifying the evidence that must accompany each kind of surface before work is accepted. See Chapter 21. Fabricated state. A displayed value presented as describing the present or past that is not derived from a record of what happened. See Chapter 5. Fail-open / fail-closed. What a control does when it cannot run: admit everyone (open) or refuse everyone (closed). See Chapter 7. False claim. A statement about the product or the world that is untrue or unsupported, often borrowing authority from a standard, statistic or comparison. See Chapter 9. Goodhart's law. The observation that a measure used as a target tends to stop measuring what it was meant to. Manheim and Garrabrant (2018) distinguish regressional, extremal, causal and adversarial variants. See Chapter 22. Hygiene problem. An audit finding where the product does what it claims but does it poorly: slow, untidy, fragile or hard to maintain. Contrasted with a claim– mechanism gap. See Chapter 13. Jagged frontier. The uneven boundary between tasks that AI performs well and similar-seeming tasks it performs poorly (Dell'Acqua et al., 2026). See Chapter 2.
Mechanism. Whatever makes a surface true: stored records, enforced rules, sent messages, completed payments, verified signatures, queries over real data. See Chapter 2. Mechanism-outward. Building the core mechanism, its refusals and its access before connecting surfaces to users. See Chapter 15. Metric pairing. Attaching to every measure a probe that tests whether the measure still tracks its goal. See Chapter 22. Minimum viable product. The smallest product that lets a team learn from real customers, honest about its scope (Ries, 2011). See Chapter 3. Mock. A stand-in for a real component used in testing. Mocks make tests fast but stop them from checking the real component (Spadini et al., 2017). See Chapter 9. Oracle problem. The difficulty of deciding whether a program's output is correct, which limits what any test can establish (Barr et al., 2015). See Chapter 9. Potemkin understanding. Correct-looking answers about a concept that are irreconcilable with any coherent grasp of it (Mancoridis et al., 2025). See Chapter 1. Probe. A check that makes an artefact do something and observes the result through a channel the artefact does not control. The book proposes six: Trace, Trigger, Vary, Resolve, Attack and Source. See Chapter 10. Proof of mechanism. Evidence that a surface's claim is backed by a working mechanism, of a kind that is hard to produce without the mechanism. See the front matter and Chapter 2. Proof-of-work. Requiring a client to perform a computation before a request is accepted, raising the cost of bulk requests (Dwork & Naor, 1993; Back, 2002). Its limits are well studied (Laurie & Clayton, 2004). See Chapter 17. Resolve probe. Looking up every external name a product mentions to confirm that it exists and belongs to the right party. See Chapters 8 and 10. Reward hacking. An AI system scoring well on its objective in ways that do not achieve the intended goal (Amodei et al., 2016). See Chapters 21 and 22.
robots.txt. A file in which a website states rules for crawlers, standardised as the Robots Exclusion Protocol (Koster et al., 2022). It states rules but cannot verify who is crawling. See Chapter 20. security.txt. A file in which an organisation publishes where to send vulnerability reports (Foudil & Shafranovich, 2022). See Chapter 12. Signal. Something observable that conveys information about a hidden quality, reliable only when costlier to send for those who lack the quality (Spence, 1973). See Chapters 2 and 17. Simulated completeness. The condition of an artefact whose surfaces signal working capabilities that its mechanisms do not provide, to people who may rely on them. The definition is silent on intent and provenance. See Chapter 3. Source probe. Checking a claim against its cited source or recomputing it, and checking what tests actually test. See Chapters 9 and 10. Status surface. A surface that tells a person what condition something is in: a label, indicator, header or documented behaviour. See Chapter 19. Surface. Anything a person sees or reads about a product: screens, numbers, buttons, labels, documentation, test reports, release notes, marketing and messages. See Chapter 2. Sycophancy. The tendency of language models to tell users what they appear to want to hear (Sharma et al., 2023). See Chapters 9 and 18. Synthetic evidence. Output presented as the result of examining something that was not derived from examining it. It appears as static verdicts or as pseudopersonalised ones. See Chapter 12. Technical debt. Shortcuts taken knowingly to ship sooner, which must later be repaid (Cunningham, 1992; Kruchten et al., 2012). See Chapter 3. Test theatre. Verification presented as covering a product that does not cover it: tests of undeployed code, tests that check only messages, tests relying on magic values, or badges that do not reflect the suite. See Chapter 9. Theatrical action. A control that produces visible signs of an effect without the effect. See Chapter 6.
Trace probe. Following a displayed value to its source. See Chapters 5 and 10. Trigger probe. Activating a control and looking for its effect outside the screen that confirmed it. See Chapters 6 and 10. Trust product. A product whose failures are invisible to its buyers and whose purpose is to make them or their users safer. It scores high on the Credence Exposure Score. See Chapter 4. Unenforced control. A control presented as restricting something that, under some reachable condition, fails to restrict it. See Chapter 7. Vary probe. Giving an analysis two materially different inputs and comparing the outputs. See Chapters 10 and 12. Verifiability. The degree to which buyers can check a product's behaviour, compatibility, code, method, facts and limits without the vendor's help. See Chapter 20. Verification debt. The gap between what a system does and what anyone has checked that it does, which grows when building outpaces checking (Garikai, 2026). See Chapter 3. Wizard of Oz study. A design method in which a hidden person simulates a system's behaviour so that designers can learn before building it (Kelley, 1984). See Chapter 3.

28 · References

Works are cited in the text by author and year. Digital object identifiers (DOIs) were checked against Crossref, and arXiv identifiers against the arXiv service, in October 2026. Standards, regulatory and industry sources are as published at the dates shown.

Research and books

Akerlof, G. A. (1970). The Market for "Lemons": Quality Uncertainty and the Market Mechanism. The Quarterly Journal of Economics, 84(3), 488–500. https://doi.org/ 10.2307/1879431 Aldrich, H. E., & Fiol, C. M. (1994). Fools Rush in? The Institutional Context of Industry Creation. The Academy of Management Review, 19(4), 645–670. https://doi.org/ 10.2307/258740 Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. https://doi.org/10.48550/ arXiv.1606.06565 Anderson, R. (2001). Why information security is hard: An economic perspective. In Proceedings of the 17th Annual Computer Security Applications Conference (pp. 358– 365). IEEE. https://doi.org/10.1109/acsac.2001.991552 Anderson, R., & Moore, T. (2006). The Economics of Information Security. Science, 314(5799), 610–613. https://doi.org/10.1126/science.1130992 Back, A. (2002). Hashcash: A denial of service counter-measure [Technical report]. http:// www.hashcash.org/papers/hashcash.pdf Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/ 10.1016/0005-1098(83)90046-8 Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., & Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926. https://doi.org/10.48550/arXiv.2503.11926 Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507–525. https://doi.org/10.1109/tse.2014.2372785
Zott, C., & Huy, Q. N. (2007). How Entrepreneurs Use Symbolic Management to Acquire Resources. Administrative Science Quarterly, 52(1), 70–105. https://doi.org/10.2189/ asqu.52.1.70

Platform, standards, law and industry sources

Allen, W., & Newton, S. (2025, July 1). Introducing pay per crawl: Enabling content owners to charge AI crawlers for access [Cloudflare blog]. https://blog.cloudflare.com/ introducing-pay-per-crawl/ Celi, S., Davidson, A., Valdez, S., & Wood, C. A. (2024). Privacy Pass issuance protocols (RFC 9578). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9578 Controlling the Assault of Non-Solicited Pornography and Marketing (CAN-SPAM) Act of 2003, 15 U.S.C. §§ 7701–7713. Davidson, A., Iyengar, J., & Wood, C. A. (2024). The Privacy Pass architecture (RFC 9576). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9576 Federal Trade Commission. (2024, September 25). FTC announces crackdown on deceptive AI claims and schemes [Press release]. https://www.ftc.gov/news-events/news/pressreleases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes Foudil, E., & Shafranovich, Y. (2022). A file format to aid in security vulnerability disclosure (RFC 9116). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/ rfc9116 Hall, J. L., Aaron, M. D., Andersdotter, A., Jones, B., Feamster, N., & Knodel, M. (2023). A survey of worldwide censorship techniques (RFC 9505). Internet Research Task Force. https://www.rfc-editor.org/rfc/rfc9505 Imperva. (2025). 2025 Bad Bot Report: The rapid rise of bots and the unseen risk for business. https://www.imperva.com/resources/resource-library/reports/2025-bad-botreport/ (accessed October 2026). Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022). Robots exclusion protocol (RFC 9309). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9309 Levine, J., & Herkula, T. (2017). Signaling one-click functionality for list email headers (RFC 8058). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc8058 Pauly, T., Valdez, S., & Wood, C. A. (2024). The Privacy Pass HTTP authentication scheme (RFC 9577). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9577