Artificial Intelligence

What Makes an AI Output Worth Trusting

The Horizon accounting system reported shortfalls in British post office branches that were treated as debts even when branch operators disputed them. The operators lacked important means of checking the data. This article draws requirements for AI use from that case: people must be able to inspect the basis of an output and challenge decisions while they can still be changed.

A close view of an impasto oil painting. A dark machine displays a polished blank tablet. Behind it, two rows of small green marks wind across the surface; a small indigo block bends over the trail.
A close view of an impasto oil painting. A dark machine displays a polished blank tablet. Behind it, two rows of small green marks wind across the surface; a small indigo block bends over the trail.

The Horizon accounting system displayed a shortfall in British post office branches when declared holdings differed from the totals derived from recorded transactions. The branch operator could not mark the amount as disputed in the system: to open the next trading period, the operator had to cover it or settle it centrally as a sum owed to the Post Office. The High Court later found that challenged shortfalls were treated as debts.1

An accounting mismatch could thus acquire the status of a debt and eventually support an allegation of theft, fraud, or false accounting. In 2021, the Court of Appeal quashed 39 convictions in which assumed reliability of Horizon data had been essential; it dismissed three appeals with materially different evidence.2

For AI deployments, the case raises a concrete question: can affected people check the basis of an output and challenge it before someone acts on it? This article examines the means of review and appeal needed to justify trust in an AI output. The application to AI is the article’s argument; the courts examined Horizon.

What the case contributes to the argument

A correctly calculated difference establishes only that declared holdings do not match the balance derived from the system’s records. It establishes neither that those records reflect the branch accurately, nor that money is missing or a particular person is responsible. For the older Horizon versions it examined, the High Court also found a material risk that errors in data entry, transfer, or processing could affect branch accounts.1 That is not a claim about every entry or every later state of the system.

The decisive issue was unequal inspectability. Fujitsu had privileged means of intervening in transaction data, while the Post Office had reports unavailable to branches; the court found some controls and records inadequate. Audit data could resolve disputed corrections but depended on access from the organizations whose system and claim were under review. The Court of Appeal accordingly distinguished whether Horizon had been essential to the evidence in each case.

Trust therefore requires more than technical consistency or the possibility of a later investigation. Counterevidence must reach the responsible person while the outcome can still change. A perfect audit trail offers little protection once a debt, sanction, or decision is practically irreversible.

What transfers to AI

AI-generated outputs introduce additional transitions between evidence and claim. A cited source may exist without supporting the sentence attached to it. A summary may contain accurate statements while omitting an objection that changes the decision. Even a faithful summary does not automatically warrant the recommendation that follows. This article makes that transfer; the British courts did not address AI.

The mechanism implies four requirements for organizational use:

  1. Access to evidence: Material claims lead to retrievable, versioned sources. Reviewers can inspect evidence beyond the interface and summary under review.
  2. Controlled intervention: Privileged changes to the system or its data are limited, attributable, and recorded. Actual behavior is monitored after deployment.
  3. Effective challenge: An objection reaches a responsible person before the decision is irreversible. That person can override the output, pause its use, and initiate an investigation.
  4. Proportionate assurance: Review effort reflects the particular use and the consequences of error. Trust attaches to a use under known conditions, not to a system in the abstract.

The voluntary NIST AI Risk Management Framework assigns related responsibilities across the system lifecycle. It calls, where appropriate, for independent assessors, feedback and appeal processes, and clear responsibility for overriding or correcting outputs, deactivating systems, and responding to incidents.3 These measures do not certify any individual output.

Even a careful process can fail. Evidence remains incomplete, independent reviewers may share the same mistaken premise, and an appeal process may exist on paper without practical force. Warranted trust is therefore not demonstrated by offering a channel for complaints. It is demonstrated when a reasoned challenge can still defeat the output and change the decision.

Footnotes

  1. High Court of Justice, Bates & Others v Post Office Limited (No. 6: Horizon Issues), [2019] EWHC 3408 (QB), especially paras. 905–913 and 963–1030. The findings concern the versions and periods examined in the litigation; the court expressly cautioned against applying them wholesale to Horizon as it stood in December 2019. ↩ ↩2

  2. Court of Appeal (Criminal Division), Hamilton & Others v Post Office Limited, [2021] EWCA Crim 577, especially paras. 1–5 and 447. The court allowed 39 appeals on both grounds and dismissed three. ↩

  3. Elham Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), National Institute of Standards and Technology, 2023, especially MEASURE 1.3 and 3.3 and MANAGE 2.4, 4.1–4.3. ↩

Oliver Wrede writes and teaches on interface design, knowledge systems, and the architecture of intelligence in organizations. He is interested in how humans, institutions, and machines reason together — and how design shapes the quality of that reasoning.

More from Oliver Wrede

Intelligence Architecture

The Business Value of Critical Thinking When AI Does the Work

AI agents can produce analyses and recommendations in seconds. Their business value also depends on whether employees ask useful questions, assess evidence, and learn from the results. Intelligence Architecture therefore needs to develop people's capabilities. How can everyday work support that development, and how can an organization assess its value?

Oliver Wrede·
Intelligence Architecture

AI and the Unwritten Rules of an Organization

Process documents describe only part of how an organization works. AI that answers from documents and stored data can miss informal responsibilities and essential coordination. This article asks how that work can be recorded for AI without eliminating useful discretion or subjecting staff to new forms of monitoring.

Oliver Wrede·