Design Theory

Robodebt: How Estimated Income Became a Debt

Australia’s Robodebt process used longer-term income records to claim that people owed money. Software spread income evenly over shorter periods, although its actual timing was unknown. The case shows what goes wrong when a public agency treats a calculation’s assumption as an established fact.

Macro view of a thickly painted oil surface: a massive dark brown block stands on a pale floor, with a single row of small identical slabs running through it. Behind the block they carry the same dark finish as the block itself; in front they are pale and hollow. A small indigo block stands exactly where the pale slabs turn dark.
Macro view of a thickly painted oil surface: a massive dark brown block stands on a pale floor, with a single row of small identical slabs running through it. Behind the block they carry the same dark finish as the block itself; in front they are pale and hollow. A small indigo block stands exactly where the pale slabs turn dark.

Australian authorities used the Robodebt system to derive debt claims from income records. The software distributed totals reported over longer periods evenly across individual fortnights. The data did not establish whether the income had actually arrived that way. The process nevertheless produced a quantified debt. The Robodebt Royal Commission’s final report documents this practice.

The problem lay in the claim drawn from the calculation. A discrepancy that had previously prompted an inquiry could be filled with substitute data and treated as a debt in the online process. The Commonwealth Ombudsman’s 2017 report also shows that the same software could calculate correctly when actual income for every fortnight was entered in full.

This distinction matters for AI as well. A precise number does not establish what an organization concludes from it. The article examines how a model assumption becomes grounds for a decision and why observed data, added assumptions, and permitted uses need to remain distinguishable.

The necessary evidentiary boundary

“False” does not necessarily mean that the division was faulty. The tax records established an aggregate; Robodebt supplied its distribution over time and concealed which part of the result had been observed and which had been modeled. At the same time, an investigative lead became a debt: missing entries could be replaced by the average while the practical work of rebuttal moved to the affected person.

The evidence supports a tightly bounded claim. In an internal comparison of 229 cases, averaging almost never reproduced the manual calculation; the group was not identified as a representative sample, however, and deviations could also favor the individual. In Prygodicz v Commonwealth of Australia (No 2), apportionment without evidence of constant income could not establish a debt. This is not a general prohibition on averaging but a limit on claims whose evidence must be more temporally precise than the available data.

How a model output becomes a fact

Pseudo-precision here is a mismatch between the reach of the data and the reach of the claim. Several organizational steps turned the result into an enforceable debt: the tax authority supplied the total, the software distributed it, the process treated the result as an established liability, and the communication placed the burden of disproof on the individual. Reviewing the formula alone is therefore not enough. Its output’s status in the process must be reviewed too.

The same test applies when AI systems supply organizations with scores, rankings, or recommendations. At least four things must remain visible: the data actually observed, the assumptions added by the model, the uncertainty of the result, and the decisions for which it may be used. Without that separation, a plausible compression can quietly become an institutional fact.

Applying the same test to our own models

Intelligence Architecture also names layers, numbers steps, and uses playbooks. These forms can preserve distinctions and support collective work. They can also make a provisional judgment look measured.

The site’s foundation article distinguishes four activities: sense, interpret, decide, act. They are analytical tools, not a claim about a universal sequence. Actual work can loop, skip steps, or involve several activities at once. The four terms earn their place only when they change a concrete inquiry or make a break in the process visible. Their number proves nothing.

Readiness and maturity scores face the same question. Before an aggregate is allowed to describe an organization or direct investment, its weighting, missing data, borderline cases, and permitted uses must be visible. The critical review therefore starts before the arithmetic: what do the observations already support, what did the model add, and who turns the result into a reason for action?

Oliver Wrede writes and teaches on interface design, knowledge systems, and the architecture of intelligence in organizations. He is interested in how humans, institutions, and machines reason together — and how design shapes the quality of that reasoning.

More from Oliver Wrede

Intelligence Architecture

The Business Value of Critical Thinking When AI Does the Work

AI agents can produce analyses and recommendations in seconds. Their business value also depends on whether employees ask useful questions, assess evidence, and learn from the results. Intelligence Architecture therefore needs to develop people's capabilities. How can everyday work support that development, and how can an organization assess its value?

Oliver Wrede·
Intelligence Architecture

AI and the Unwritten Rules of an Organization

Process documents describe only part of how an organization works. AI that answers from documents and stored data can miss informal responsibilities and essential coordination. This article asks how that work can be recorded for AI without eliminating useful discretion or subjecting staff to new forms of monitoring.

Oliver Wrede·