Robodebt: How Estimated Income Became a Debt
Australia’s Robodebt process used longer-term income records to claim that people owed money. Software spread income evenly over shorter periods, although its actual timing was unknown. The case shows what goes wrong when a public agency treats a calculation’s assumption as an established fact.

Australian authorities used the Robodebt system to derive debt claims from income records. The software distributed totals reported over longer periods evenly across individual fortnights. The data did not establish whether the income had actually arrived that way. The process nevertheless produced a quantified debt. The Robodebt Royal Commission’s final report documents this practice.
The problem lay in the claim drawn from the calculation. A discrepancy that had previously prompted an inquiry could be filled with substitute data and treated as a debt in the online process. The Commonwealth Ombudsman’s 2017 report also shows that the same software could calculate correctly when actual income for every fortnight was entered in full.
This distinction matters for AI as well. A precise number does not establish what an organization concludes from it. The article examines how a model assumption becomes grounds for a decision and why observed data, added assumptions, and permitted uses need to remain distinguishable.
The necessary evidentiary boundary
“False” does not necessarily mean that the division was faulty. The tax records established an aggregate; Robodebt supplied its distribution over time and concealed which part of the result had been observed and which had been modeled. At the same time, an investigative lead became a debt: missing entries could be replaced by the average while the practical work of rebuttal moved to the affected person.
The evidence supports a tightly bounded claim. In an internal comparison of 229 cases, averaging almost never reproduced the manual calculation; the group was not identified as a representative sample, however, and deviations could also favor the individual. In Prygodicz v Commonwealth of Australia (No 2), apportionment without evidence of constant income could not establish a debt. This is not a general prohibition on averaging but a limit on claims whose evidence must be more temporally precise than the available data.
How a model output becomes a fact
Pseudo-precision here is a mismatch between the reach of the data and the reach of the claim. Several organizational steps turned the result into an enforceable debt: the tax authority supplied the total, the software distributed it, the process treated the result as an established liability, and the communication placed the burden of disproof on the individual. Reviewing the formula alone is therefore not enough. Its output’s status in the process must be reviewed too.
The same test applies when AI systems supply organizations with scores, rankings, or recommendations. At least four things must remain visible: the data actually observed, the assumptions added by the model, the uncertainty of the result, and the decisions for which it may be used. Without that separation, a plausible compression can quietly become an institutional fact.
Applying the same test to our own models
Intelligence Architecture also names layers, numbers steps, and uses playbooks. These forms can preserve distinctions and support collective work. They can also make a provisional judgment look measured.
The site’s foundation article distinguishes four activities: sense, interpret, decide, act. They are analytical tools, not a claim about a universal sequence. Actual work can loop, skip steps, or involve several activities at once. The four terms earn their place only when they change a concrete inquiry or make a break in the process visible. Their number proves nothing.
Readiness and maturity scores face the same question. Before an aggregate is allowed to describe an organization or direct investment, its weighting, missing data, borderline cases, and permitted uses must be visible. The critical review therefore starts before the arithmetic: what do the observations already support, what did the model add, and who turns the result into a reason for action?

