Artificial Intelligence

How a Risk Score Changed Child-Welfare Work

In Allegheny County, Pennsylvania, software helps staff decide whether a child-welfare referral should be investigated. In its first version, a score of 18 or more made investigation the default; only a supervisor could approve an exception. This article examines how that rule changed staff responsibilities and decision authority.

A close-up of a thickly painted oil painting: a tool wall with neatly hung wrenches, one slot outlined in burnt orange; the small indigo block does not hang on the wall — it sits below, embedded in a network of recessed channels that visibly rearrange themselves around it.
A close-up of a thickly painted oil painting: a tool wall with neatly hung wrenches, one slot outlined in burnt orange; the small indigo block does not hang on the wall — it sits below, embedded in a network of recessed channels that visibly rearrange themselves around it.

The Allegheny Family Screening Tool began supporting child-welfare hotline staff in Pennsylvania in August 2016. It calculated risk scores from administrative records about the people named in a referral. The first documented version attached a rule to the score for later out-of-home placement: at 18 or above, the referral should be investigated unless a supervisor approved an exception.

That score established neither whether the allegation was true nor whether a child faced immediate danger. It estimated the probability of placement outside the home within two years, conditional on the referral first being investigated. Even so, it helped determine what staff were expected to do next.

The connection between calculation and procedure makes this case useful for examining how software changes decision authority. It also matters for current AI deployments: an assessment gains influence when the organization specifies what should follow from it and who may depart from that rule.

What 18 meant in the model

Version 1 produced two scores for each child. The output that acquired procedural force was the placement-risk score. It estimated the likelihood that a child would be placed outside the home within two years, conditional on the referral first being investigated. It did not determine whether the allegation was true, and it was not a measure of immediate danger. Those distinctions mattered because hotline staff used the language of risk and safety for information arriving in the call itself.

The model was a logistic regression based on linked administrative records about people associated with a referral. It did not use the substance of the allegation. A screener might therefore hear about an acute incident that was entirely absent from the model. In this article, AI refers broadly to a system that derives an assessment from data; nothing in the case depends on a language model or an autonomous agent.

The 1-to-20 display was not a probability scale. Predicted probabilities were divided into twenty bands containing roughly five percent of cases each. Equal steps on the display did not imply equal changes in estimated risk. The county’s boundary at 18 identified the highest three bands, or 15 percent of the distribution.

The county’s published overview dates the launch of Version 1 to August 2016. Version 2 replaced it in December 2018 with a revised algorithm, data sources, and associated policies. Version 1 remains instructive because ordinary regression acquired organizational force through workflow rules, long before the present interest in generative AI.

The number changed who had to disagree

Below 18, the score contributed to a screener’s recommendation. At 18 or above, the system labeled the case a “mandatory screen-in.” Only a supervisor could keep such a referral from proceeding. The result was a default with an exception, not an automated disposition.

Early operating data make that distinction tangible. Supervisors overrode almost one quarter of mandatory screen-ins, and override rates varied between supervisors. Screening outcomes were more closely associated with the workers’ own risk-and-safety assessments than with the AFST score. According to the research team’s case study, staff were not simply rubber-stamping the model.

The overrides did not restore the former process. Once policy made investigation the expected outcome, departing from it required attention, disagreement, and the correct authority. The exception was therefore part of the system’s influence, and variation between supervisors became an implementation issue worth examining.

This is also where evaluation has to follow the authority map rather than the model alone. It needs to identify the claim made by the output, whether that output supplies information, sets a default, or starts an action, and who may depart from it. Measures must cover missed cases, unequal effects, and the burden placed on each role as well as model performance.

The county’s process evaluation documents the broader redistribution of judgment. Adoption alone would be a poor measure of this intervention: it cannot show whether the new arrangement improved decisions or merely moved work between roles.

What the evidence cannot separate

Because the county altered roles and rules at the same time, the deployment was not a controlled test of the score’s effect. Later changes in screening could have come from the prediction, the revised responsibilities, supervisor practice, or their interaction. The case provides rich evidence about implementation but weak evidence for attributing an outcome to the model alone.

The validation history deepens that uncertainty. The developers later reported two mistakes that had made initial performance estimates too optimistic. Repeated referrals for the same child, and records for siblings on the same referral, could appear on opposite sides of the training-test split. Feature selection had also used the full dataset before validation. A corrected procedure changed the performance estimates, though not the deployed model.

Even the revised evaluation faced a “selective labels” problem. Training and testing covered referrals that had been investigated, while the tool was meant to inform selection from all referrals. Comparable later outcomes were not available for many screened-out cases. Fairness could not be reduced to a metric either. Public agencies tend to hold more records about poor families and groups with greater exposure to government services; professional judgment carries biases of its own.

The record therefore leaves the central causal question open. It does not tell us how much of any later change came from the prediction, the new distribution of responsibility, the default at 18, or the interaction among them.

Oliver Wrede writes and teaches on interface design, knowledge systems, and the architecture of intelligence in organizations. He is interested in how humans, institutions, and machines reason together — and how design shapes the quality of that reasoning.

More from Oliver Wrede

Intelligence Architecture

The Business Value of Critical Thinking When AI Does the Work

AI agents can produce analyses and recommendations in seconds. Their business value also depends on whether employees ask useful questions, assess evidence, and learn from the results. Intelligence Architecture therefore needs to develop people's capabilities. How can everyday work support that development, and how can an organization assess its value?

Oliver Wrede·
Intelligence Architecture

AI and the Unwritten Rules of an Organization

Process documents describe only part of how an organization works. AI that answers from documents and stored data can miss informal responsibilities and essential coordination. This article asks how that work can be recorded for AI without eliminating useful discretion or subjecting staff to new forms of monitoring.

Oliver Wrede·