Using an Algorithm to Predict Hospital Encounters, Any-Cause Injury and Abusive Injuries at CPS Intake
Last month’s Sounding Board discussed the controversy over use of an algorithmic tool (Allegheny County Family Screening Tool (AFST) to screen General Protective Services (GPS) reports in Allegheny County, Pennsylvania. We summarized the findings of an independent study of the use of AFST-1 conducted by researchers from Stanford, which found modest initial positive effects on “screening accuracy,” effects that decreased over time.
We questioned the logic of evaluating screening decisions based on re-report data that might have been affected by factors other than information contained in GPS reports, e.g., by the motivation and attitudes of mandated reporters.
Our main conclusion was that use of AFST had led to a reduction in GPS screen-out rates from 60% to 47%, the average GPS screen-out rate in Pennsylvania, in part by increasing the number of high-risk white families screened in for GPS services. We also noted that the rate of first entries into care declined by almost 20% from 2016-2020, the first few years of AFST implementation.
But the most important and potentially useful study of AFST was published in JAMA Pediatrics: “Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals.”
This study tracked hospital “encounters,” such as emergency room visits and hospitalizations, for 47,305 children with General Protective Services referrals (non-abuse) in Allegheny County from April 1, 2010, to May 4, 2016. Rates of any-cause injury and abusive injury were compared for children with an AFST score of 20 (the 5% of reports with the highest risk), and children with AFST scores of 1-10 (the lowest-risk 50% of reports).
The authors state: “we examined whether a risk model trained to predict future child protection events was sensitive to several external measures of child harm.” And “our objective was to validate the risk of placement into foster care as a reasonable proxy for child harm …”
However, what this study achieved is something more: the development of a child safety measure that is by far more cogent than the rates of re-report and multiple substantiations used by the federal Administration for Children and Families (ACF) and states’ child welfare systems to measure child safety. In addition, the study is a reminder of what is at stake in the debate regarding use of algorithms: protection of children from serious physical harm, either inflicted or unintentional.
Arguably, it is the lack of cogent safety measures tied to concrete physical harm that has made it impossible to ascertain whether child protection systems have — or have not — made incremental progress in achieving their mission. The outcome measure used in this study is not the only measure of child safety that should be utilized by state child welfare systems and ACF, but it is a vital one.
What the study of hospital encounters found
The authors state:
“…among children referred for maltreatment and classified as highest risk (AFST score of 20) the rate of experiencing an any cause injury was 14.5 per 100 compared with children who scored as low risk who had an any cause injury rate of 4.9 per 100. For abuse-associated injury encounters, the rate for high-risk children was 2.0 per 100 and that of low-risk children was 0.2 per 100 … for suicide and self- harm, the high-risk encounter rate was 1.0 per 100 and that of low-risk children was 0.1 per 100.”
In other words, the 5% of highest-risk children had an any-cause injury rate almost three times higher than 50% of the lowest-risk children, and the rate of abusive injury, and self-harm and suicide was 10 times the rate of the lowest-risk children. AFST is a powerful algorithm when applied to one of the most important safety outcomes in child protection.
An algorithm that can connect 5% of children reported to child protection services (CPS) mostly for minor neglect and associated risk factors with rates of any-cause injury or abusive injury 10 times higher than the rates for 50% of the lowest-risk children has the potential to direct resources to families that need them the most, and to inform discussion of how to revise definitions of child neglect. The predictive tool can also be used to separate CPS reports into an investigative track vs. an assessment track in child welfare agencies that employ differential response at intake.
High-risk vs. low-risk ratios for Black children were not as large as for other groups. For any-cause injury hospital encounters, “the rate for the high-risk group (of Black children) was 16.1 per 100 compared with 8.1 per 100 for low risk.” The abusive injury rate for the highest-risk Black children was 1.9 per 100 vs. 0.5 per 100 for the lowest-risk children. The authors do not speculate why ratios between high-risk children and low-risk children were substantially less for Black children in screening of GPS reports.
Given these important findings, it is surprising that the authors do not discuss the effect of use of AFST on numbers or rates of injury-related hospital encounters in Allegheny County. However, a recent randomized control trial of AFST in Colorado found a 32% decline in hospitalizations due to child injury for caseworkers using AFST compared with caseworkers engaged in business as usual.
If this study stands up to intense scrutiny, the most important question will not concern the tool or the rules for its use, but rather the services or interventions caseworkers and offices utilized to reduce child injuries. Were more children placed in foster care, or were in-home services provided earlier and to the families who needed them the most?
Risk assessment research often gives full attention to a tool and rules for its use while treating services like a bit player, not worthy of attention. However, predictive tools, in and of themselves, achieve nothing. It is the interventions that follow their use that should be given center stage.
Predicting foster placement of young children
The Cross Jurisdiction Model Replication Project (CJMR) developed by Putnam-Hornstein, Ahn and Vaithianathan, along with scholars from Alaska, Kentucky and Mathematica, utilizes 82 data elements from birth records to predict foster care placement by age 3 for birth cohorts. The authors comment that the study period includes “the year of life when maltreatment reporting, substantiation and foster care placement rates are highest (that is during infancy) and captured the period when about 70 percent of maltreatment fatalities occur (that is before age 3).” The predictive tool developed by CJMR was applied to birth cohorts, not CPS reports, a crucial difference.
Putnam-Hornstein’s 2011 study of fatal intentional and unintentional injuries among children in California with CPS reports by age 5 found that only one-fifth to one-third of young children who died from injury had been reported to CPS prior to death. To prevent most child injury deaths, prevention programs must reach families and offer services before a CPS report is received on the family. For infants and toddlers at highest risk of injury or death reported to CPS, CPS intervention is often too little, too late.
In AFST, Putnam-Hornstein and Vaithianathan have demonstrated that an algorithm developed to predict foster care placement can predict any-cause injuries and abusive injuries. The CJMR algorithm has the potential to be used for this purpose by prevention programs that reach out to parents absent a CPS report or immediately after a CPS report is received, regardless of whether the report is screened in for investigation or screened out. The CJMR algorithm “applied to the Allegheny County cohort … proved more accurate at differentiating risk (of foster placement) than a similar model created using Allegheny County’s own smaller date set.”
The CJMR study found that the cumulative incidence of a first foster care placement by age 3 for the California birth cohorts was 1.8 percent. The model was subsequently tested in Alaska and Kentucky, which used much the same methodology. Children in the birth cohort with the highest decile of risk scores had a placement rate of 0.68% by age 3. The model accurately predicted foster care placement (true positives) for 59.4% of highest risk children and incorrectly predicted foster care placement (false positives) for 40.6% of children in the highest-risk decile. The model was less successful when applied to birth cohorts in Alaska and Kentucky, i.e., the model accurately predicted 34.6% and 32.8% of placements for children in the highest decile. Curiously, the model was more accurate by far in predicting foster care placement for children in the second highest decile (75.7%, 57.6%, 56.8%) in all three states.
Concretely, what this means is that in birth cohorts whose placement rate to age 3 for the two highest-risk deciles of infants and toddlers was less than 1% (1.22% combined), an algorithm that utilized data from birth records accurately predicted foster care placement for one-third to three-quarters of this population. This is impressive targeting of a population of children at high risk of foster placement, any-cause injury and abusive injury by any reasonable standard. This model can be accessed on a GitHub website and has been made available to other jurisdictions at no cost.
According to the authors, “The model was not designed to identify individual children who would experience foster care placement. Instead, it aimed to use prediction methodologies to understand population wide differences in foster care placement.”
Translation: The CJMR is not a case-planning tool for working with a specific family; it was designed to inform prevention/early intervention strategies for the highest-risk populations of infants and toddlers in birth cohorts, hopefully prior to CPS reports. Like all risk assessment models, however developed, the CJMR has a high rate of false positives. It would be a misuse to refer to its predictions in dependency actions that follow involuntary child removal.
How to use an algorithmic tool to prevent any-cause injuries, abusive injuries, and foster care in birth cohorts
The heated controversy over use of AFST in Allegheny County has had the unfortunate effect of conflating the use of “big data” predictive tools with an expansion of CPS involvement with at-risk children and families, absent allegations of child maltreatment. However, it is possible to utilize the tools to inform expansion of public health services to the highest-risk families prior to or separate from a CPS investigation. One does not need a predictive tool to understand the importance of reaching out to parents of young children enrolled in publicly funded substance abuse or mental health treatment, or when there has been a pattern of domestic violent incidents in a family. However, there may be other high-risk family profiles that are not obvious, even to experienced caseworkers.
Preventive and early intervention services should include:
- Poverty-related services that address basic needs such as food, housing, medical/dental care and transportation.
- Evidence-based, skill-oriented services designed to improve emotionally responsive parenting.
- Emotional and practical support for completion of substance abuse and mental health treatment programs; and DV services.
- Childcare services and respite care urgently needed to reduce childcare burden in low-income, single-parent families.
- Support services for care of disabled and chronically ill children, an element of risk in families often overlooked in CPS assessments.
- Support for continuing education and job training required to increase family income and spark hope for the future.
Application of a predictive tool at birth and expanding the role of public health nurses is not the only strategy for delivering prevention/early intervention services. Some communities may choose to develop comprehensive family support centers that parents can access as they wish. However, waiting for troubled families to seek out services as they feel the need increases the likelihood that early intervention programs will serve more lower-risk motivated parents than high-risk families who want to be left alone.
Every prevention program is faced with the challenge that people who receive services will likely not be the neediest, while families with the greatest need may avoid helpers of any description. Black, American Indian, and Latino families may be reluctant to seek services due to past negative encounters with public agencies and distrust of professionals.
Utilization of a predictive tool applied to all births can enable outreach and offer of services to the highest-risk families and children before a family is reported to CPS.
Some key features of the use of AFST and of the CJMR algorithms bear repeating:
- AFST is a screening tool that identifies the 15%-25% of children at highest risk for foster care and injury-related hospital encounters; neither AFST or CJMR are case-planning tools utilized by investigators.
- AFST and CJMR can be used with a wide range of services and interventions; they are targeting tools, not an intervention strategy. One jurisdiction may use them well, another poorly.
- The main strength of AFST is its ability to differentiate the5% of high-risk children with a 3-to-10-times-higher rate of any-cause and abusive injury-related hospital encounters than the lowest-risk 50% of families.
In collaboration with other public and private human service agencies, child protection programs should develop and test practice models for this relatively small group of high-risk children.