Note: This article was updated Monday, June 5 at 12:42am EST
Facing criticism and the pressure of public opposition from dozens of child welfare researchers, the California-based RAND Corporation has pulled down a child welfare spending simulation that projected billions of dollars in savings from investments in maltreatment prevention, family preservation services and kinship care.
The RAND report was published on its website on May 23, and taken down Friday, June 2.
RAND intends to review the model’s assumptions about the lifetime rates at which children are involved in the child welfare system, and plug new numbers into the existing model to produce a second report.
According to the statement replacing the report on the RAND website:
“In the wake of the release of our findings on May 23, 2017, we received some feedback from the wider child welfare research community about several of the inputs used in our model specifically those related to lifetime rates of child maltreatment and resulting engagement with the child welfare system.”
The firm said it believes the savings projections will not vary much, but child welfare researchers say there might be other problems with the model.
“We identified clear errors sufficient to warrant a retraction,” said Children’s Data Network Co-Director Emily Putnam-Hornstein, in an e-mail to The Imprint. “But the lifetime estimates were far from our only concerns.”
The report of the simulation, “Improving Child Welfare Outcomes,” has been “withdrawn pending further review” by RAND, according to the statement on the page of its website that once housed the report.
The RAND project, which was funded by a grant from the Pritzker Foster Care Initiative, simulates the flow of children through the child welfare system. It then models how investments in the expansion and improvement of evidence-based services would affect overall public spending.
The report found that investments in prevention, family preservation and kinship care would result in a total of $35.1 billion: a $4.3 billion increase in prevention spending, a $22.8 billion savings from family preservation, and a $16.6 billion savings from kinship care.
The firm said it used annual rates as a proxy for lifetime occurrence of a number of key points in the system: screened-in reports of maltreatment; substantiations of maltreatment; and entry into foster care.
These were the inputs challenged by a group of 34 researchers focused primarily on child welfare data. They outlined to RAND in a letter that the most recent cumulative estimates were, in all three of those instances, ten times higher than the ones used for the simulation.
Disparity between the model’s assumptions and recent research estimates had been noted by Chronicle of Social Change Publisher Daniel Heimpel in an article about the report the day it was released. Later, during a webinar moderated by Heimpel, the report’s lead author Jeanne Ringel said RAND had conducted sensitivity tests with higher maltreatment figures and recorded similar outcomes in terms of spending and saving.
The researchers, led by Putnam-Hornstein, conducted a conference call with RAND officials earlier this week.
“We reviewed the problems and asked that the report be pulled immediately,” Putnam-Hornstein said. “RAND refused, claiming their baseline estimates were simply ‘conservative.’”
Ringel, in an e-mail to The Imprint, said “at no point did we refuse to take the report down.”
She said Putnam-Hornstein “told us that the report needed to be taken down by 5:00 PM, or approximately six hours later, or the letter would be published. We informed Putnam-Hornstein that we would consider her comments and decide the best course of action. After doing so, we decided to take the report down.”
Hours after the call, Ringel notified Putnam-Hornstein and the other researchers that “the report does require modification,” and that “we are hoping to take the report down from the website either later today or first thing tomorrow.”
On Thursday evening, the report remained on the RAND website. The researchers requested that The Imprint publish an open letter they had written publicly calling for a retraction. The letter was scheduled to run Monday, but was withdrawn by the researchers after RAND took down its report.
“Much is risked by putting forth estimates and models that are known to have serious flaws,” Putnam-Hornstein wrote in an e-mail Thursday to Ringel, the report’s lead author. “When the flaws inevitably surface, it calls into question any confidence in related implications – even if these happen to remain useful.”
The report was removed the following day. The statement published by RAND promises to re-run the model with the higher numbers cited by the 34 researchers.
“We are currently evaluating how altering our assumptions to reflect higher lifetime rates would affect the policy scenario results,” RAND said in the statement. “Based on preliminary modeling, we don’t think the results will change materially.”
It is unclear whether RAND’s second attempt at the report will only change the lifetime rate assumptions, or whether it will review the other assumptions involved in the simulation.
Ringel, in an email to The Imprint, said that RAND is “updating the model inputs to reflect higher lifetime rates. As part of that process, we will review the model assumptions and make any other changes as appropriate.”
Putnam-Hornstein said she was pleased by the withdrawal of the report, but remains “concerned that RAND continues to minimize the importance of the error that was made, its relevance to the conclusions drawn, and its implications for the credibility of the model and the research overall.”
Ringel disagreed.
“We certainly are not minimizing the issues raised; otherwise we wouldn’t have withdrawn the report,” said Ringel. “We are open to constructive criticism, which helps advance the quality and rigor of our research. We take the feedback received about this report very seriously and are addressing the error and reviewing its implications for both the model and the overall research.”
The technical review team for the simulation included one independent researcher: Fred Wulczyn, senior fellow at Chapin Hall. He confirmed to The Imprint that RAND has contacted him about continuing with the project.
“It’s not appropriate for me to comment on a review process that’s still underway and a report that’s still in development,” Wulczyn said.