Dr. Mark Testa stands inside of the UNC School of Social Work on June 29, 2017, at the Tate-Turner-Kuralt Building in Chapel Hill. Dr. Testa is a Sandra Reeves Spears and John B. Turner Distinguished Professor. He is also a nationally recognized child welfare expert and the architect of innovative reforms in the fields of child protection and foster care. (Johnny Andrews/UNC-Chapel Hill)
Child welfare is running an evidence deficit.
There are too few programs in circulation with credible evidence of their effectiveness in improving the safety, family permanence and well-being of vulnerable children. Of the 452 programs catalogued by the California Evidence-Based Clearinghouse for Child Welfare, only 31 (7 percent) have received the highest acknowledgment of effectiveness, a rating of “well-supported.” Just five of those 31 are specifically designed or commonly used for children served by the child welfare system.
The Family First Prevention Services Act will enable states to claim federal Title IV-E entitlement funds for services to prevent the use of foster care beginning October 1, 2019. But it also mandates that at least one-half of that money has to be spent on programs that meet the standards of being “well-supported.” There is a pressing need to expand the supply of programs that meet those standards by increasing the rigor of existing systems for outcomes monitoring, program evaluation and quality improvement.
Programs receiving a rating of well-supported must have results supported by at least two randomized controlled trials or quasi-experimental evaluations of the program’s effectiveness in different, usual care settings. Given the sizable evidence deficit, public and private child welfare agencies worry they won’t be able to satisfy the rigorous evidence standards necessary to receive federal funding.
Fortunately, there are rigorous evaluation designs that can quickly become incorporated into existing administrative data systems. In medical settings, they are called pragmatic trials. By automating the unbiased allocation of children, families or providers to intervention and comparison groups, it is possible to generate credible evidence of the comparative effectiveness of an intervention compared to usual or alternative services.
By tracking the desired outcomes with existing administrative data, it is practicable to ramp up the testing of promising interventions at low cost to determine what works better, for whom, and under what conditions.
The feasibility of incorporating rigorous evaluation designs into existing data systems has been repeatedly demonstrated through the Title IV-E waiver experiments launched by states in the late 1990s and early 2000s. Assisted guardianship demonstrations fielded in Illinois, Tennessee and Wisconsin relied on the automated randomization of sibling groups to a program that offered relative caregivers a choice: guardianship subsidies, adoption assistance or long-term foster care.
Through this automated randomization, states were able to demonstrate that offering a guardianship assistance option boosted permanency rates overall, saving taxpayer dollars on longer stays in foster care. Due to the rigor of the evaluations and replicability of the results, the Congressional Budget Office scored the program a cost saver. In 2008, the Guardianship Assistance Program became federal law with passage of the Fostering Connections to Success and Increasing Adoptions Act.
Similar uses of automated allocation routines and administrative data to test child welfare interventions have also borne fruit. The Illinois Drug Recovery Coach and Birth through Three IV-E waiver demonstrations utilized the rotational assignment of cases to local offices and provider agencies to form statistically comparable groups. The Safe Families for Children evaluation funded by the Laura and John Arnold Foundation relied on the alternation method to allocate every other referral to the intervention or services-as-usual. A low-cost rigorous evaluation funded by the Coalition for Evidence-Based Policy involved the allocation of eligible births to a postpartum nurse home visiting program based on the weekday the child was born.
Automating unbiased assignment to treatment groups and tracking outcomes with administrative data hold great promise for expanding the supply of evidence-supported programs. But it is only a first step. States will need to mount multiple demonstrations in collaboration with public and private agencies.
Even then, building evidence of program effectiveness is not an easy task. Reviews of randomized controlled trials conducted in multiple arenas show that for every five promising interventions, one will be identified as a reproducible success. But those odds should not sap morale; rather, it should challenge our sector to multiply evidence-building efforts.
Improving upon existing practice is hard, repetitive work. The better the child welfare workforce is trained in the competencies of evidence-building, the greater the motivational investment in rigorous evaluation, increasing staying power in the face of predictable setbacks. Eventually, the yield of well-supported programs will increase.
The Administration for Children and Families has funded the development of several evidence-building frameworks and online curricula that can help prepare the workforce for fulfilling the purposes of the legislation. My reading of the Family First Act is that states will be able to claim half of the costs of training public agency staff and members of approved child welfare agencies to oversee and evaluate the services aimed at preventing foster care.
Let’s not shrink from the challenge. The time for strengthening the evidence-building competencies of the child welfare workforce is now.
Mark Testa is the Sandra Reeves Spears and John B. Turner Distinguished Professor in the School of Social Work at the University of North Carolina at Chapel Hill.
If you are interested in reading more about federal child welfare and juvenile justice policy, read our annual special issue “Kids on the Hill: A Special Issue on Child Welfare Policy” by clicking here!
