Checklists, Big Data and the Virtues of Human Judgment

Los Angeles County struggles to strike the right balance between human judgment and increasingly sophisticated predictive tools when determining the risk that a child will be abused.  

On weekdays, calls to Los Angeles County’s child abuse hotline reach their peak between 2 p.m. and 6 p.m.—right after school. On average, 70 to 80 calls about child maltreatment in Los Angeles County reach the hotline per hour during that span, according to the Department of Children and Family Services (DCFS), the agency charged with responding to alleged abuse.

There are about 85 social workers manning the phones at any given time. They ask callers to explain how child abuse or neglect took place.

The number of calls made to the largest child welfare system in the United States creeps up each year, said Carlos Torres, an assistant regional manager for the DCFS hotline. In 2014, the hotline received 220,000 calls, he said.

After listening and marking down answers on a computer program, the social workers decide whether a situation meets the criteria for an in-person response. They also decide whether DCFS should respond by the end of their current shift, within 24 hours, or within five days, Torres said.

These decisions, based on small bits of information shared by a caller, determine where DCFS directs its limited human resources. DCFS responds with an in-person investigation to 35 percent of the calls, Torres said. In these cases, a social worker drives to the home, interviews the family, gathers information, and enters his or her findings into a web-based decision-making tool, which, like a questionnaire that an insurance company gives to prospective clients, estimates risk; in this case, risk that a child will be abused.

When everything goes right, DCFS can save a child from harm. When something goes wrong, the result can be heartbreaking. A 2011 report on recurring systemic issues that led to child deaths in Los Angeles County put the onus largely on flawed investigations and problems with the decision-making tool employed. In the search for solutions, public officials have looked toward new technologies, such as analytics software used primarily by private companies, to see if that can keep more children out of harm’s way. As public officials make these kinds of inquiries, in Los Angeles County and across the globe, they confront the conundrum of human judgement versus machine. Some say technological advances hold the answers, while others say that only savvy people are up to the task. 

How DCFS Assesses Risk and Safety

DCFS uses Structured Decision Making (SDM), an actuarial decision-making model, to assess whether and to what extent children are at risk of maltreatment. DCFS adopted SDM as a limited-release pilot program in 1999 and later expanded its use across Los Angeles County. It is the most widely used risk assessment toolkit in the U.S. today, according to the Children’s Research Center,  a division of the National Council on Crime and Delinquency, which created SDM.

DCFS Social workers access SDM on their computers. They find information and make observations about a family during their investigations. After processing this information, the tool classifies a family’s risk level as very high, high, moderate or low.

SDM contains different tools designed for use at different stages of an investigation. When calls are made to the DCFS hotline, social workers use a portion of SDM that helps them prioritize responses. After they visit homes, social workers use the model’s safety assessment and risk assessment tools. Later on, they use reassessments tools to help them decide whether to close a case or continue their intervention.

In 2008 Researchers at the University of California at Berkeley evaluated the effectiveness SDM in Los Angeles County’s Service Planning Area (SPA) 6, which includes Compton and most of South LA. The study found that two-thirds of social workers thought SDM improved their decision-making.

Other reports have criticized SDM and its implementation. In 2012, The Los Angeles County Children’s Special Investigation Unit (CSIU) released an alarming report focusing on child deaths. The report looked at 14 child fatalities and one critical injury that were caused by severe abuse. In 11 of these cases, a social worker either failed to use SDM or misused it.

Two-month-old Cynthia F. died when her intoxicated parents left her face down inside her crib, the report explains. The DCFS hotline was contacted when Cynthia was born because her mother was in an in-patient drug program and on methadone. The SDM hotline tool was left blank, so no investigation was required. The hotline employee who responded to the call was unaware that Cynthia’s mother had already lost custody of four other children due to drug abuse, according to the report.

In a 2014 Chronicle of Social Change column about risk assessment, Ira M. Schwartz, former dean at the University of Pennsylvania’s School of Social Work, cited research done by Dr. Jane Barlow and her colleagues in the United Kingdom, which concluded there is limited evidence that actuarial-based tools, such as SDM, are effective.

“The available tools are simply not accurate enough and fall short of what child welfare staff need in order to make truly well-informed decisions,” Schwartz wrote.

The CSIU report also says SDM can be manipulated to justify pre-determined outcomes or avoid higher-level review. SDM is a manually operated tool, meaning that it only uses information that a social worker puts into it.

DCFS Director Phillip Browning said he agrees that potential manipulation is a limitation of SDM. Under his leadership, DCFS is now looking at alternatives. The one getting the most attention would use predictive analytics made possible through merging and rapidly analyzing huge public records data sets to ascribe a risk score to families that come into contact with the system.

Testing out Predictive Analytics

Predictive analytics is new paradigm, where big data can be crunched in a way that helps determine which children are at greater risk of being abused. Once a risk level has been determined, child protective service agencies could direct their limited human resources toward families that are considered high-risk and help social workers to prioritize these cases.

Predictive analytics tools are widely used in business, but they have not yet been applied to child protection. In New Zealand and Allegheny County, Pennsylvania, human services agencies are working on plans to start using them for child protection.

When the Los Angeles County Blue Ribbon Commission on Child Protection was tasked with reducing child deaths, its recommendations included the adoption of a predictive analytics tool created by Eckerd, a private service provider in Florida.

A few years earlier, in 2008, DCFS had started looking into its own predictive analytics model called Approach to Understanding Risk Assessment (AURA). This program was created by SAS Institute, a $3 billion company that develops business analytics software and services.

Through AURA, SAS has been analyzing the risk factors of anonymous old cases, to show DCFS whether it can identify the risk factors that correlate with critical incidents, such as severe injuries to children or child deaths. In SAS Institute’s most recent analysis, it looked at DCFS’ de-identified data from January to June of 2014 and using the AURA algorithm, it connected high risk factors with child abuse or neglect at an 85 percent correlation rate, according to Jennie Feria, an executive assistant to Browning.

AURA creates a numerical risk score, “along the same lines as your FICO score,” which is used in credit reports, Feria said.

When AURA looked at 36,000 DCFS cases over a certain period, it identified 11,000 of them as high risk, Browning said during a visit to a class called Media for Policy Change at the University of Southern California (USC).

“If I’m a [social] worker and I have 30 cases, and this tool shows that in those 10 cases, the likelihood of something bad happening is greater than the others, it gives you a sense of where you should put your time,” Browning said. “If you have limited time in the day and can only look at two cases, which two should you put your time on?”

Concerns about Using Decision-Making Tools

Kathryn Icenhower, CEO of Shields for Families, a nonprofit organization that provides comprehensive services for high-risk families in Los Angeles, thinks SDM is flawed because it focuses on families’ weaknesses, not on their strengths. A predictive analytics tool could amplify that issue, she said, by assigning families with scores akin to credit scores.

“Your past is stuck with you regardless of what you might have done to address your issues,” Icenhower said.

Icenhower believes that the importance of high-quality employees towers above the importance of high-quality tools.

“Families are not science,” Icenhower said. “We don’t have the ability to predict what families are going to do. We can’t do this with tools and data. We can use that to inform us, but if you want to do a good assessment you need a good assessor, not a good assessment tool.”

The Importance of Data Sharing

Los Angeles County committed itself to an overhaul of its child welfare system in 2013, forming a Blue Ribbon Commission on Child Protection.The report emphasized that a lack of information-sharing among county departments that work with children and families prevented crucial information from getting to DCFS. In many cases, county departments had records that suggested a family was at risk, but they did not share it with one another or make it readily available.

The Los Angeles County Office of Child Protection, which was formed to implement the reforms in this report, opened in February. The new office’s interim director, Fesia Davenport, has chosen data-sharing as one of the office’s initial projects. The office is now working on proposals to increase the sharing of data that’s already legally accessible among agencies that work with children and families, explained Carrie Miller, a manager at the Office of Child Protection.

Los Angeles County has a Family and Children’s Index (FCI), a database that some county agencies use to add information about families with whom they come in contact. This system, developed in the 1980s, usually just displays that another agency has a relevant record, without specifying what happened, Miller explained.

“We are looking at how to give information to social workers that would inform their investigations,” Miller said. “It’s not existent in a system that they have access to. They have to track it down.”

When social workers use SDM, they rely on administrative records as well as information that families voluntarily disclose during home visits, explained Isis Lopez, a social worker at DCFS. When social workers take a new case and administrative records are limited, they have to rely on what families tell them, Lopez said.

A modernized data-sharing system could make social workers depend less on information families tell them. Asking a parent whether he or she has had a drug problem might not yield the same answer as a report from Substance Abuse Prevention and Control.

Research on Who is ‘High-Risk’

In late 2014, The Children’s Data Network (CDN), a data and research collaborative housed at USC that focuses on the linkage and analysis of administrative records, released a slew of studies on children born in 2006 and 2007. The collaborative found that two areas of Los Angeles County had rates of substantiated child abuse or neglect that were significantly higher than all of the others.

The sprawling county is broken down into eight Service Planning Areas (SPAs) to allow the Department of Public Health and other agencies to provide more relevant health and clinical services. The SPA with the highest rates was SPA 1, which comprises the rural Antelope Valley. There, 9.5 percent of children born in those two years had a substantiated case of child abuse or neglect. Second highest was SPA 6, which contains Compton, Lynnwood and South Los Angeles, where 7.9 percent of children born in those years had a substantiated case.

The average in California was 5.1 percent, and the average in Los Angeles County was 5.2 percent, according to CDN. LA County’s lowest rate of substantiated cases—1.7 percent—was in SPA 5, which includes the wealthier communities of Santa Monica, Beverly Hills, Malibu and Venice, according to the study.

The Children’s Data Network partnered with the California Child Welfare Indicators Project in 2014 to release a study that showed that four factors were significantly correlated with reported and substantiated child abuse and neglect. Families in which children were born without fathers listed on their birth certificates; mothers were on public health insurance; mothers had not completed high school; mothers were teenagers—these were all connected with risk.

The studies also found high rates of re-reported maltreatment of infants after unsubstantiated reports.

“Data from the current study, however, do not provide evidence of failed or ineffective services,” the CDN website reads. “Rather, findings align with earlier analyses that underscore just how few infants and families may receive services after an initial report of abuse or neglect.”

Technological Tools and Human Judgement

The Office of Child Protection is planning to meet with county department heads soon to discuss its proposal to increase data sharing, and then to present its idea to the Los Angeles County Board of Supervisors, said Karen Herberts, a staff member at the office.

DCFS is moving forward with a Request for Proposals (RFP) process to put the idea of a predictive analytics model into practice, but the process will take about two years, Feria said in an e-mail. Going through the required RFP process means other companies can propose analytics models in competition with SAS. After DCFS selects a vendor, the Los Angeles County Board of Supervisors will have to vote on it.

Meanwhile, in western Pennsylvania, the Allegheny County Department of Human Services has teamed up with an international research team to create a predictive analytics tool. In New Zealand, Parliament is even closer to integrating big data and risk models into its child protection practices. But no child protective services agency has put predictive analytics into practice yet.

As he considers the merits of a predictive analytics tool in Los Angeles County, Browning is also stressing that, even in the absence of new technologies, social workers at DCFS can improve their decision-making.

“The three most important things that I tell staff every day are common sense, critical thinking and accountability,” Browning said to graduate students at USC. “Common sense is pretty rare these days, guys. And so if all of our workers used common sense and critical thinking I think we’d be a much better organization.”

Your support allows The Imprint to provide independent, nonpartisan daily news covering the issues faced by vulnerable children and families.

Subscribe or Donate

National nonprofit @PSYyouth is staffing up to extend its reach on preventing youth #homelessness

OPINION: Residential care leaders say its time to root out the bad actors in #childwelfare