Re: good ! question?

Margaret,

You raise two really interesting points. One is the issue of what we use for outcomes (in this instance, referrals, substantiations and removals ---we try to use injury when we can, but most systems don?t reliably collect that information). The reason we always test tools on each of these measures is that each one has limitations with respect to how ?real world? they really are. If we had knowledge certain that abuse/neglect occurred we?d only need the one outcome measure. Lacking that certainty, if we relied solely on re-referrals we?d include as ?recurrence? many families who did not maltreat after the assessment, but were reported. If we relied solely on substantiation, we?d miss families where maltreatment occurred, but could not be substantiated. There is plenty written and discussed these days about how substantiation decisions can vary pretty wildly for reasons that have little to do with whether a child was maltreated. Removals as an outcome suggest a pretty good indicator not only that something happened, but that it was severe enough to result in removal. There are false positives and false negatives in each outcome. Each is imperfect, but when the risk level has a strong correlation with each, it helps mitigate the potential bias in any one outcome measure.

The rates of re-referral, re-substantiation and removal within each risk category will always decline from measure to measure: there will be more referrals than substantiations and more substantiations than placements. What is interesting is that the relative proportions also seem to vary by risk, with a higher ratio of substantiations to referrals and removals to substantiations in high risk groups. For example?

Low risk has a re-referral rate of 28%, re-substantiation=9% and removal=.6%. So, for every 100 low risk families someone reports, about 32 will be substantiated and less than 1 will be removed.

Very high risk has a re-referral rate of 74%, re-substantiation=46% and removal =22%. So, for every 100 very high risk families someone reports, 62 will be substantiated and almost 30 will be removed.

This suggests that higher risk families not only have increased probability of ANY future maltreatment, but also that it is more likely to be severe.

The 22% removal rate among very high risk families might strike us as low unless we think about risk and safety as distinct notions. The risk tool is just one component of decision making, and it?s our position that a child should never be removed based on risk. When there is high risk, there is an opportunity to provide services that will reduce risk. Safety assessment should be the basis for removal decisions (and even there, upon identifying a threat to safety, the first consideration is whether an in-home safety plan can be developed with the family to mitigate the danger). There is a correlation between safety and risk, as we would expect. But risk alone can be addressed with services.

Raelene Freitag, MSW, Ph.D.

Director

Children's Research Center

426 S. Yellowstone Dr. #250

Madison, WI 53719

608-831-1180

www.nccd-crc.org

Please do not send any identifying or confidential information (such as names, birthdates, social security numbers) via e-mail. It is possible for third parties to intercept information transmitted in an e-mail. Identification numbers (such as case or referral numbers) may be included where necessary. Intercepting persons cannot use these numbers to identify a client unless they have access to the host application or database.

From: bounce-2250391-6833790@xxxxxxxxxxxxxxxx [mailto:bounce-2250391-6833790@xxxxxxxxxxxxxxxx] On Behalf Of M. W. Sullivan
Sent: Wednesday, January 16, 2008 7:34 AM
To: Child Maltreatment Researchers
Subject: Re: dumb question?

Unless I'm mis-reading. This graph seems to show that increasing risk on the assessment is associated with incrementally more referrals and nearly doubles the rate of substantiation from moderate risk. Unclear whether highest risk results in more home removal---were these differences significant?

It seems that this data is useful to show that reffals and substantiation rates are valid--ie reflect some "real-world" status. Home removal is more equivocal---why isn't the rate much higher?

Margaret W. Sullivan, Ph. D.
Professor & Associate Director
Institute for the Study of Child Development/Pediatrics
UMDNJ-Robert Wood Johnson Medical School
97 Paterson Street
New Brunswick, New Jersey 08903-0019
"The approach to the problem is more important than the solution."
Telecommunications:
Phone: 732-235-7164 FAC:732-235-6189
Link: www2. umdnj.edu/iscdweb/index.html

----- Original Message -----

From: Raelene Freitag

To: Child Maltreatment Researchers

Sent: Tuesday, January 15, 2008 12:00 PM

Subject: RE: dumb question?

Sheri,

It?s a good question. The term ?risk? is often used quite loosely, and is used rather impressionistically. Sometimes it has likert-type scale definitions or anchor points so there is a relative increase or decrease. Alternatively, there are actuarial risk instruments that have very specific outcome data for families in various risk levels. For example, a recent study of over 5500 California family risk assessments conducted by workers in the field was used to construct a risk assessment instrument that creates 4 levels of risk. The known rates of each outcome measured are presented below:

Raelene Freitag, MSW, Ph.D.

Director

Children's Research Center

So, when a worker in California describes a family as high risk, they are saying that the family shares characteristics with families that are very likely to be re-referred in the next two years (74.8%--and often this will occur multiple times), have about a 50-50 chance of having at least one new substantiation, and more than 1 in 5 (22.1%) will end up in foster care. In contrast, low risk families are unlikely to be re-referred, more than 90% will not be re-substantiated, and only about half a percent will end up in foster care. It?s really important to note that risk assessment is good for classification, but not prediction. It should not be used as a basis for decisions about whether to remove a child (safety assessment should be used for that). But a reliable and valid risk tool can be very effective at identifying families who would benefit from services to prevent maltreatment. It?s also important to note that we see virtually the same recurrence ?by-risk distribution whether or not the CURRENT referral is substantiated. It?s helpful to think about serving higher risk families regardless of whether the current incident is substantiated or not.

You are completely right about the importance of balancing statistical information with clinical judgment. It?s my argument that we can?t ignore the evidence that for some things like estimating probability of future events simple statistical models typically outperform human judgment. We also can?t ignore that even the most robust risk tool can?t predict the future, or take every variable into account. I encourage our profession to move past ?either/or? thinking about this and take a ?both/and? approach. It?s important to advance our skill at integrating data-driven information such as risk assessments into the human relationships between worker and family.

If you want to learn more about actuarial risk, you can get several publications, including some of the risk studies, from our web-site below. Follow the link to CRC, then publications, and look for risk-related publications there. Best wishes.

Raelene Freitag, MSW, Ph.D.

426 S. Yellowstone Dr. #250

Madison, WI 53719

608-831-1180

www.nccd-crc.org

Please do not send any identifying or confidential information (such as names, birthdates, social security numbers) via e-mail. It is possible for third parties to intercept information transmitted in an e-mail. Identification numbers (such as case or referral numbers) may be included where necessary. Intercepting persons cannot use these numbers to identify a client unless they have access to the host application or database.

From: bounce-2239133-6833790@xxxxxxxxxxxxxxxx [mailto:bounce-2239133-6833790@xxxxxxxxxxxxxxxx] On Behalf Of D F MCMAHON
Sent: Monday, January 14, 2008 5:10 PM
To: Child Maltreatment Researchers
Subject: dumb question?

I don't know if it is or it isn't. Child welfare--in the legal system and in the child welfare system per se--is rife with terms that indicate scale--degrees of risk (low, medium, high for example); imminent risk of harm; etc etc.

To what extent does either the child welfare system or the legal system quantify these concepts? For example, if a risk assessment indicates high risk for abuse/neglect (with abuse/neglect definitions including additional quasi-quantitative concepts pertaining to likelihood of serious physical or emotional harm as a consequence)--does this basically mean a reasonable person would have the willies (scale) or that there is actually a range of probability that can be applied to the risk (e.g. 1% risk of serious physical or emotional harm)?

I'm asking not to challenge, but to try to better understand the framework here--yet it is also the case that 1) human beings are not very good at assessing degree of risk--there may be consistency, but not necessarily accuracy--e.g. if you ask people to state the likelihood that X will occur given Y there are many examples in which groups of people pretty consistently say the odds are Z even though the odds are actually far greater or less--and 2) perceived risk is affected by sociological factors (e.g. different attitudes about the risk of germs depending on whether you are talking to a family running a small dairy farm where the kids are in 4-H and raise rabbits and chickens, vs. an affluent couple in the suburbs who have one child).

It strikes me that child welfare in our society entails a pretty significant tension between a statistical, sort of public-health aspect and an individual aspect ("if just one life is saved" then all else is justified). Since I am not a professional or a theoretician in this area are there things one can read regarding this or are there observations anyone might have? Say, someone who is knowledgeable (genuinely knowledgeable) about child welfare but also able to identify underlying philosophical notions and, perhaps, conflicts?

(If you understand the question but can state it more clearly, please feel free to do so)

Sheri McMahon