Skip to main content



Child-Maltreatment-Research-L (CMRL) List Serve

Browse or Search All Past CMRL Messages

Welcome to the database of past Child-Maltreatment-Research-L (CMRL) list serve messages (10,000+). The table below contains all past CMRL messages (text only, no attachments) from Nov. 20, 1996 - June 11, 2018 and is updated quarterly.

Instructions: Postings are listed for browsing with the newest messages first. Click on the linked ID number to see a message. You can search the author, subject, message ID, and message content fields by entering your criteria into this search box:

Message ID: 8476
Date: 2010-05-11

Author:Julia Littell

Subject:Re: Regarding "Assessing Parent Education Programs for Families Involved with Child Welfare Services: Evidence and Implications"

Hi Michelle, With all due respect, there is growing evidence that nonsystematic (haphazad) reviews lead to the wrong conclusions. Your review ("Assessing Parent Education Programs for Families involved with Child Welfare Services: Evidence and Implications") is a case in point, but my earlier comments were not aimed at your review alone, Below, I discuss issues related to research reviews that include different types of evidence, but the bottom line is that some statements in your review are misleading, as is often the case with nonsystematic reviews of research. The example of a misleading statement that I gave earlier was in reference to the study that you cited by Brunk et al. (1987). According to your review, outcomes of MST and PT in this study were: "improvements in the restructuring of parent-child relations; increased the responsiveness of neglectful parents; reduced overall stress; decreased parental psychiatric symptomology; reduction in severity of identified problems; decreased maltreated children's passive compliance" (Figure 5, p. 23). This statement is misleading for several reasons: 1. These treatments were not compared with a placebo or no-treatment control group, so we don't whether changes over time were due to treatment(s) or to other factors (threats to internal validity, including maturation, history, testing, selection, etc.). 2. These treatments were compared with each other and they did not have identical outcomes (a fact that was mentioned in the text of your review, on p. 15, but not in Figure 5). 3. Outcome data were only available immediately after treatment ended and only for 33 families (data were not available for families who dropped out of treatment), so we don't know whether gains were sustained or whether results are generalizable to other families. I picked this example because there is a published analysis of the Brunk study--and how it has been misinterpreted in nonsystematic reviews (Littell, 2008). Other examples could be cited as well. No matter how careful a review is, it is not useful for practice or policy purposes if the summary or conclusions are misleading! The term "haphazard review" was coined by Petticrew and Roberts (2006) to draw attention to the problems of non-systematic reviews in the social sciences. Calling a review "haphazard" does not mean that the reviewers were less than careful. It does mean that they didn't follow current evidence-based standards for the conduct of research reviews (i.e., the review was not systematic, as that term is used today; see Littell, Corcoran, & Pillai, 2008). Current standards reduce the likelihood that a review will be marred by bias and error. Current standards can be used in reviews that include heterogeneous study designs, including observational studies and other nonrandomized studies. Although the Cochrane Collaboration didn't officially include guidelines for incorporating quasi-experiments and other non-randomized designs in systematic reviews until 2008, many Cochrane groups (e.g., the Effective Practice and Organization of Care or EPOC group) were doing this much earlier. More importantly, the Campbell Collaboration guidelines (which are arguably more relevant for your review than Cochrane guidelines) incorporated non-randomized designs in 2004 (Shadish & Myers, 2004). Cochrane and Campbell reviews are the most reliable sources of evidence, but some reviewers do not have the time or resources to do a full Cochrane or Campbell review. Even so, it makes sense for reviewers to follow current, evidence-based standards for research reviews (PRISMA and the Cochrane Handbook) to the extent possible. Some adherence to these standards is better than none. Strategies that minimize bias and error in research reviews include: * using clear inclusion and exclusion criteria that are established in advance (using the Cochrane PICOS framework), * conducting thorough searches of gray literature to minimize the impact of publication bias, * fully documenting search strategies, * documenting study inclusion and exclusion decisions and inter-rater agreement on these decisions, * utilizing all available reports on included studies (to track attrition and other implementation problems), * reliably extracting data from studies onto standardized forms, * formally assessing the risk of bias in included studies, * presenting results in terms of effect sizes (using meta-analysis when possible), and * checking the accuracy of summary statements. Policy makers and practitioners want reliable syntheses of evidence. It is up to reviewers to make sure that research reviews are done well (i.e., according to current, scientific standards). Best wishes, Julia On Fri, Apr 30, 2010 at 2:18 PM, Michelle Johnson > wrote: In the current era of "evidence," Cochrane procedures are the most rigorous guidelines available to us. This is not a question. However, in literatures reflecting a wide range of methodologies, other types of reviews are not without value or purpose. During the planning stages of this project we considered the development of a systematic review using Cochrane procedures. Based on a preliminary search and our collective understanding of the knowledge base in this area, we determined that this nascent body of literature, heterogeneous in study design and comprised predominately of non-randomized studies, was not conducive to the application of procedures for the review of randomized control trials. While the current version of the Cochrane Handbook for Systematic Reviews of Interventions (Version 5) has evolved to include chapters on evidence other than randomized control trials, most of the text is oriented to clinical trials, particularly randomized trials. As the state of the literature on our topic came into focus, we were presented with difficult questions. These questions centered on what we could contribute, particularly in light of the information needs expressed by the social service agencies that had contracted with us for the review. Rather than restrict our review to a few RCTs, we opted for a more comprehensive effort to assess the general state of the literature on parenting program development and evaluation in the context of maltreating populations, and to examine the extent to which program elements intersect with leading etiological theories of child maltreatment. The review reveals the development of knowledge in this area, points to future directions for research, and provides practitioners with some considerations for contracting for parenting programs, which they are typically required to do in an era of limited information on the topic. To characterize something as haphazard suggests that it is marked by a lack of planning, order, or direction. This review was not a random exercise. -- Michelle A. Johnson, Ph.D., MSW Assistant Professor UCLA School of Public Affairs, Department of Social Welfare 3250 Public Affairs Building, Box 951656 Los Angeles, CA 90095-1656 Office: (310) 825-9661 Fax: (310) 206-7564 E-mail: maj@ucla.edu Child Maltreatment Researchers digest wrote: CHILD-MALTREATMENT-RESEARCH-L Digest for Wednesday, April 14, 2010. 1. Re: child-maltreatment-research-l digest: April 12, 2010 ---------------------------------------------------------------------- Subject: Re: child-maltreatment-research-l digest: April 12, 2010 From: Julia Littell > Date: Tue, 13 Apr 2010 08:55:30 -0400 X-Message-Number: 1 It is important to note that, like most reviews upon which "evidence based practices" (Triple P, PCIT, MST, FFT) are based, this is a *haphazard* review. Although it is extensive and thoughtful, it does not adhere to current scientific standards for reviewing research (Cochrane Handbook, PRISMA). How can we expect practitioners to use evidence about practice if reviewers don't use evidence about reviewing? This is not an academic concern; summaries presented in Figure 5 are misleading (e.g., compare summary of MST/PT on p. 23 to a detailed analysis of the study cited here in Littell, J. H. (2008). Evidence-based or biased? The quality of published reviews of evidence-based practices. *Children and Youth Services Review, 30, *1299-1317). On Tue, Apr 13, 2010 at 6:55 AM, Michelle Johnson > wrote: The Bay Area Social Services Consortium at UC Berkeley conducted an extensive literature review on the outcomes of 58 parent education programs with families determined to be at-risk of child maltreatment and/or abusive or neglectful. The report, "Assessing Parent Education Programs for Families Involved with Child Welfare Services: Evidence and Implications," is located at http://www.bassc.net/html/research/projects_details.html More studies have come out since the 2006 publication. This report may be a helpful starting point in your search. -- Michelle A. Johnson, Ph.D., MSW Assistant Professor UCLA School of Public Affairs, Department of Social Welfare 3250 Public Affairs Building, Box 951656 Los Angeles, CA 90095-1656 Office: (310) 825-9661 Fax: (310) 206-7564 E-mail: maj@ucla.edu -- Julia H. Littell, Ph.D., Professor Graduate School of Social Work and Social Welfare Bryn Mawr College 300 Airdale Rd. Bryn Mawr, PA 19010, USA

Hi Michelle, With all due respect, there is growing evidence that nonsystematic (haphazad) reviews lead to the wrong conclusions. Your review ("Assessing Parent Education Programs for Families involved with Child Welfare Services: Evidence and Implications") is a case in point, but my earlier comments were not aimed at your review alone, Below, I discuss issues related to research reviews that include different types of evidence, but the bottom line is that some statements in your review are misleading, as is often the case with nonsystematic reviews of research. The example of a misleading statement that I gave earlier was in reference to the study that you cited by Brunk et al. (1987). According to your review, outcomes of MST and PT in this study were: "improvements in the restructuring of parent-child relations; increased the responsiveness of neglectful parents; reduced overall stress; decreased parental psychiatric symptomology; reduction in severity of identified problems; decreased maltreated children's passive compliance" (Figure 5, p. 23). This statement is misleading for several reasons: 1. These treatments were not compared with a placebo or no-treatment control group, so we don't whether changes over time were due to treatment(s) or to other factors (threats to internal validity, including maturation, history, testing, selection, etc.). 2. These treatments were compared with each other and they did not have identical outcomes (a fact that was mentioned in the text of your review, on p. 15, but not in Figure 5). 3. Outcome data were only available immediately after treatment ended and only for 33 families (data were not available for families who dropped out of treatment), so we don't know whether gains were sustained or whether results are generalizable to other families. I picked this example because there is a published analysis of the Brunk study--and how it has been misinterpreted in nonsystematic reviews (Littell, 2008). Other examples could be cited as well. No matter how careful a review is, it is not useful for practice or policy purposes if the summary or conclusions are misleading! The term "haphazard review" was coined by Petticrew and Roberts (2006) to draw attention to the problems of non-systematic reviews in the social sciences. Calling a review "haphazard" does not mean that the reviewers were less than careful. It does mean that they didn't follow current evidence-based standards for the conduct of research reviews (i.e., the review was not systematic, as that term is used today; see Littell, Corcoran, & Pillai, 2008). Current standards reduce the likelihood that a review will be marred by bias and error. Current standards can be used in reviews that include heterogeneous study designs, including observational studies and other nonrandomized studies. Although the Cochrane Collaboration didn't officially include guidelines for incorporating quasi-experiments and other non-randomized designs in systematic reviews until 2008, many Cochrane groups (e.g., the Effective Practice and Organization of Care or EPOC group) were doing this much earlier. More importantly, the Campbell Collaboration guidelines (which are arguably more relevant for your review than Cochrane guidelines) incorporated non-randomized designs in 2004 (Shadish & Myers, 2004). Cochrane and Campbell reviews are the most reliable sources of evidence, but some reviewers do not have the time or resources to do a full Cochrane or Campbell review. Even so, it makes sense for reviewers to follow current, evidence-based standards for research reviews (PRISMA and the Cochrane Handbook) to the extent possible. Some adherence to these standards is better than none. Strategies that minimize bias and error in research reviews include: * using clear inclusion and exclusion criteria that are established in advance (using the Cochrane PICOS framework), * conducting thorough searches of gray literature to minimize the impact of publication bias, * fully documenting search strategies, * documenting study inclusion and exclusion decisions and inter-rater agreement on these decisions, * utilizing all available reports on included studies (to track attrition and other implementation problems), * reliably extracting data from studies onto standardized forms, * formally assessing the risk of bias in included studies, * presenting results in terms of effect sizes (using meta-analysis when possible), and * checking the accuracy of summary statements. Policy makers and practitioners want reliable syntheses of evidence. It is up to reviewers to make sure that research reviews are done well (i.e., according to current, scientific standards). Best wishes, Julia On Fri, Apr 30, 2010 at 2:18 PM, Michelle Johnson > wrote: In the current era of "evidence," Cochrane procedures are the most rigorous guidelines available to us. This is not a question. However, in literatures reflecting a wide range of methodologies, other types of reviews are not without value or purpose. During the planning stages of this project we considered the development of a systematic review using Cochrane procedures. Based on a preliminary search and our collective understanding of the knowledge base in this area, we determined that this nascent body of literature, heterogeneous in study design and comprised predominately of non-randomized studies, was not conducive to the application of procedures for the review of randomized control trials. While the current version of the Cochrane Handbook for Systematic Reviews of Interventions (Version 5) has evolved to include chapters on evidence other than randomized control trials, most of the text is oriented to clinical trials, particularly randomized trials. As the state of the literature on our topic came into focus, we were presented with difficult questions. These questions centered on what we could contribute, particularly in light of the information needs expressed by the social service agencies that had contracted with us for the review. Rather than restrict our review to a few RCTs, we opted for a more comprehensive effort to assess the general state of the literature on parenting program development and evaluation in the context of maltreating populations, and to examine the extent to which program elements intersect with leading etiological theories of child maltreatment. The review reveals the development of knowledge in this area, points to future directions for research, and provides practitioners with some considerations for contracting for parenting programs, which they are typically required to do in an era of limited information on the topic. To characterize something as haphazard suggests that it is marked by a lack of planning, order, or direction. This review was not a random exercise. -- Michelle A. Johnson, Ph.D., MSW Assistant Professor UCLA School of Public Affairs, Department of Social Welfare 3250 Public Affairs Building, Box 951656 Los Angeles, CA 90095-1656 Office: (310) 825-9661 Fax: (310) 206-7564 E-mail: majucla.edu Child Maltreatment Researchers digest wrote: CHILD-MALTREATMENT-RESEARCH-L Digest for Wednesday, April 14, 2010. 1. Re: child-maltreatment-research-l digest: April 12, 2010 ---------------------------------------------------------------------- Subject: Re: child-maltreatment-research-l digest: April 12, 2010 From: Julia Littell > Date: Tue, 13 Apr 2010 08:55:30 -0400 X-Message-Number: 1 It is important to note that, like most reviews upon which "evidence based practices" (Triple P, PCIT, MST, FFT) are based, this is a *haphazard* review. Although it is extensive and thoughtful, it does not adhere to current scientific standards for reviewing research (Cochrane Handbook, PRISMA). How can we expect practitioners to use evidence about practice if reviewers don't use evidence about reviewing? This is not an academic concern; summaries presented in Figure 5 are misleading (e.g., compare summary of MST/PT on p. 23 to a detailed analysis of the study cited here in Littell, J. H. (2008). Evidence-based or biased? The quality of published reviews of evidence-based practices. *Children and Youth Services Review, 30, *1299-1317). On Tue, Apr 13, 2010 at 6:55 AM, Michelle Johnson > wrote: The Bay Area Social Services Consortium at UC Berkeley conducted an extensive literature review on the outcomes of 58 parent education programs with families determined to be at-risk of child maltreatment and/or abusive or neglectful. The report, "Assessing Parent Education Programs for Families Involved with Child Welfare Services: Evidence and Implications," is located at http://www.bassc.net/html/research/projects_details.html More studies have come out since the 2006 publication. This report may be a helpful starting point in your search. -- Michelle A. Johnson, Ph.D., MSW Assistant Professor UCLA School of Public Affairs, Department of Social Welfare 3250 Public Affairs Building, Box 951656 Los Angeles, CA 90095-1656 Office: (310) 825-9661 Fax: (310) 206-7564 E-mail: majucla.edu -- Julia H. Littell, Ph.D., Professor Graduate School of Social Work and Social Welfare Bryn Mawr College 300 Airdale Rd. Bryn Mawr, PA 19010, USA