An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

Justin D. Smith*, Thomas J. Dishion, Kimbree Brown, Karina Ramos, Naomi B. Knoble, Daniel S. Shaw, Melvin N. Wilson

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

Original languageEnglish (US)
Pages (from-to)62-70
Number of pages9
JournalPrevention Science
Volume17
Issue number1
DOIs
StatePublished - Jan 1 2016

Funding

This work was supported by the National Institute on Drug Abuse (NIDA) through a pilot study grant awarded to Justin Smith by the Center for Prevention Implementation Methodology for Drug Abuse and Sex Risk Behavior (DA027828) and DA016110 to Thomas Dishion, Daniel Shaw, and Melvin Wilson. Justin Smith and Naomi Knoble received support from National Institute of Mental Health research training grant T32 MH20012, awarded to Elizabeth Stormshak. The authors wish to thank Cady Berkel, Amanda Chiapa, Guillermo Prado, and Juan Villamar for their helpful comments on earlier drafts of this manuscript; Hendricks Brown and Carlos Gallo for their assistance with the statistical analyses; the Early Steps team in Eugene, Pittsburgh, and Charlottesville; and the families who participated in the study.

Keywords

  • Competence
  • Family check-up
  • Fidelity
  • Implementation
  • Observational coding

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health

Fingerprint

Dive into the research topics of 'An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention'. Together they form a unique fingerprint.

Cite this