Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study

Justin D. Smith*, Jenna Rudo-Stern, Thomas J. Dishion, Elizabeth A. Stormshak, Samantha Montag, Kimbree Brown, Karina Ramos, Daniel S. Shaw, Melvin N. Wilson

*Corresponding author for this work

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions—complete and 20-min segments—for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

Original languageEnglish (US)
Pages (from-to)16-28
Number of pages13
JournalJournal of Clinical Child and Adolescent Psychology
Volume48
Issue number1
DOIs
StatePublished - Jan 2 2019

Fingerprint

Non-Randomized Controlled Trials
Direction compound

ASJC Scopus subject areas

  • Developmental and Educational Psychology
  • Clinical Psychology

Cite this

Smith, Justin D. ; Rudo-Stern, Jenna ; Dishion, Thomas J. ; Stormshak, Elizabeth A. ; Montag, Samantha ; Brown, Kimbree ; Ramos, Karina ; Shaw, Daniel S. ; Wilson, Melvin N. / Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention : A Quasi-Experimental Study. In: Journal of Clinical Child and Adolescent Psychology. 2019 ; Vol. 48, No. 1. pp. 16-28.
@article{f98112c5257f4064a8d9b462916c590f,
title = "Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study",
abstract = "Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions—complete and 20-min segments—for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ 2  = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.",
author = "Smith, {Justin D.} and Jenna Rudo-Stern and Dishion, {Thomas J.} and Stormshak, {Elizabeth A.} and Samantha Montag and Kimbree Brown and Karina Ramos and Shaw, {Daniel S.} and Wilson, {Melvin N.}",
year = "2019",
month = "1",
day = "2",
doi = "10.1080/15374416.2018.1561295",
language = "English (US)",
volume = "48",
pages = "16--28",
journal = "Journal of Clinical Child and Adolescent Psychology",
issn = "1537-4416",
publisher = "Routledge",
number = "1",

}

Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention : A Quasi-Experimental Study. / Smith, Justin D.; Rudo-Stern, Jenna; Dishion, Thomas J.; Stormshak, Elizabeth A.; Montag, Samantha; Brown, Kimbree; Ramos, Karina; Shaw, Daniel S.; Wilson, Melvin N.

In: Journal of Clinical Child and Adolescent Psychology, Vol. 48, No. 1, 02.01.2019, p. 16-28.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention

T2 - A Quasi-Experimental Study

AU - Smith, Justin D.

AU - Rudo-Stern, Jenna

AU - Dishion, Thomas J.

AU - Stormshak, Elizabeth A.

AU - Montag, Samantha

AU - Brown, Kimbree

AU - Ramos, Karina

AU - Shaw, Daniel S.

AU - Wilson, Melvin N.

PY - 2019/1/2

Y1 - 2019/1/2

N2 - Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions—complete and 20-min segments—for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ 2  = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

AB - Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions—complete and 20-min segments—for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ 2  = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

UR - http://www.scopus.com/inward/record.url?scp=85060909085&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060909085&partnerID=8YFLogxK

U2 - 10.1080/15374416.2018.1561295

DO - 10.1080/15374416.2018.1561295

M3 - Article

C2 - 30702355

AN - SCOPUS:85060909085

VL - 48

SP - 16

EP - 28

JO - Journal of Clinical Child and Adolescent Psychology

JF - Journal of Clinical Child and Adolescent Psychology

SN - 1537-4416

IS - 1

ER -