Diversity Supplement: Implementation of Evidence-Based Preventative Parenting Programs

The parent grant attempts to identify methods that would allow community agencies to monitor implementation and maintain effect sizes when programs are delivered at scale. It examines the impact of rater (independent observer, supervisor, or provider) and sampling (amount, activity type). However, monitoring of implementation using coding of session videos is very costly and takes a great deal of time, making it an inefficient approach for use by community agencies that are interested in providing feedback to providers in a timely manner. The focus of this proposed supplement is to refine and apply new technology that would automate implementation assessment and reduce the reliance on human coding. These innovative methods will reduce the cost of conducting implementation assessments, reduce the training required for assessing implementation, improve the reliability of assessments, and enable a rapid feedback system to support consistent, high implementation. The proposed diversity supplement will use existing available data, including videos of the delivery of the NBP (collected as part of the multicourt effectiveness trial (R01DA026874)), transcripts of these sessions, and independent observer assessments of implementation (each collected as part of the parent grant studying the implementation of NBP R01DA033991), to develop automated measures of fidelity, quality, and responsiveness. These transcripts from the parent grant contain detailed information of linguistic expressions linked to fidelity, quality, and responsiveness all done by human-based assessments. We have a unique opportunity to complement these transcripts with machine-based assessments that exploit linguistic and non-linguistic cues derived from computational methods (a combination of machine learning and computational linguistics). We propose in this diversity supplement to develop, train, and test innovative computational methods that can measure fidelity, quality, and responsiveness in ways comparable and potentially superior to human-based assessments. The capacity to efficiently monitor the delivery of NBP in real world settings will provide critical methodology that can improve quality and outcomes of the NBP intervention particularly when it is delivered at scale, and when delivering the program as a service rather than an effectiveness trial. These innovative methods could have wide applicability in efficacy and effectiveness trials and in studies of sustainability as well as in developing efficient implementation strategies for complex interventions other than NBP. Efficient fidelity and quality assessments provide a way to inexpensively monitor implementation in a timely fashion so that feedback can be provided to supervisors and the facilitators who are delivering the program in real world settings. In the past few decades, there has been great progress in natural language processing and computational linguistics in speech recognition (Gemmeke et al, 2011), dialogue modeling (Stolcke et al, 2000), machine translation (Koehn et al, 2007), automatic essay scoring (Kakkonen et al, 2005), and sentiment analysis (Wilson et al, 2005). However, the use of computational methods in psychotherapy has been limited. In one case, natural language processing has been used to identify psychometric properties automatically (Pennebaker et al, 2007). Regarding automatic fidelity measurement, a limited number of researchers have focused their attention on developing computational methods to assess fidelity and quality for drug and alcohol prevention (Gallo et al, 2014) a