Automatic classification of communication logs into implementation stages via text analysis

Dingding Wang, Mitsunori Ogihara, Carlos Gallo, Juan A. Villamar, Justin D. Smith, Wouter Vermeer, Gracelyn Cruden, Nanette Benbow, C. Hendricks Brown*

*Corresponding author for this work

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Background: To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes. Results: We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage. Conclusions: This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

Original languageEnglish (US)
Article number119
Number of pages1
JournalImplementation Science
Volume11
Issue number1
DOIs
StatePublished - Sep 6 2016

Fingerprint

Health
Communication
Costs and Cost Analysis
Therapeutics
Machine Learning

Keywords

  • Machine learning
  • Social systems informatics
  • Text mining
  • Unobtrusive measures

ASJC Scopus subject areas

  • Health Policy
  • Health Informatics
  • Public Health, Environmental and Occupational Health

Cite this

@article{f0f8703b621e4d358b5146b3d1e38aed,
title = "Automatic classification of communication logs into implementation stages via text analysis",
abstract = "Background: To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes. Results: We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage. Conclusions: This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.",
keywords = "Machine learning, Social systems informatics, Text mining, Unobtrusive measures",
author = "Dingding Wang and Mitsunori Ogihara and Carlos Gallo and Villamar, {Juan A.} and Smith, {Justin D.} and Wouter Vermeer and Gracelyn Cruden and Nanette Benbow and Brown, {C. Hendricks}",
year = "2016",
month = "9",
day = "6",
doi = "10.1186/s13012-016-0483-6",
language = "English (US)",
volume = "11",
journal = "Implementation Science",
issn = "1748-5908",
publisher = "BioMed Central",
number = "1",

}

Automatic classification of communication logs into implementation stages via text analysis. / Wang, Dingding; Ogihara, Mitsunori; Gallo, Carlos; Villamar, Juan A.; Smith, Justin D.; Vermeer, Wouter; Cruden, Gracelyn; Benbow, Nanette; Brown, C. Hendricks.

In: Implementation Science, Vol. 11, No. 1, 119, 06.09.2016.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Automatic classification of communication logs into implementation stages via text analysis

AU - Wang, Dingding

AU - Ogihara, Mitsunori

AU - Gallo, Carlos

AU - Villamar, Juan A.

AU - Smith, Justin D.

AU - Vermeer, Wouter

AU - Cruden, Gracelyn

AU - Benbow, Nanette

AU - Brown, C. Hendricks

PY - 2016/9/6

Y1 - 2016/9/6

N2 - Background: To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes. Results: We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage. Conclusions: This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

AB - Background: To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes. Results: We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage. Conclusions: This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

KW - Machine learning

KW - Social systems informatics

KW - Text mining

KW - Unobtrusive measures

UR - http://www.scopus.com/inward/record.url?scp=84984979183&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84984979183&partnerID=8YFLogxK

U2 - 10.1186/s13012-016-0483-6

DO - 10.1186/s13012-016-0483-6

M3 - Article

C2 - 27600612

AN - SCOPUS:84984979183

VL - 11

JO - Implementation Science

JF - Implementation Science

SN - 1748-5908

IS - 1

M1 - 119

ER -