Human detection of political speech deepfakes across transcripts, audio, and video

Matthew Groh; Aruna Sankaranarayanan; Nikhil Singh; Dong Young Kim; Andrew Lippman; Rosalind Picard

doi:10.1038/s41467-024-51998-z

Human detection of political speech deepfakes across transcripts, audio, and video

Matthew Groh^*, Aruna Sankaranarayanan, Nikhil Singh, Dong Young Kim, Andrew Lippman, Rosalind Picard

^*Corresponding author for this work

Management and Organizations

Research output: Contribution to journal › Article › peer-review

6 Scopus citations

Abstract

Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.

Original language	English (US)
Article number	7629
Journal	Nature communications
Volume	15
Issue number	1
DOIs	https://doi.org/10.1038/s41467-024-51998-z
State	Published - Dec 2024

Funding

The authors would like to acknowledge support for signing videos from Truepic and $7000 in funding for participant recruitment from Truepic for Experiments 2 through 5, funding from MIT Media Lab member companies, and Kellogg School of Management, thanks Colin Cassidy, J-L Cauvin, Austin Nasso for providing voice impressions for stimuli, thank the following users who contributed sounds for stimuli from Freesound.org including aaronstar, aleclubin, cmilan, funwithsound, jgarc, johnsonbrandediting, klankbeeld, macohibs, mzui, noisecollector, peridactyloptrix, speedygonzo, zabuhailo, thank David Rand, Gordon Pennycook, Rahul Bhui, Yunhao (Jerry) Zhang, Ziv Epstein, and members of the Affective Computing lab at the MIT Media Lab and the Human Cooperation lab at MIT Sloan School of Management for helpful feedback on early versions of this manuscript, Anna Murphy, Shreya Kalyan, Theo Chen, and Alicia Guo for research assistance, and Craig Ferguson for feedback on hosting the experiment.

ASJC Scopus subject areas

General Chemistry
General Biochemistry, Genetics and Molecular Biology
General Physics and Astronomy

Access to Document

10.1038/s41467-024-51998-z

Cite this

@article{6b66c93bd3c54d1eaac955d2f6d181d1,

title = "Human detection of political speech deepfakes across transcripts, audio, and video",

abstract = "Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.",

author = "Matthew Groh and Aruna Sankaranarayanan and Nikhil Singh and Kim, {Dong Young} and Andrew Lippman and Rosalind Picard",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2024.",

year = "2024",

month = dec,

doi = "10.1038/s41467-024-51998-z",

language = "English (US)",

volume = "15",

journal = "Nature communications",

issn = "2041-1723",

publisher = "Nature Research",

number = "1",

}

TY - JOUR

T1 - Human detection of political speech deepfakes across transcripts, audio, and video

AU - Groh, Matthew

AU - Sankaranarayanan, Aruna

AU - Singh, Nikhil

AU - Kim, Dong Young

AU - Lippman, Andrew

AU - Picard, Rosalind

PY - 2024/12

Y1 - 2024/12

N2 - Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.

AB - Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.

UR - http://www.scopus.com/inward/record.url?scp=85202948945&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85202948945&partnerID=8YFLogxK

U2 - 10.1038/s41467-024-51998-z

DO - 10.1038/s41467-024-51998-z

M3 - Article

C2 - 39223110

AN - SCOPUS:85202948945

SN - 2041-1723

VL - 15

JO - Nature communications

JF - Nature communications

IS - 1

M1 - 7629

ER -

Human detection of political speech deepfakes across transcripts, audio, and video

Abstract

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this