NSF-BSF-NCS-FO: Enhancing speech and deep learning research through holistic acoustic analysis

NCS Focus Areas: Data-Intensive Neuroscience and Cognitive Science; Individuality and Variation. Speech variability across talkers provides a treasure trove of information for cognitive neuroscientists, leading to important insights into the cognitive mechanisms underlying language processing and providing early signs of brain dysfunction. Current studies of speech are hamstrung by analyses that require preselecting specific temporal scales and acoustic dimensions. We propose a radically different approach: using unsupervised deep learning to discover a representational space for analysis of acoustic variation. To test this highly general approach, we will assess whether it out-performs current methods for analyzing individual variation in bilingual speech; the results will inform development of deep learning methods and cognitive neuroscience theory. Integrative value and transformative potential. While our approach is high-risk – an entirely novel analysis of acoustics – it may transform cognitive neuroscience and computer science research. As it is unsupervised, it can be applied to speech from any language or any domain of language usage, learning a representational space that integrates information across multiple temporal scales and units of analysis. Using leading-edge research and new acoustic data, our multi-national, interdisciplinary team will collaboratively analyze the structure of the emergent representational space. This integrative approach will allow computer scientists to better understand modern deep learning architectures for speech, and allow cognitive neuroscientists to better understand the representation of speech.