A human-in-the-loop system for sound event detection and annotation

Bongjun Kim, Bryan A Pardo

Research output: Contribution to journalArticlepeer-review

33 Scopus citations


Labeling of audio events is essential for many tasks. However, finding sound events and labeling them within a long audio file is tedious and time-consuming. In cases where there is very little labeled data (e.g., a single labeled example), it is often not feasible to train an automatic labeler because many techniques (e.g., deep learning) require a large number of human-labeled training examples. Also, fully automated labeling may not show sufficient agreement with human labeling for many uses. To solve this issue, we present a human-in-the-loop sound labeling system that helps a user quickly label target sound events in a long audio. It lets a user reduce the time required to label a long audio file (e.g., 20 hours) containing target sounds that are sparsely distributed throughout the recording (10% or less of the audio contains the target) when there are too few labeled examples (e.g., one) to train a state-of-the-art machine audio labeling system. To evaluate the effectiveness of our tool, we performed a human-subject study. The results show that it helped participants label target sound events twice as fast as labeling them manually. In addition to measuring the overall performance of the proposed system, we also measure interaction overhead and machine accuracy, which are two key factors that determine the overall performance. The analysis shows that an ideal interface that does not have interaction overhead at all could speed labeling by as much as a factor of four.

Original languageEnglish (US)
Article number13
JournalACM Transactions on Interactive Intelligent Systems
Issue number2
StatePublished - Jul 2018


  • Human-in-the-loop system
  • Interactive machine learning
  • Sound event detection

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Artificial Intelligence


Dive into the research topics of 'A human-in-the-loop system for sound event detection and annotation'. Together they form a unique fingerprint.

Cite this