Soundprism: An online system for score-informed source separation of music audio

Zhiyao Duan*, Bryan A Pardo

*Corresponding author for this work

Research output: Contribution to journalArticle

60 Scopus citations

Abstract

Soundprism, as proposed in this paper, is a computer system that separates single-channel polyphonic music audio played by harmonic sources into source signals in an online fashion. It uses a musical score to guide the separation process. To the best of our knowledge, this is the first online system that addresses score-informed music source separation that can be made into a real-time system. The proposed system consists of two parts: 1) a score follower that associates a score position to each time frame of the audio performance; 2) a source separator which reconstructs the source signals for each time frame, informed by the score. The score follower uses a hidden Markov approach, where each audio frame is associated with a 2-D state vector (score position and tempo). The observation model is defined as the likelihood of observing the frame given the pitches at the score position. The score position and tempo are inferred using particle filtering. In building the source separator, we first refine the score-informed pitches of the current audio frame by maximizing the multi-pitch observation likelihood. Then, the harmonics of each source's fundamental frequency are extracted to reconstruct the source signal. Overlapping harmonics between sources are identified and their energy is distributed in inverse proportion to the square of their respective harmonic number. Experiments on both synthetic and human-performed music show both the score follower and the source separator perform well. Results also show that the proposed score follower works well for highly polyphonic music with some degree of tempo variations.

Original languageEnglish (US)
Article number5887382
Pages (from-to)1205-1215
Number of pages11
JournalIEEE Journal on Selected Topics in Signal Processing
Volume5
Issue number6
DOIs
StatePublished - Oct 1 2011

Keywords

  • Multi-pitch estimation
  • Online algorithm
  • Score following
  • Source separation

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Soundprism: An online system for score-informed source separation of music audio'. Together they form a unique fingerprint.

  • Cite this