Soundprism: An online system for score-informed source separation of music audio

Zhiyao Duan*, Bryan A Pardo

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

85 Scopus citations

Abstract

Soundprism, as proposed in this paper, is a computer system that separates single-channel polyphonic music audio played by harmonic sources into source signals in an online fashion. It uses a musical score to guide the separation process. To the best of our knowledge, this is the first online system that addresses score-informed music source separation that can be made into a real-time system. The proposed system consists of two parts: 1) a score follower that associates a score position to each time frame of the audio performance; 2) a source separator which reconstructs the source signals for each time frame, informed by the score. The score follower uses a hidden Markov approach, where each audio frame is associated with a 2-D state vector (score position and tempo). The observation model is defined as the likelihood of observing the frame given the pitches at the score position. The score position and tempo are inferred using particle filtering. In building the source separator, we first refine the score-informed pitches of the current audio frame by maximizing the multi-pitch observation likelihood. Then, the harmonics of each source's fundamental frequency are extracted to reconstruct the source signal. Overlapping harmonics between sources are identified and their energy is distributed in inverse proportion to the square of their respective harmonic number. Experiments on both synthetic and human-performed music show both the score follower and the source separator perform well. Results also show that the proposed score follower works well for highly polyphonic music with some degree of tempo variations.

Original languageEnglish (US)
Article number5887382
Pages (from-to)1205-1215
Number of pages11
JournalIEEE Journal on Selected Topics in Signal Processing
Volume5
Issue number6
DOIs
StatePublished - Oct 2011

Funding

Manuscript received September 30, 2010; revised February 22, 2011 and May 20, 2011; accepted June 05, 2011. Date of publication June 16, 2011; date of current version September 16, 2011. This work was supported by the National Science Foundation under Awards 0643752 and 0812314. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Gaël Richard.

Keywords

  • Multi-pitch estimation
  • Online algorithm
  • Score following
  • Source separation

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Soundprism: An online system for score-informed source separation of music audio'. Together they form a unique fingerprint.

Cite this