Multi-pitch streaming of harmonic sound mixtures

Zhiyao Duan, Jinyu Han, Bryan A Pardo

Research output: Contribution to journalArticlepeer-review

39 Scopus citations


Multi-pitch analysis of concurrent sound sources is an important but challenging problem. It requires estimating pitch values of all harmonic sources in individual frames and streaming the pitch estimates into trajectories, each of which corresponds to a source. We address the streaming problem for monophonic sound sources. We take the original audio, plus frame-level pitch estimates from any multi-pitch estimation algorithm as inputs, and output a pitch trajectory for each source. Our approach does not require pre-training of source models from isolated recordings. Instead, it casts the problem as a constrained clustering problem, where each cluster corresponds to a source. The clustering objective is to minimize the timbre inconsistency within each cluster. We explore different timbre features for music and speech. For music, harmonic structure and a newly proposed feature called uniform discrete cepstrum (UDC) are found effective; while for speech, MFCC and UDC works well. We also show that timbre-consistency is insufficient for effective streaming. Constraints are imposed on pairs of pitch estimates according to their time-frequency relationships.We propose a new constrained clustering algorithm that satisfies as many constraints as possible while optimizing the clustering objective. We compare the proposed approach with other state-of-the-art supervised and unsupervised multi-pitch streaming approaches that are specifically designed for music or speech. Better or comparable results are shown.

Original languageEnglish (US)
Pages (from-to)138-150
Number of pages13
JournalIEEE Transactions on Audio, Speech and Language Processing
Issue number1
StatePublished - Jan 2014


  • Cochannel speech
  • Constrained clustering
  • Multi-pitch analysis
  • Pitch streaming
  • Timbre tracking

ASJC Scopus subject areas

  • Acoustics and Ultrasonics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Multi-pitch streaming of harmonic sound mixtures'. Together they form a unique fingerprint.

Cite this