CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION

Erika Rumbold; George Tzanetakis; Bryan Pardo

CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION

Erika Rumbold, George Tzanetakis, Bryan Pardo

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The standard quality metric for audio source separation is the signal-to-distortion ratio (SDR), despite correlating poorly to human perception. In this paper, we investigate the correlation between subjective listener evaluations and multiple objective quality metrics, including the SDR. Music tracks are separated using five source separation models, and the resulting separation quality is calculated using five objective evaluation metrics. A listener study was conducted to also obtain subjective evaluations of the music separation quality. It was found that none of the observed objective metrics correlated well to human perception, with the best correlation coefficient being 0.246. We also found that the objective metrics did not agree with each other regarding which is the best and worst performing music source separation models.

Original language	English (US)
Title of host publication	Proceedings of the 21st Sound and Music Computing Conference, SMC 2024
Editors	Guilherme Campos, Guilherme Campos, Ines Salselas, Ines Salselas, Jose Vieira, Jose Vieira, Marco Conceicao, Marco Conceicao, Nuno Fonseca, Rui Penha, Rui Penha, Angela da Ponte, Angela da Ponte, Dimitrios Andrikopoulos, Dimitrios Andrikopoulos
Publisher	Sound and music Computing network
Pages	538-545
Number of pages	8
ISBN (Electronic)	9789893520758
State	Published - 2024
Event	21st Sound and Music Computing Conference, SMC 2024 - Porto, Portugal Duration: Jul 4 2024 → Jul 6 2024

Publication series

Name	Proceedings of the Sound and Music Computing Conferences
ISSN (Electronic)	2518-3672

Conference

Conference	21st Sound and Music Computing Conference, SMC 2024
Country/Territory	Portugal
City	Porto
Period	7/4/24 → 7/6/24

ASJC Scopus subject areas

Music
Computer Science Applications
Media Technology

Cite this

Rumbold, E., Tzanetakis, G., & Pardo, B. (2024). CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION. In G. Campos, G. Campos, I. Salselas, I. Salselas, J. Vieira, J. Vieira, M. Conceicao, M. Conceicao, N. Fonseca, R. Penha, R. Penha, A. D. Ponte, A. D. Ponte, D. Andrikopoulos, & D. Andrikopoulos (Eds.), Proceedings of the 21st Sound and Music Computing Conference, SMC 2024 (pp. 538-545). (Proceedings of the Sound and Music Computing Conferences). Sound and music Computing network.

Rumbold, Erika ; Tzanetakis, George ; Pardo, Bryan. / CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION. Proceedings of the 21st Sound and Music Computing Conference, SMC 2024. editor / Guilherme Campos ; Guilherme Campos ; Ines Salselas ; Ines Salselas ; Jose Vieira ; Jose Vieira ; Marco Conceicao ; Marco Conceicao ; Nuno Fonseca ; Rui Penha ; Rui Penha ; Angela da Ponte ; Angela da Ponte ; Dimitrios Andrikopoulos ; Dimitrios Andrikopoulos. Sound and music Computing network, 2024. pp. 538-545 (Proceedings of the Sound and Music Computing Conferences).

@inproceedings{11a0088a03444130a48a3c0d7b9748e5,

title = "CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION",

abstract = "The standard quality metric for audio source separation is the signal-to-distortion ratio (SDR), despite correlating poorly to human perception. In this paper, we investigate the correlation between subjective listener evaluations and multiple objective quality metrics, including the SDR. Music tracks are separated using five source separation models, and the resulting separation quality is calculated using five objective evaluation metrics. A listener study was conducted to also obtain subjective evaluations of the music separation quality. It was found that none of the observed objective metrics correlated well to human perception, with the best correlation coefficient being 0.246. We also found that the objective metrics did not agree with each other regarding which is the best and worst performing music source separation models.",

author = "Erika Rumbold and George Tzanetakis and Bryan Pardo",

note = "Publisher Copyright: {\textcopyright} 2024 Erika Rumbold et al.; 21st Sound and Music Computing Conference, SMC 2024 ; Conference date: 04-07-2024 Through 06-07-2024",

year = "2024",

language = "English (US)",

series = "Proceedings of the Sound and Music Computing Conferences",

publisher = "Sound and music Computing network",

pages = "538--545",

editor = "Guilherme Campos and Guilherme Campos and Ines Salselas and Ines Salselas and Jose Vieira and Jose Vieira and Marco Conceicao and Marco Conceicao and Nuno Fonseca and Rui Penha and Rui Penha and Ponte, {Angela da} and Ponte, {Angela da} and Dimitrios Andrikopoulos and Dimitrios Andrikopoulos",

booktitle = "Proceedings of the 21st Sound and Music Computing Conference, SMC 2024",

}

Rumbold, E, Tzanetakis, G & Pardo, B 2024, CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION. in G Campos, G Campos, I Salselas, I Salselas, J Vieira, J Vieira, M Conceicao, M Conceicao, N Fonseca, R Penha, R Penha, AD Ponte, AD Ponte, D Andrikopoulos & D Andrikopoulos (eds), Proceedings of the 21st Sound and Music Computing Conference, SMC 2024. Proceedings of the Sound and Music Computing Conferences, Sound and music Computing network, pp. 538-545, 21st Sound and Music Computing Conference, SMC 2024, Porto, Portugal, 7/4/24.

CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION. / Rumbold, Erika; Tzanetakis, George; Pardo, Bryan.
Proceedings of the 21st Sound and Music Computing Conference, SMC 2024. ed. / Guilherme Campos; Guilherme Campos; Ines Salselas; Ines Salselas; Jose Vieira; Jose Vieira; Marco Conceicao; Marco Conceicao; Nuno Fonseca; Rui Penha; Rui Penha; Angela da Ponte; Angela da Ponte; Dimitrios Andrikopoulos; Dimitrios Andrikopoulos. Sound and music Computing network, 2024. p. 538-545 (Proceedings of the Sound and Music Computing Conferences).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION

AU - Rumbold, Erika

AU - Tzanetakis, George

AU - Pardo, Bryan

PY - 2024

Y1 - 2024

N2 - The standard quality metric for audio source separation is the signal-to-distortion ratio (SDR), despite correlating poorly to human perception. In this paper, we investigate the correlation between subjective listener evaluations and multiple objective quality metrics, including the SDR. Music tracks are separated using five source separation models, and the resulting separation quality is calculated using five objective evaluation metrics. A listener study was conducted to also obtain subjective evaluations of the music separation quality. It was found that none of the observed objective metrics correlated well to human perception, with the best correlation coefficient being 0.246. We also found that the objective metrics did not agree with each other regarding which is the best and worst performing music source separation models.

AB - The standard quality metric for audio source separation is the signal-to-distortion ratio (SDR), despite correlating poorly to human perception. In this paper, we investigate the correlation between subjective listener evaluations and multiple objective quality metrics, including the SDR. Music tracks are separated using five source separation models, and the resulting separation quality is calculated using five objective evaluation metrics. A listener study was conducted to also obtain subjective evaluations of the music separation quality. It was found that none of the observed objective metrics correlated well to human perception, with the best correlation coefficient being 0.246. We also found that the objective metrics did not agree with each other regarding which is the best and worst performing music source separation models.

UR - http://www.scopus.com/inward/record.url?scp=105000395130&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=105000395130&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:105000395130

T3 - Proceedings of the Sound and Music Computing Conferences

SP - 538

EP - 545

BT - Proceedings of the 21st Sound and Music Computing Conference, SMC 2024

A2 - Campos, Guilherme

A2 - Salselas, Ines

A2 - Vieira, Jose

A2 - Conceicao, Marco

A2 - Fonseca, Nuno

A2 - Penha, Rui

A2 - Ponte, Angela da

A2 - Andrikopoulos, Dimitrios

PB - Sound and music Computing network

T2 - 21st Sound and Music Computing Conference, SMC 2024

Y2 - 4 July 2024 through 6 July 2024

ER -

Rumbold E, Tzanetakis G, Pardo B. CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION. In Campos G, Campos G, Salselas I, Salselas I, Vieira J, Vieira J, Conceicao M, Conceicao M, Fonseca N, Penha R, Penha R, Ponte AD, Ponte AD, Andrikopoulos D, Andrikopoulos D, editors, Proceedings of the 21st Sound and Music Computing Conference, SMC 2024. Sound and music Computing network. 2024. p. 538-545. (Proceedings of the Sound and Music Computing Conferences).

CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE EVALUATIONS OF MUSIC SOURCE SEPARATION

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this