Second-order methods for L1 regularized problems in machine learning

Samantha Hansen; Jorge Nocedal

doi:10.1109/ICASSP.2012.6289101

Second-order methods for L1 regularized problems in machine learning

Samantha Hansen^*, Jorge Nocedal

^*Corresponding author for this work

Industrial Engineering and Management Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Scopus citations

Abstract

This paper proposes a Hessian-free Newton method for solving large-scale convex functions with an L1 regularization term. These problems arise in supervised machine learning models in which it is important to seek a sparse parameter vector. The proposed method operates in a batch setting, which is well suited for parallel computing environments, and employs sub-sampled Hessian information to accelerate progress of the iteration. The method consists of two phases, an active-set prediction phase that employs first-order and second-order information, and subspace phase that performs a Newton-like step. Numerical results on a speech recognition problem illustrate the practical behavior of the method.

Original language	English (US)
Title of host publication	2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 - Proceedings
Pages	5237-5240
Number of pages	4
DOIs	https://doi.org/10.1109/ICASSP.2012.6289101
State	Published - Oct 23 2012
Event	2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 - Kyoto, Japan Duration: Mar 25 2012 → Mar 30 2012

Other

Other	2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012
Country	Japan
City	Kyoto
Period	3/25/12 → 3/30/12

Keywords

Hessian-Free Newton
Iterative Shrinkage
L1 Regularization
Logistic Regression
Newton Method

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

10.1109/ICASSP.2012.6289101

Fingerprint Dive into the research topics of 'Second-order methods for L1 regularized problems in machine learning'. Together they form a unique fingerprint.

Cite this

@inproceedings{2aaa715949fc41fea0e1b6352bb5eda5,

title = "Second-order methods for L1 regularized problems in machine learning",

abstract = "This paper proposes a Hessian-free Newton method for solving large-scale convex functions with an L1 regularization term. These problems arise in supervised machine learning models in which it is important to seek a sparse parameter vector. The proposed method operates in a batch setting, which is well suited for parallel computing environments, and employs sub-sampled Hessian information to accelerate progress of the iteration. The method consists of two phases, an active-set prediction phase that employs first-order and second-order information, and subspace phase that performs a Newton-like step. Numerical results on a speech recognition problem illustrate the practical behavior of the method.",

keywords = "Hessian-Free Newton, Iterative Shrinkage, L1 Regularization, Logistic Regression, Newton Method",

author = "Samantha Hansen and Jorge Nocedal",

year = "2012",

month = oct,

day = "23",

doi = "10.1109/ICASSP.2012.6289101",

language = "English (US)",

isbn = "9781467300469",

pages = "5237--5240",

booktitle = "2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 - Proceedings",

note = "2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 ; Conference date: 25-03-2012 Through 30-03-2012",

}

Hansen, S & Nocedal, J 2012, Second-order methods for L1 regularized problems in machine learning. in 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 - Proceedings., 6289101, pp. 5237-5240, 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012, Kyoto, Japan, 3/25/12. https://doi.org/10.1109/ICASSP.2012.6289101

TY - GEN

T1 - Second-order methods for L1 regularized problems in machine learning

AU - Hansen, Samantha

AU - Nocedal, Jorge

PY - 2012/10/23

Y1 - 2012/10/23

N2 - This paper proposes a Hessian-free Newton method for solving large-scale convex functions with an L1 regularization term. These problems arise in supervised machine learning models in which it is important to seek a sparse parameter vector. The proposed method operates in a batch setting, which is well suited for parallel computing environments, and employs sub-sampled Hessian information to accelerate progress of the iteration. The method consists of two phases, an active-set prediction phase that employs first-order and second-order information, and subspace phase that performs a Newton-like step. Numerical results on a speech recognition problem illustrate the practical behavior of the method.

AB - This paper proposes a Hessian-free Newton method for solving large-scale convex functions with an L1 regularization term. These problems arise in supervised machine learning models in which it is important to seek a sparse parameter vector. The proposed method operates in a batch setting, which is well suited for parallel computing environments, and employs sub-sampled Hessian information to accelerate progress of the iteration. The method consists of two phases, an active-set prediction phase that employs first-order and second-order information, and subspace phase that performs a Newton-like step. Numerical results on a speech recognition problem illustrate the practical behavior of the method.

KW - Hessian-Free Newton

KW - Iterative Shrinkage

KW - L1 Regularization

KW - Logistic Regression

KW - Newton Method

UR - http://www.scopus.com/inward/record.url?scp=84867613625&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867613625&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2012.6289101

DO - 10.1109/ICASSP.2012.6289101

M3 - Conference contribution

AN - SCOPUS:84867613625

SN - 9781467300469

SP - 5237

EP - 5240

BT - 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 - Proceedings

T2 - 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012

Y2 - 25 March 2012 through 30 March 2012

ER -