On the numerical performance of finite-difference-based methods for derivative-free optimization

Hao Jun Michael Shi*, Melody Qiming Xuan, Figen Oztoprak, Jorge Nocedal

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

The goal of this paper is to investigate an approach for derivative-free optimization that has not received sufficient attention in the literature and is yet one of the simplest to implement and parallelize. In its simplest form, it consists of employing derivative-based methods for unconstrained or constrained optimization and replacing the gradient of the objective (and constraints) by finite-difference approximations. This approach is applicable to problems with or without noise in the functions. The differencing interval is determined by a bound on the second (or third) derivative and by the noise level, which is assumed to be known or to be accessible through difference tables or sampling. The use of finite-difference gradient approximations has been largely dismissed in the derivative-free optimization literature as too expensive in terms of function evaluations or as impractical in the presence of noise. However, the test results presented in this paper suggest that it has much to be recommended. The experiments compare newuoa, dfo-ls and cobyla against finite-difference versions of l-bfgs, lmder and knitro on three classes of problems: general unconstrained problems, nonlinear least squares problems and nonlinear programs with inequality constraints.

Original languageEnglish (US)
Pages (from-to)289-311
Number of pages23
JournalOptimization Methods and Software
Volume38
Issue number2
DOIs
StatePublished - 2023

Keywords

  • Derivative-free optimization
  • finite differences
  • noisy optimization
  • nonlinear optimization
  • zeroth-order optimization

ASJC Scopus subject areas

  • Software
  • Control and Optimization
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'On the numerical performance of finite-difference-based methods for derivative-free optimization'. Together they form a unique fingerprint.

Cite this