About Derivative-Free Solvers
The defining characteristic of the derivative-free solvers is that they do not need to compute derivatives of the objective function with respect to the control variables. They do not even require the objective function to be differentiable in principle. This characteristic makes them suitable for problems where the objective function is nonsmooth or contains noise.
One typical example of a noisy objective function is when the control variables define geometry dimensions. The geometry changes induced by modifying the control variables then lead to different finite element meshes, superimposing different discretization errors on the objective function when evaluated for different control variable values.
Since the derivative-free solvers do not trust the pointwise behavior of the objective function to be a good indicator of where to search for the next, improved, update to the control variables, they must rely on sampling the objective function at different positions in the control variable space. This is more expensive than following a single path toward the optimum but also more robust. Some of the performance penalty is offset by the fact that evaluations that do not depend on one another can be done in parallel — for example, in a cluster environment.
Finally, derivative-free solvers can be further subdivided into local, “hill-climbing,” methods and global, evolutionary or statistical, methods. The former type starts from an initial guess and strives to improve the objective function in a stepwise manner. Imagine a group of people trying to climb a hill together in dense fog; as long as they stay together and move upward, they find a top but not necessarily the highest one. Global methods, in contrast, try to produce a map of the entire design space, refining it iteratively in areas that appear to be good candidates for containing the global optimum.
The Optimization Module provides five different derivative-free algorithms:
•
The Nelder–Mead solver walks toward improved objective function values by iteratively replacing the worst corner of a simplex in the control variable space. See The Nelder–Mead Solver.
•
The BOBYQA solver walks toward improved objective function values by using an iteratively constructed quadratic approximation of the objective. See The BOBYQA Solver.
•
The COBYLA solver solves a sequence of linear approximations constructed from objective and constraint values sampled at the corners of a simplex in control variable space. See The COBYLA Solver.
•
The EGO solver iteratively searches for improved objective function values by constructing a Gaussian process surrogate model to approximate the objective function. See The EGO Solver.
These methods are each described in more details below.