scijit.optimize.minimize¶

scijit.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None, lower=None, upper=None, maxiter=0)¶

Minimize a scalar function of one or more variables.

Eleven algorithms behind one method string. Seven answer to the standard method names; four more are the remaining PRIMA derivative-free solvers.

Callable from Python and from inside @njit. Both entries run the same compiled backends and reach them through the same runners. They differ in the CLASS an argument error carries, which the Raises section names, because the compiled entry decides those while the call compiles.

fun is a plain @njit function in every case, including for the four PRIMA methods.

Parameters:

funcallable: A plain @njit function called fun(x, *args), in either of two shapes: -> f returns the objective, -> (f, g) returns it together with its gradient. The return type is read when the call compiles, so neither shape needs a flag and neither costs an evaluation.
x0array_like: Initial guess, one dimension.
argstuple, optional: Extra parameters, unpacked into the argument list of fun and jac after x, so args=(a, b) calls fun(x, a, b). Its elements may be of different types and shapes. Default (), which calls fun(x). A non-tuple args is a one-item tuple, so an ndarray reaches fun as ONE argument.
methodstr or None, optional: 'Nelder-Mead', 'Powell', 'CG', 'BFGS', 'L-BFGS-B', 'SLSQP', 'COBYLA', or one of the additional 'UOBYQA', 'NEWUOA', 'BOBYQA', 'LINCOA'. Matched case-insensitively. None (default) selects 'SLSQP' with constraints, 'L-BFGS-B' with bounds and 'BFGS' otherwise. From inside @njit it must be a literal written at the call site: the result’s field set is the method’s, and a compiled function has one return type per signature. A variable there raises numba.TypingError. The Python entry takes any string.
jaccallable, bool, int or None, optional: jac(x, *args) -> array, the gradient, called with the same args as fun. None (default), False and 0 all mean no gradient: the gradient then comes from an fg-style fun, or, where the method needs one and fun returns only f, from forward differences.
hess, hesspcallable or None, optional: Accepted and ignored, with a RuntimeWarning naming the method. No method reached from here uses second-order information.
bounds(n, 2) array_like or None, optional: One (min, max) pair per variable. None (default) is unbounded. Use -np.inf / np.inf for a one-sided bound. Used by 'L-BFGS-B', 'SLSQP', 'COBYLA', 'BOBYQA' and 'LINCOA'. The other six warn and ignore them. A (0, 2) array is also unbounded; it counts as bounds for the method default, because that is settled from the type rather than the length. See Notes.
constraintstuple, optional: Accepted only as empty. 'SLSQP' and 'COBYLA' raise on a non-empty one; the other nine warn and ignore it. fmin_slsqp() and fmin_cobyla() take constraints.
tolfloat or None, optional: Headline tolerance for the chosen method. None (default) uses that method’s own: xatol/fatol 1e-4 for Nelder-Mead, xtol/ftol 1e-4 for Powell, gtol 1e-5 for CG and BFGS, gtol 1e-5 and ftol 2.220446049250313e-09 for L-BFGS-B, acc 1e-6 for SLSQP, rhoend 1e-4 for COBYLA and 1e-6 for the PRIMA four. A value given sets every tolerance the method has, so 0.0 runs it to its iteration cap.
callbackcallable or None, optional: Called once per iteration as callback(xk), with xk the current iterate. Either a plain Python callable or an @njit function of that shape. A Python callable whose single parameter is named intermediate_result is called with an OptimizeResult instead, carrying x and fun. Raising StopIteration halts the solve, and the result then carries status 99 and success False. Served by 'Nelder-Mead', 'Powell', 'CG', 'BFGS', 'L-BFGS-B' and 'SLSQP'.
optionsdict or None, optional: Solver options. maxiter reaches maxiter. Three methods have a tolerance PAIR and each name reaches a slot of its own: xatol and fatol on Nelder-Mead, xtol and ftol on Powell, ftol and gtol on L-BFGS-B. For the rest, xtol, ftol, gtol, pgtol, acc, rhoend and tol reach tol; rhobeg reaches 'COBYLA' and the PRIMA four, npt reaches 'NEWUOA', 'BOBYQA' and 'LINCOA', maxfev reaches 'Nelder-Mead' and 'Powell', and maxfun reaches 'L-BFGS-B'. Any other key draws an OptimizeWarning naming it and is ignored, and the key set is per method. From inside @njit this is a dict LITERAL at the call site.
lower, upperndarray, optional: The bounds as two separate arrays, which is the form the Fortran drivers take. Each is empty (default) or length n. Ignored when bounds is non-empty.
maxiterint, optional: Iteration budget. 0 (default) uses the method default: 200*n for Nelder-Mead, CG and BFGS, 1000*n for Powell, 15000 for L-BFGS-B, 100 for SLSQP, 1000 objective evaluations for COBYLA and 500*n for the PRIMA four.

Returns:

resOptimizeResult

x is the minimizer and fun its objective value. Every method also carries nfev, status, message and success. The rest of the field set is the METHOD’s, and a field the method did not compute is absent:

method	also carries
‘Nelder-Mead’	`nit`, `final_simplex`
‘Powell’	`nit`, `direc`
‘CG’	`nit`, `jac`, `njev`
‘BFGS’	`nit`, `jac`, `njev`, `hess_inv`
‘L-BFGS-B’	`nit`, `jac`, `njev`, `hess_inv`
‘SLSQP’	`nit`, `jac`, `njev`, `multipliers`
‘COBYLA’	`maxcv`
the PRIMA four	nothing further

Reading an absent field raises AttributeError, and numba.TypingError from inside @njit. res.keys() lists what a given result holds.

Raises:

ValueError

If method is not one of the eleven; if constraints is non-empty on 'SLSQP' or 'COBYLA'; if jac is neither None nor an @njit function; if jac is given together with an fg-style fun; if bounds is not (n, 2) or has an upper bound below its lower one; if x0 has more than one dimension; if lower or upper has a length that is neither 0 nor n; or if fun or jac returns a gradient whose length is not len(x0).

From inside @njit the first four of those are numba.TypingError instead, because the condition is decided while the call compiles rather than while it runs.

A callback given to 'COBYLA', 'UOBYQA', 'NEWUOA', 'BOBYQA' or 'LINCOA' raises ValueError, and numba.TypingError from inside @njit where method is a literal. Those five reach PRIMA, whose wrapper passes no callback_fcn.

TypeError

If fun or jac does not bind (x, *args) for the args given, which from inside @njit is a numba.TypingError; if x0 is complex, in any of the array, list and tuple spellings, from both entry points; if options is neither a mapping nor None, which from inside @njit is a numba.TypingError.

See also

scipy.optimize.minimize: The scipy routine this mirrors.
scijit.optimize.fmin_l_bfgs_b: L-BFGS-B with its full control set.
scijit.optimize.fmin_slsqp: SLSQP with constraints.
scijit.optimize.fmin_cobyla: COBYLA with nonlinear constraints.
scijit.optimize.minimize_scalar: One variable, no starting point.

Notes

THE METHOD TABLE. gradient is what the method does with jac; bounds is whether it uses them; prange is whether concurrent calls are safe.

method	gradient	bounds	prange	backend
‘Nelder-Mead’	ignored	no	yes	pure port
‘Powell’	ignored	no	yes	pure port
‘CG’	uses	no	yes	pure port
‘BFGS’	uses	no	yes	pure port
‘L-BFGS-B’	uses	yes	yes	Fortran
‘SLSQP’	uses	yes	yes	Fortran
‘COBYLA’	ignored	yes	yes	Fortran
‘UOBYQA’	ignored	no	yes	Fortran
‘NEWUOA’	ignored	no	yes	Fortran
‘BOBYQA’	ignored	yes	yes	Fortran
‘LINCOA’	ignored	yes	yes	Fortran

Every method is safe to call from a numba.prange loop. The reverse-communication solvers keep their state in caller-owned arrays; PRIMA reaches its callback through a Fortran module variable carrying !$omp threadprivate, so that resolves to one slot per thread. The 32-thread measurement behind the PRIMA half is in scijit.optimize.minimize_newuoa; the pure ports hold no shared state to corrupt.

constraints is empty for every method. Nonlinear constraints reach COBYLA through fmin_cobyla(), equality and inequality constraints reach SLSQP through fmin_slsqp(), and linear constraints reach LINCOA through minimize_lincoa().

nit and jac. Nelder-Mead, Powell, CG, BFGS, L-BFGS-B and SLSQP report an iteration count. COBYLA and the PRIMA four count objective evaluations instead and carry no nit. CG, BFGS, L-BFGS-B and SLSQP report the gradient at the solution; the other seven carry no jac.

THE RESULT’S FIELD ORDER is the same for every method: x, fun, the method’s own outputs, the counters, then status, message, success. scipy’s orders differ from each other, 'BFGS' and 'L-BFGS-B' carrying one field set in two orders, so there is no one order to match. Reaching a field by name or by attribute is unaffected.

multipliers on 'SLSQP' holds the constraint multipliers of the quadratic subproblem the solver finished on, one per constraint, the equalities first. Its length is the number of constraints, so (0,) where there are none. Bound rows contribute no entry.

WHICH SCIPY METHODS ARE ABSENT. 'Newton-CG', 'TNC', 'COBYQA', 'dogleg', 'trust-ncg', 'trust-krylov', 'trust-exact' and 'trust-constr' raise ValueError('Unknown solver <method>'), which is scipy’s own text for a name it does not have. A callable method, which scipy forwards **options to, raises as well.

scipy 1.18’s 'COBYLA' is PRIMA, which is the library behind this one too, so the two run the same implementation rather than two versions of the same idea.

THE OBJECTIVE. Both shapes are accepted and told apart by the return type. An fg objective handed to a method that takes a scalar one is split into two compiled functions, each of which calls it and drops half of what it computed: the evaluation counts match scipy’s, the work per evaluation does not. A scalar objective with jac handed to SLSQP is joined into one fg for the same reason.

args is unpacked into the objective’s argument list, fun(x, *args). A non-tuple args is read as a one-item tuple and so reaches fun as one argument. The four PRIMA methods reach the objective through a C function pointer, which carries one double*: the elements cross it flattened and are rebuilt before the call, so the same spellings work.

jac accepts None, False, 0 or a compiled gradient. scipy’s '2-point', '3-point' and 'cs' are not implemented; jac=None already forward-differences on the methods that need a gradient, which is what scipy’s '2-point' does. jac=True is spelled by having fun return (f, g). From inside @njit, False and 0 have to be written at the call site: a variable holding one is refused, because its value is not known when the call compiles, and scipy accepts it.

constraints is unavailable as a list of dicts holding callables.

THE CALLBACK. From inside @njit only an @njit callback is reachable, because a Python callable cannot cross into compiled code as an argument. A Python callback reaches the solver through a module-level slot and takes the GIL once per iteration, so two solves running at once in a numba.prange loop overwrite each other’s callback and the loop stops running in parallel. An @njit callback travels as an argument and does neither.

An exception other than StopIteration from a Python callback reaches the caller after the solve has run its exit path, where scipy raises it from inside the solver loop, so the traceback carries no solver frames. An @njit callback halts the solve on ANY exception, and the result then carries the status 99 a StopIteration gives.

res is a scijit.optimize.OptimizeResult, not scipy’s. Its field set is the solver’s, as scipy’s is: COBYLA carries maxcv, Nelder-Mead final_simplex, Powell direc, SLSQP multipliers, and BFGS and L-BFGS-B hess_inv. A field the method did not compute is absent, and reading it raises AttributeError from Python and a TypingError when the call compiles.

'Nelder-Mead' and 'Powell' DO NOT USE bounds, and warn that they cannot. scipy’s do use them: on the box [(0, 0.5), (0, 0.5)] with (x[0] - 1)**2 + (x[1] - 2)**2 from (0, 0), scipy returns [0.5, 0.5] and [0.49997985, 0.5] where this returns [1., 2.] on both. The warning is what makes that visible, and it is a warning scipy does not raise.

A (0, 2) bounds array is unbounded and still counts as bounds for the method=None default, because that is settled from the type rather than the length. scipy has no counterpart for this spelling.

x0 with every variable pinned by equal bounds goes to the solver here. scipy short-circuits it and returns a result built without calling the solver, whose field set is smaller again.

A gradient whose length is not len(x0) raises, and so does a lower or upper of the wrong length. scipy 1.18 raises on neither of the two Fortran methods offered here.

options reaches the arguments listed under Parameters, and every other key draws an OptimizeWarning naming it, which is scipy’s class and scipy’s text for a key IT does not read. So a key scipy reads and this does not, disp, return_all, maxcor, eps, maxls, initial_simplex, direc, norm, adaptive, catol and iprint among them, is announced rather than honoured.

tol and tol name one quantity twice, and so do maxiter and maxiter. Where both are given, options wins, which is scipy’s setdefault order.

A complex x0 raises TypeError. scipy 1.18 accepts one and returns complex128, and the imaginary part never moves from its starting value: measured on f(z) = |z - (2+3j)|**2, whose minimiser is 2+3j and whose minimum is 0, x0 = 1+1j gives x = 1.9999999504278225+1j and fun = 9.0, x0 = 0j gives x = 2.000000136913649+0j and fun = 9.0, and x0 = 5-2j gives x = 2.0000004414283588-2j and fun = 9.0, each with success=True.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html

Examples

The objective returns the value and the gradient together:

>>> import numpy as np
>>> from numba import njit
>>> from scijit.optimize import minimize
>>> @njit
... def fg(x):
...     f = (x[0] - 1.0) ** 2 + (x[1] - 2.5) ** 2
...     return f, np.array([2.0 * (x[0] - 1.0), 2.0 * (x[1] - 2.5)])
>>> @njit
... def run():
...     return minimize(fg, np.array([0.0, 0.0]))
>>> res = run()
>>> res.x
array([1. , 2.5])
>>> res.success
True

scipy’s shape, an objective returning only the value:

>>> @njit
... def f(x):
...     return (x[0] - 1.0) ** 2 + (x[1] - 2.5) ** 2
>>> @njit
... def run_nm():
...     return minimize(f, np.array([0.0, 0.0]), method='Nelder-Mead')
>>> np.round(run_nm().x, 6)
array([1.000008, 2.499982])

Nelder-Mead stops on a simplex smaller than xatol, 1e-4 by default, so the last digits above are the tolerance rather than the arithmetic.

Extra parameters, one array and one scalar, unpacked after x:

>>> @njit
... def fa(x, target, w):
...     return w * ((x[0] - target[0]) ** 2 + (x[1] - target[1]) ** 2)
>>> @njit
... def run_args():
...     return minimize(fa, np.array([0.0, 0.0]),
...                     args=(np.array([1.0, 2.5]), 3.0))
>>> np.round(run_args().x, 6)
array([1. , 2.5])

A derivative-free PRIMA method, from the same plain function:

>>> @njit
... def run_newuoa():
...     return minimize(f, np.array([0.0, 0.0]), method='NEWUOA')
>>> np.round(run_newuoa().x, 6)
array([1. , 2.5])

Bounded, with SLSQP:

>>> @njit
... def run_bounded():
...     return minimize(fg, np.array([0.0, 0.0]), method='SLSQP',
...                     bounds=np.array([[0.0, 0.5], [0.0, 0.5]]))
>>> run_bounded().x
array([0.5, 0.5])