Naming for "xtol" and "ftol" criteria in PINTS (max_unchanged_iterations)

[MichaelClerx]

As part of #1502 I'm wondering what to do about the names of the stopping criteria based on a minimum change per iteration

At the moment we have set_max_unchanged_iterations, which sets an ftol type criterion (stop when the best function value changes by less than X for N iterations). I'm adding one that sets an xtol type criterion (stop when the best parameter value changes by less than X for N iterations). Current names in some software:

Stop when	Matlab	Scipy (fmin)	nlopt	pycma	PINTS
f(x) change < t	FunctionTolerance, TolFun	ftol	ftol	tolfun	unchanged_iterations
x change < t	StepTolerance, TolX	xtol	xtol	tolx	?
f(x) < t			stopval	ftarget	threshold

• https://uk.mathworks.com/help/optim/ug/tolerances-and-stopping-criteria.html
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html
• https://nlopt.readthedocs.io/en/latest/NLopt_Introduction/#termination-conditions
• https://github.com/CMA-ES/pycma/blob/c52672315c4f045c1dc8d95a1ff73b89c42bb071/cma/evolution_strategy.py#L315-L318

I'd gone for unmoved_iterations but initial feedback says no. So now thinking we (deprecate and) rename the unchanged_iterations methods to something that includes the word function?

[MichaelClerx]

Highly used function though, so would love input from as many @pints-team/devs as possible!

@chonlei @DavAug @FarmHJ who else uses optimisers?

[EricWay1024]

Honestly speaking the name set_max_unchanged_iterations was a bit confusing when I first looked at it, so I think it could be a good idea to rename it to something like set_function_tolerance_iterations while naming the xtol method as set_parameter_tolerance_iterations.

[Rebecca-Rumney]

I have been using scipy optimisation, and fmin is a legacy function. The current minimize function uses fatol and xatol for absolute differences and uses ftol and xtol for relative differences. If we then use ftol and xtol, people used to scipy might think we mean the relative differences not absolute. Looking at NLOpt, they use ftol_rel and ftol_abs to differentiate.

Although I do agree, set_max_unchanged_iterations is a little confusing on what it is setting.

[DavAug]

As part of #1502 I'm wondering what to do about the names of the stopping criteria based on a minimum change per iteration

At the moment we have set_max_unchanged_iterations, which sets an ftol type criterion (stop when the best function value changes by less than X for N iterations). I'm adding one that sets an xtol type criterion (stop when the best parameter value changes by less than X for N iterations). Current names in some software:

Stop when Matlab Scipy (fmin) nlopt pycma PINTS
f(x) change < t FunctionTolerance, TolFun ftol ftol tolfun unchanged_iterations
x change < t StepTolerance, TolX xtol xtol tolx ?
f(x) < t stopval ftarget threshold

• https://uk.mathworks.com/help/optim/ug/tolerances-and-stopping-criteria.html
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html
• https://nlopt.readthedocs.io/en/latest/NLopt_Introduction/#termination-conditions
• https://github.com/CMA-ES/pycma/blob/c52672315c4f045c1dc8d95a1ff73b89c42bb071/cma/evolution_strategy.py#L3927C1-L3937C63

I'd gone for unmoved_iterations but initial feedback says no. So now thinking we (deprecate and) rename the unchanged_iterations methods to something that includes the word function?

Thank you Michael for working on this!

I may be alone on this, but I actually find the set method to max_unchanged_iterations quite clear. Perhaps, the missing information is just that this relates to the stopping criterion.

So, I am wondering whether set_stopping_criteria would do the trick, complemented by a detailed doc string (given that stopping criteria could mean anything)? The arguments to this method could be function_threshold, parameter_threshold, and n_iterations. The max_unchanged_iterations could then, perhaps, also be replaced by a stopping_criteria method which returns all three values in a meaningful way.

Looking forward to hearing your thoughts! :)

[mjowen]

I'd avoid being too similar to the ftol and xtol (or xatol and fatol) standards as I think they carry the implication of checking with the previous iteration rather than the default in pints of 200 iterations ago.
I don't think anywhere (apart from pints) has a method for specifying the number of iterations in these types of termination criteria.

So, I am wondering whether set_stopping_criteria would do the trick, complemented by a detailed doc string (given that stopping criteria could mean anything)? The arguments to this method could be function_threshold, parameter_threshold, and n_iterations. The max_unchanged_iterations could then, perhaps, also be replaced by a stopping_criteria method which returns all three values in a meaningful way.

I like the idea of keeping it all isolated to a single function. Makes it clear that if you're using multiple termination criteria then they should stack rather than override each other.
Threshold gives me the implication that it would terminate when f is less than function_threshold rather than the change in f. I'd probably go for function_tolerance and parameter_tolerance to be in place of max_unchanged_iterations and max_unmoved_iterations, a buffer (or function_buffer and parameter_buffer) as the number of unchanged iterations and then its clearly separated from a max_iterations termination criteria