Solver - add a wrapper for scipy L-BFGS solver

[Badr-MOUFAD]

This adds a BFGS to the solvers' toolbox of skglm.
It proceeds as follows:

• add wrapper around scipy.optimize.minimize(method="L-BFGS-B")
• unitest against scikit-learn L2-Logistic regression

[mathurinm]

Can you run it quickly on moderate size datasets in https://github.com/benchopt/benchmark_logreg_l2 ?

[mathurinm]

I believe this is quite small, is it enough on a 1000 x 1000 dataset for example?

[Badr-MOUFAD]

50?

[Badr-MOUFAD]

Can you run it quickly on moderate size datasets in https://github.com/benchopt/benchmark_logreg_l2 ?

Here are the benchmark figures of skglm lbfgs vs. scikit-learn lbfgs

[mathurinm]

Can you run it quickly on moderate size datasets in https://github.com/benchopt/benchmark_logreg_l2 ?

Here are the benchmark figures of skglm lbfgs vs. scikit-learn lbfgs

I can't see the iterations on it, how many iterations does it take to converge?
On some datasets there seem to be a very small difference in final objective value (1e-6), it's not blocking but do you know the cause?

[Badr-MOUFAD]

My apologies, I performed the benchmark using an old version of benchopt.

Here the new figures.

On some datasets there seem to be a very small difference in final objective value (1e-6), it's not blocking but do you know the cause?

Several parameters determine the convergence of scipy L-BFGS-B, among which ftol. In the new benchmark I set it to zero so that to account only for max_iter. With that being applied, we get an objective slightly below scikit-learn LBFGS

Should we fix that, @mathurinm?