Logging data in between algorithm iterations

[YuhanLiin]

It'd be nice to have a way of logging information between each step when fitting an iterative algorithm for debugging purposes. For algorithms that use argmin we can simply call add_observer on Executor.

[bytesnake]

perhaps this can be emulated by setting the maximal number of iterations to 1 (or another value if you want to get informed every <n> iterations) and using fit_with. The estimated object should contain information about the current solution

[YuhanLiin]

This is only viable for algorithms where Fit and FitWith use the same algorithm, which isn't always the cases. Also some stats like cost aren't available even on estimated objects. I think using something like log might work here.

[bytesnake]

I think that having debug information in the estimated model is more useful than sparkling prints through the code, but as you said sometimes this is not feasible

[YuhanLiin]

Putting debug info inside models (estimated or completed) is great for debugging purposes, but the Python ML libraries also print debug info in between fitting iterations, which is incredibly useful when dealing with divergence or NaN problems. I think having logging in the code is fine as long as there's an easy way to turn it off or redirect it elsewhere, rather than mindlessly dumping everything on console. We'd have to do some research to figure out the solution that fits our needs.