FileCallbackHandler

[mbchang]

like StdoutCallbackHandler, but writes to a file

When running experiments I have found myself wanting to log the outputs of my chains in a more lightweight way than using WandB tracing. This PR contributes a callback handler that writes to file what StdoutCallbackHandler would print.

Example Notebook

See the included filecallbackhandler.ipynb notebook for usage. Would it be better to include this notebook under modules/callbacks or under integrations/?

Who can review?

Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested:

@agola11

[agola11]

There seems to be a lot of duplicated code between StdOutCallbackHandler and this one, can we just add a file option to StdoutCallbackHandler and if it's set, call print_text with the file?

[agola11]

you don't need to implement methods you aren't using, BaseCallbackHandler is not an abstract class

[eyurtsev]

@agola11 is there any code to guarantee that cleaned up is invoked by handlers on failure? Not critical in this case, but occurred to me that maybe there are some situations we'd want to have cleanup invoked

I don't know enough about the python data model... del i assume dell might not be invoked on exception

[mbchang]

Sounds good; I removed them.

[mbchang]

Modifying StdoutCallbackHandler could work, but I wonder if it could cause difficulties in separating functionality, which was why I thought that using an explicitly separate handler could make things clearer to the user.

Specifically, there are four cases the user might want:

1. do not print to stdout, do not print to file
2. print to stdout, do not print to file (this is the current behavior that verbose=True triggers)
3. do not print to stdout, print to file
4. print to stdout, print to file

If we modify StdoutCallbackHandler to also print to file if a file is given, this would allows us to handle cases 1, 2, 4. Any suggestions on how we could handle case 3?

If StdoutCallbackHandler handles both printing to file and printing to stdout could create ambiguity on the user side with respect to what verbose=True would do. If the user simply sets verbose=True, the behavior would be that it prints to stdout, but does not write to file. If the user wants to both print to stdout and also write to file, then the user would pass the StdoutCallbackHandler in explicitly. This ambiguity might not be a big issue, but just flagging this as a possible concern. What do you think?

[mbchang]

UPDATE: I realized that one solution to the above concerns could be for the user to pass in two StdoutCallBackHandlers, one with a file and one without a file, to the chain. So I will modify StdoutCallBackHandlers to take a file in as an optional input.

[mbchang]

On second thought, passing in two StdoutCallBackHandlers might not work. This is because when the callbacks are configured, if verbose=True, then this line checks instantiates a new StdoutCallBackHandler to log to stdout. But if I pass an StdoutCallBackHandler with a file argument, then setting verbose=True will not instantiate another StdoutCallBackHandler to print to stdout.

We would get the following unexpected behavior below, where setting verbose=True does not result in printing stuff to stdout.

[hwchase17]

@agola11 mind taking another look?

[agola11]

makes sense thanks!