CLI demos revamp, filesystem cleanliness

[pgunn]

Hello,
Two of the changes I'm working on that will likely make it into the next release, barring surprises, are:
A) a revamp of the CLI demos of Caiman to be more "applications" that you probably don't need to edit to do useful stuff with your data
B) Some changes to file handling code to make temporary files behave better (for those demos and generally)

In one of the recent releases of Caiman, I added the ability to set CNMFParams using a json file, the idea being that if you don't want to, you don't need to have those settings in your code, and you might prefer to tweak your code and the settings separately. This makes scripting a bit easier too.

The CLI demos were designed with the idea that you'd either look at them to see how something is done (with the idea that you'd write your own scripts or notebooks from scratch using the Caiman APIs) or as a template that you'd edit your specifics into.

With the revamped demos, I hope that you usually wouldn't need to edit them (you still can), and that instead you can get all their functionality through commandline arguments -- significantly by providing a json file with CNMFParams (that can specify your data source, although they're also designed to let you override your json file with a different commandline argument).

While doing this, I took a stab at a medium-term fix for another problem that Caiman has - it tends to drop off temporary mmap and other files in a variety of places where it shouldn't; it now should make at least a moderate effort to do that less, putting temporary files in the temp directory under caiman_data. I doubt I caught everything for this, and longer-term I expect we'll make per-run directories with well-defined outputs and saved parameters from every run, but I think this is a useful step in that direction.

The original demos will still be available; there's a pointer to them in a new readme under the demos directory.

This set of changes isn't quite done yet, but it's getting close.

[pgunn]

The CLI demos have been badly neglected for awhile; while the revised demos are not perfect, they're in a much better shape than before (many didn't show their results before exiting because of changes between the Spyder era of the code and the really-from-the-cli era).

[vncntprvst]

Hi Pat,
I noticed the warning In setting CNMFParams, non-pathed parameters were used; this is deprecated. In some future version of Caiman, allow_legacy will default to False (and eventually will be removed)") int the outputs recently. I suppose this is related to the changes you're discussing above.
In our lab's pipeline, we do update parameters (ironically, from a json file), but not using the change_params function, or at least not directly, since we typically use Mesmerize. Our code looks something like this:

    df.caiman.add_item(
        algo='cnmf', 
        input_movie_path=df.iloc[index], 
        params=params_cnmf,
        item_name=df.iloc[index]["item_name"], 
    )
                    
    for _, row in df.iterrows():
        log_and_print(f"Running batch item {row.name}, id {row.uuid}, algo {row.algo}.")
        process = row.caiman.run()

It's not immediately clear what "non-pathed parameters" mean.
Is there an documented example somewhere to demonstrate the intended use of change_params? Or could you write one?
Thanks!

[pgunn]

@vncntprvst

Hello,
The new CLI demos offer an example; the basic idea is to have the parameters in input match the parameter output when you ask for a json representation of the CNMF object; CNMFParams had this structure internally from the beginning, but it accepted setting of parameters without the paths (e.g. data.fr versus just fr). Unfortunately, this is both bad for cleanliness and it led to some bugs because some parameters shared names with the categories (like ring_CNN) leading to subtle bugs.

https://github.com/flatironinstitute/CaImAn/blob/main/demos/general/params_demo_pipeline.json
offers an example of a json file (and CNMFParams now can load settings from a json file). Code doing the same with parameters inline would just pass in a dict of dicts rather than a flat dict, matching the json file.

I will probably put together a PR to mesmerize at some point, although there's no huge rush (for you or for me) - the old flat APIs will eventually go away but it will be quite some time before that switch will be flipped. The most important concerns were taken care of with the CLI demo revision, and I'll be making similar changes to the Jupyter notebooks in some upcoming release.

Cheers,
Pat

[vncntprvst]

That sounds good. Thanks for the explanation!