Integration with Bio-Formats

[sbesson]

The development of the OME-NGFF Bio-Formats readers has been done as an extension of Bio-Formats. The reader is declared in readers.txt but the source code is managed in a separate repository and produces standalone JARs which need to be added to the class path so that the ImageReader constructor can load it. This strategy allowed to have a fast development velocity, merging and releasing new versions of the reader - see https://github.com/ome/ZarrReader/tags - without requiring the entire Bio-Formats stack to be released.

As the reader is maturing and being tested and exposed in the OME stack (Bio-Formats command-line utility, OMERO), this strategy raised a few usability issues. As a matter of example, the usage in the Bio-Formats command-line utility requires:

• to download an uber-JAR containing the reader and all its dependency to be downloaded in addition to the command-line tools bundle
• to configure the classpath to recognize the JAR using the BF_CP command - see ZarrReader: Update to support 0.4.0 datasets #8 (comment)

The usability questions are also relevant to the test-suite component i.e. to test OME-NGFF datasets using the automated repository tests and to a lesser extent to the bio-formats-plugins component i.e the ImageJ/Fiji plugin

Option 1: maintain the statu quo

This is the approach which requires the least amount of work i.e. maintain the code base of ZarrReader separate and completely downstream of the Bio-Formats ecosystem. Concretely this means there are two parallel build paths

formats-api -> formats-bsd, formats-gpl -> bio-formats-tools, bio-formats-plugins,test-suite

and

formats-api -> ZarrReader

This approach still requires separate JARs to be downloaded independently in all cases. Work could likely be done to improve the pluggability experience e.g. by allowing the JAR to be downloaded and copied the right place so that it's automatically detected. But this will likely be per-component and will not scale easily.

Option 2: move ZarrReader upstream back to Bio-Formats

That is the existing paradigm when adding new format readers. The primary advantage of this approach is that it requires no re-architecture of the downstream components which should start including ZarrReader as a core component. One major aspect to discuss is velocity i.e. assuming we want to start consuming changes of ZarrReader very rapidly, would be comfortable with releasing Bio-Formats with a higher frequency than currently?

Architecturally ZarrReader would likely either move as part of the BSD components

formats-api -> formats-bsd (incl ZarrReader), formats-gpl -> bio-formats-tools,bio-formats-plugins,test-suite

or posssible as a new separate BSD-licensed component e.g.

formats-api -> formats-bsd, formats-{zarr,ngff}, formats-gpl -> bio-formats-tools,bio-formats-plugins,test-suite

Option 3: decouple the downstream Bio-Formats components

The third approach to the above would be to keep Zarr reader as a separate repository/JAR downstream of the core BIo-Foramts API + readers but integrate it by creating a dependency of downstrem components like the command line-tools:

(formats-api -> formats-bsd, formats-gpl) -> ZarrReader -> bio-formats-tools,bio-formats-plugins,test-suite

This is probably the solution that requires the biggest amount of design/thoughts as it forces to extract components into separate repository, rethink how they integrate together and what versioning means. For instance in this proposal a Bio-Formats tools could potentially be released on every bump to ZarrReader alone.
On the plus side, this would force us to address some long-standing limitations in our stack (e.g. the fact that the command-line tools depends on bioformats_package.jar which itself depends on bioformats-tools) and potentially provide a blueprint for including other readers without necessarily managing them as part of the core Bio-Formats repository.