Qt widget for loading pose datasets as napari Points layers

[niksirbi]

Description

What is this PR

• Bug fix
• Addition of a new feature
• Other

Why is this PR needed?
This is the 2nd in a series of multiple PRs (following #218), in which I'll try breaking down the work-in-progress contained in #112, and debug any obstacles I encounter along the way. These PRs will be merged into the napari-dev branch, until the plugin becomes minimally functional for users, at which point we'll merge the napari-dev branch into main.

This PR closes #47.

What does this PR do?
It replaces the placeholder "hello" Qt widget with an actual widget that can load movement poses dataset into napari as a Points layer, via functions defined in napari/convert.py. The position data variable is represented in the Points array, while confidence is stored in the layer properties, alongside individual and keypoint names. This means that when hovering over any point in napari, one can see these properties displayed.

In future PRs, the properties will be used to control the appearance of points (i.e. mapping various properties to point size, colour, etc.) For now, the points have a fixed style (coloured by individual ID for multi-individual datasets, by keypoint ID for single-individual datasets). To facilitate the consistent styling of these points, I've defined dataclasses in napari/layer_styles.py, where we can store the default styling per layer type.

Currently, the plugin with the poses loader widget looks like this:

What does this PR NOT do?
These features, though planned, are not essential enough to be part of the "napari prototype" #31:

• No loading of bboxes datasets napari widget for loading bboxes datasets from file #310
• No creation of a napari Tracks layer (that would enable us to display trajectory "tails") Visualise pose tracks as a napari Tracks layer #4

Code structure

The new modules all reside within the movement/napari folder, and include:

• _meta_widget.py: this creates a container of collapsible widgets (imported from brainglobe-utils). Within this container we can stack many widgets, each handling a different task/workflow step. All widgets except for the currently expanded (active) one will appear collapsed. For now this container only houses one widget - the poses loader (see next point).
• _loader_widgets.py: contains the PosesLoader class, which defines the first (and for now, only) collapsible widget. PosesLoader is essentially a frontend for the load_poses.from_file() function. When a file is being loaded, the _on_load_clicked method will call the poses_to_napari_tracks function (see next point) which does all the data wrangling required to transform a movement poses dataset into napari compatible array + properties.
• convert.py: which contains the aforementioned poses_to_napari_tracks function. The idea is that this module will house all movenet->napari and napari->movement conversion utilities. There's something I need to clarify here: why am I creating the data for a napari Tracks layer if my intention is to create a Points layer? That's because the data structure for both of these the same: Points are specified as an array with [z, y, x] columns (in out case that's [time, y, x]), whereas Tracks require columns [track_id, time, y, x]. So by creating a Tracks array, we maintain the track_id information (which we will need in the future anyway), and we get a Points array for "free" (by taking the last 3 columns).
• layer_styles.py: dataclasses defining styles for napari layers. Here I've created a base LayerStyle class with attributes that are common across layer types, and a PointsStyle child class with attributes specific to Points layers. The latter class implements a set_color_by method, which simplifies the re-coloring of the points according to a chosen property (e.g. keypoint or individual).

New unit test modules exactly mirror the above files (see below).

References

• Part of Make a napari frontend prototype #31.
• Preceded by Create skeleton for napari plugin with collapsible widgets #218
• To be followed by Write a guide on how to visualise poses in napari #283

After the current PR is merged, the only absolutely required issue for completing the prototype would be #283. After we have a guide to using the napari plugin, we can merge the prototype to main, and release v0.1 of movement.

How has this PR been tested?

Unit tests have been written for all new modules, and can be found in thetests/test_unit/test_napari_pluginfolder.
The test modules map 1-to-1 to the aforementioned new code modules, so:

• test_meta_widget.py
• test_poses_loader_widget.py
• test_convert.py
• test_layers_styles.py

There is one untested widget method, PosesLoader._on_browse_clicked which opens a file dialog to select a poses file. Despite spending lots of time on this, I couldn't figure out a way to successfully mock all the actions required for testing the file dialog, without actually opening one. Suggestions welcome. Alternatively, this can be opened as a separate issue. I basically decided to open this PR for review without this unit test, because the whole matter was dragging for far too long, and perhaps testing that widget is not so important.

How to review this PR

• Install movement from this branch with dev dependencies: pip install -e .[dev]. This will also include the optional dependencies specified under the napari extra.
• Launch napari with the movement plugin docked: napari -w movement.
• For manual testing of the plugin, it's easiest to use our sample data (locally found under ~/.movement/data).
• Use the "load poses" widget to load the corresponding poses file (for the EPM video, you may use any of the EPM files, from DLC or SLEAP).
• If you wish to visualise the poses against a background, you have two options:
  • Drag and drop a corresponding sample frame into napari (those can be found under ~/.movement/data/frames. This should always work, give you are using the right image for the poses file
  • Install the napari-video plugin, and drag and drop one of the videos found in ~/.movement/data/videos (choose to open it with the video plugin when napari prompts you). Initially I'd planned to include napari-video as a dependency, but I ran into issues during CI. We have to find a way around these as part of Napari plugin reader for videos #49. For what it's worth, the video plugin currently works fine in my M2 Macbook, is laggy on Ubuntu, and fails on Ubuntu CI. See below in "Known issues" for more info.

Known issues

• When displaying properties in the viewer (by hovering over a point), there are way too many decimal points shown for properties with float values. This has been filed as a napari issue.
• As mentioned above, the napari-video plugin currently doesn't perform reliably cross-platform, see related issues:
  • Napari plugin reader for videos #49
  • Why not use opencv-python-headless also for newer versions of Python? postpop/videoreader#6
  • inaccurate seeking janclemenslab/napari-video#3
• Codecov complains because of the aforementioned lack of tests for PosesLoader._on_browse_clicked

Because reviewing this PR will take a long time, I suggest @lochhh and @sfmig you start to slowly work your way through it, while I focus on solving these known issue in parallel.

Is this a breaking change?

No.

Does this PR require an update to the documentation?

Yes, but this will be done as part of #283.

Checklist:

• The code has been tested locally
• Tests have been added to cover all new functionality (almost!)
• The documentation has been updated to reflect any changes
• The code has been formatted with pre-commit

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.80%. Comparing base (9a91fc7) to head (faa1549).
Report is 1 commits behind head on napari-dev.

Additional details and impacted files

@@              Coverage Diff               @@
##           napari-dev     #253      +/-   ##
==============================================
+ Coverage       99.78%   99.80%   +0.02%     
==============================================
  Files              16       18       +2     
  Lines             931     1051     +120     
==============================================
+ Hits              929     1049     +120     
  Misses              2        2              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[niksirbi]

Regarding the un-tested PosesLoader._on_browse_clicked widget, @IgorTatarnikov has pointed me to some tests he wrote for a similar widget here. I should be able to adapt them for our use-case.

[niksirbi]

Hey @lochhh thank you for the comprehensive review!

I've basically implemented all of your excellent suggestions, so have a look and see if you're happy with how I've patched things. I made a bit of a mess with syncing the 3 branches - main, napari-dev, and this one - so the commit history may look a bit scary, but I think should be correct now, without conflicts.

Regarding your additional points:

Is it possible to hide "brainmapper" from the Plugins dropdown?

Good point, it's distracting. I guess this is sneaking in through the brainglobe-utils dependency. I will look into removing that in a separate PR.

There is no option to load an image layer, except via "new image from clipboard". I can't drag and drop either.

Does selecting FIle > Open File(s)... from the top napari menu work? For images that should work out of the box. For videos, it will prompt you to choose a plugin (i.e. napari-video) if you have that installed. Let me know if this works and I'll make it part of the user guide.

We have in movement CLI movement info. We could also have movement gui as an alias to napari -w movement.

Excellent idea! I've opened this as a separate small PR. I don't really like the word "gui" because it's an acronym. How do we feel about movement launch (which also matches datashuttle launch, about which me and Joe had a similar conversation).

[lochhh]

Thanks again @niksirbi for addressing all comments/suggestions. Just a couple more suggestions and we're good to go 🚀

There is no option to load an image layer, except via "new image from clipboard". I can't drag and drop either.

Does selecting FIle > Open File(s)... from the top napari menu work? For images that should work out of the box. For videos, it will prompt you to choose a plugin (i.e. napari-video) if you have that installed. Let me know if this works and I'll make it part of the user guide.

This method works. When loading videos, there was no prompt to choose any plugins. I currently have 4 plugins installed: brainglobe-utils, movement, napari-console, and napari-svg. I also tried opening the single mouse EPM video - it crashed. I then tried with the smaller AEON 3 mice video and loaded the poses on top, this worked!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[niksirbi]

Thanks, again! Suggestions accepted as is.

This method works. When loading videos, there was no prompt to choose any plugins. I currently have 4 plugins installed: brainglobe-utils, movement, napari-console, and napari-svg. I also tried opening the single mouse EPM video - it crashed. I then tried with the smaller AEON 3 mice video and loaded the poses on top, this worked!

Hmm I'll have to look more into the video thing, we already have an issue for that: #49

Anyway, my minimal goal with this PR was being able to overlay poses as Points on an Image layer (in the simplest case, a frame), and its seems we have achieved that, at least.

I will squash-merge this into napari-dev once CI passes.