Add Skeleton Templates #1122

aaprasad · 2023-01-17T23:29:44Z

Description

Finished MVP for skeleton templates:

stored skeleton jsons corresponding to each of the reference datasets in sleap/skeletons
added dropdown to select from one of the above skeleton templates
Modified the skeleton's load button to load from the selected skeleton or a user defined file labeled Custom in the dropdown

Types of changes

Does this address any currently open issues?

#1106

Outside contributors checklist

Review the guidelines for contributing to this repository
Read and sign the CLA and add yourself to the authors list
Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
Add tests that prove your fix is effective or that your feature works
Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

Added `skeletons` folder containing json for each skeleton from [reference dataset](https://sleap.ai/datasets.html). Changed `name` section to corresponding dataset name

fixed fly32 name attribute to correctly match dataset name

Added dropdown functionality to select from preset templates or custom .json file Changed load button to load template or custom depending on selection

codecov · 2023-01-17T23:58:30Z

Codecov Report

Merging #1122 (3c84be4) into develop (57ee08a) will increase coverage by 0.40%.
The diff coverage is 91.52%.

@@             Coverage Diff             @@
##           develop    #1122      +/-   ##
===========================================
+ Coverage    71.92%   72.32%   +0.40%     
===========================================
  Files          131      132       +1     
  Lines        23035    23326     +291     
===========================================
+ Hits         16567    16870     +303     
+ Misses        6468     6456      -12

Impacted Files	Coverage Δ
sleap/gui/commands.py	`59.27% <67.44%> (+0.71%)`	⬆️
sleap/gui/dialogs/merge.py	`52.77% <90.26%> (+29.59%)`	⬆️
sleap/gui/widgets/views.py	`96.15% <96.15%> (ø)`
sleap/nn/viz.py	`66.87% <97.67%> (+28.53%)`	⬆️
sleap/gui/app.py	`78.46% <98.36%> (+1.28%)`	⬆️
sleap/instance.py	`89.07% <100.00%> (+0.32%)`	⬆️
sleap/io/dataset.py	`83.38% <100.00%> (+0.01%)`	⬆️
sleap/skeleton.py	`91.15% <100.00%> (+0.49%)`	⬆️
sleap/util.py	`92.36% <100.00%> (+0.44%)`	⬆️
sleap/gui/dataviews.py	`77.21% <0.00%> (-1.27%)`	⬇️
... and 6 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

fixed assert function in `test_OpenSkeleton` to not depend on order of `skeleton.nodes` thru comparing set diff

roomrys

Few small non-functional changes. Otherwise LGTM. Good to start next PR after small changes.

tests/gui/test_commands.py

sleap/gui/commands.py

talmo · 2023-01-26T17:28:57Z

Notes on finishing the integration:

Preview image + description should show on selecting the skeleton from the dropdown, not just after clicking Load
Description should have auto-generated text describing the skeleton, e.g.: f"Nodes: {len(skeleton.nodes)}. Edges: {len(skeleton.edges)}. Node names: {'\n'.join(skeleton.node_names)}"
We need a "Clear" button that deletes the current skeleton so that it's easier to actually switch between skeletons instead of merging
- Danger zone: Clearing the skeleton might delete all annotations from existing instances? At a minimum we should add a confirmation dialog box.
Unify node names so that the same type of body part is named the same across skeleton variants (e.g., "tail_base" and "tti" in mice_of vs mice_hc)

sleap/gui/app.py

sleap/util.py

tests/nn/test_viz.py

tests/gui/test_dialogs.py

roomrys · 2023-02-16T20:21:16Z

Notes on finishing the integration:

Preview image + description should show on selecting the skeleton from the dropdown, not just after clicking Load

Description should have auto-generated text describing the skeleton, e.g.: f"Nodes: {len(skeleton.nodes)}. Edges: {len(skeleton.edges)}. Node names: {'\n'.join(skeleton.node_names)}"

We need a "Clear" button that deletes the current skeleton so that it's easier to actually switch between skeletons instead of merging

Danger zone: Clearing the skeleton might delete all annotations from existing instances? At a minimum we should add a confirmation dialog box.

Unify node names so that the same type of body part is named the same across skeleton variants (e.g., "tail_base" and "tti" in mice_of vs mice_hc)

All of the above except for 3 have been implemented. I think the replace skeleton table takes care of this though?

roomrys · 2023-02-16T20:31:56Z

sleap/io/dataset.py

@@ -2618,6 +2618,7 @@ def video_callback(
                        for i, filename in enumerate(filenames):
                            if missing[i]:
                                filenames[i] = new_paths[i]
+                                missing[i] = False


Flagging this since it is outside of the scope of the PR (noticed that when loading file through API, found videos are not replaced if USE_DUMM_FOR_MISSING_VIDEOS was set to True) - this line fixes that and only uses dummy for truly missing videos

roomrys

Nits

sleap/skeleton.py

sleap/nn/viz.py

roomrys · 2023-02-22T23:16:20Z

The new skeleton dialog:

Open

Collapsed

talmo

Let's take a leap of faith and see what happens!!

* GUI Training: Use hidden params from loaded config (#1053) * Add optional unragging arg to model export (#1054) * Fix config option to `split_by_inds` (#1060) * Convert training, validation, and test to Labels object * Add test for split_by_inds * Use Labels.extract instead of Labels(List[LabeledFrames]) * Tracking: robust assignment of the best score to an instance (#1062) * Set max instances for top down models (#1070) * Add optional unragging arg to model export * Add option to set max instances for multi-instance models * Fix test * Don't create instances during inference if no points were found (#1073) * Don't create instances during inference if no points were found * Add points check for all predictors * Fix single instance predictor logic and test * Add tests for all predictors Co-authored-by: roomrys <38435167+roomrys@users.noreply.github.com> * Add one-line fix to VideoWriterSkyvideo (#1082) * Fix parser for sleap-export (#1085) * Refactor commands to load project as `AppCommand`s (#1098) * Add working Proof of Concept * Create command class for loading project * Split `LoadProjectFile` as a subclass of `LoadLabelsObject` * Reroute last existing reference * Remove debugging code * Flexibly resize input layer of `tf.keras.Model` upon loading trained model (#1084) * Add initial implementation (auto output stride problematic) * Add to load_predictor test (error when auto-compute output stride) * Use output stride from config instead of auto-computing * Fix output-stride/padding modulo error and do not resize on export * Fix resizing bug in multi-class predictors * Non-functional clean-up * Rename new input layer to original name * Add inference integration test * Minimize config surgery, generalize layer iteration Co-authored-by: Talmo Pereira <talmo@salk.edu> * Add Option to Make Trail Shade Darker/Lighter (#1103) * Make trails 60% darker than track color * Add menu option for shade of trails * Remove unexpected indent (fat-fingered) * Create signal that updates plot instead of removing and replotting items (#1134) * Create signal that updates plot instead of redrawing * Remove debug code * Non-functional self-review changes * Fix symmetric skeletons (via table input) (#1136) Ensure variable initialized before calling it * Nix export of tracking results (#1068) * [io] export tracking results to NIX file * [io] nix added to export filter only if available * [nixio] refactor, add scores link data as mtag * [nixio] speeding up export by chunked writing * [nixio] rename point score to node score * [nixio] fix missing dimension descriptor for node scores * [export analysis] support multiple formats also for bulk export * [nixio] export centroid, some documentation * [nixio] fix double dot before filename suffix * [nixio] fix bug when not all nodes were found * [nixio] housekeeping * [nix] add nix analyis output format to convert * [nix] tiny fix, catch file write error and properly close file * [inference] main takes optional args. Can be imported to run inference form scripts * [convert] simplify if else structure and outfile handling for analysis export * [nix] use pathlib instead of os * [nix] catch if there are instances with a None frame_idx ... not sure why this occurred. The nix adaptor cannot save instances that are not related to a frame. * [nix] move checks to top of write function * [nix] use absolute imports * [nix] use black to reformat * [commands] revert qtpy import and apply code style * [convert] use absolute imports, apply code style * [commands]fix imports * [inference/nix]fix linter complaint, adjust nix types for scores * [nix] add test case for nix export format * [nix] extended testing, some modifications of adaptor * [skeleton] add __eq__ to Skeleton ... make Node.name and Node.weight instance variables instead of class variables * [nix] add nixio to requirements, remove unused nix_available, ... allow for non-unique entries in node, track and skeleton. Extend node map to store the skeleton it is part of * [nix] make the linter happy * [Node] force definition of a name Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [nix] use getattr for getting grayscale information Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [nix] fixes according to review * [convert] break out of loop upon finding the video Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [commands.py] use pathilb instead of splitting filename Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [dev requirements] remove linebreak at last line * [skeleton] revert attribute creation back to original * [nix] break lines in class documentation * Ensure all file references are closed * Make the linter happy * Add tests for ExportAnalysis and (docs for) sleap-convert Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Fix body vs symmetry subgraph filtering (#1142) Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Handle changing backbones in training editor GUI (#1140) Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Added scaling functionality for both the instances and bounding box. (#1133) * Create VisibleBoundingBox class. * Added instance scaling functionality in addition to bounding box scaling functionality. * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Added new testing for scaling operation and simplified VisibleBoundingBox class code. * Added type hinting to the scaling padding and removed erroneous bounding rect initialization. Co-authored-by: Talmo Pereira <talmo@salk.edu> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Add better error message for top down (#1121) * Add better error message for top down * Add test for error message * Raise different error, fix test * Hotfix for video save #1098 (#1148) * Add a hotfix for #1098 * WIP: Add test for detecting changes on load * Finialize change on load test * Remove unused imports * Skip test if on windows since files are being used in parallel * Add central padding to SizeMatcher (#1129) * add center padding to size matcher * add test for center padding * add ensure_float option to inference layer * reformat resizing and test_resizing * Remove redundant operation * Replace existing constants with fixtures --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Added MoveNet as an external model reference (#1141) * add center padding to size matcher * add test for center padding * add ensure_float option to inference layer * reformat resizing and test_resizing * add MoveNet as an external model inference * add the movenet to the from_model_paths * add tests * add comments to movenet predictor * add tensorflow_hub to the requirements.txt * modified default video path * resolved most of the comments except expanding the predictor * expanded Predictor.from_model_paths function to include any pre-trained models. * add test_load_model * added from_trained_models in class Predictor and modified test_load_model for it. * modified test_load_model to be more generalized. * moved pretrained model from Predictor.from_trained_model to Predictor.from_model_paths and added a test for it. * Fix Predictor.from_model_paths and tests * Rename load_movenet_model to make_model_movenet * minor clean-up * Remove redundant operation * Replace existing constants with fixtures * Handle loading movenet models via load_model API * Clean-up doc strings --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Resumable Training (#1130) * add resume training functionality * add testing function for resume training functionality * linting black * Resumable Training 2 - CLI Options (#1131) * add cli options for resumable training * add test for cli resume training * black linting for cli resumable training * simplify resumable checkpoint CLI fn to a single CLI arg (#1132) * simplify resumable checkpoint CLI fn to a single CLI arg * Adam/resumable training 3 (#1150) * correct path of labels_path for test_training * add resume training to gui * add train from scratch message * Add finishing touches to resumable training PR (#1150) (#1168) * Refactor/update 'use trained' and 'resume training' checkbox logic * Simplify checkbox logic and reset model field when resume training * Reset checkboxes upon changing config selection * Handle case for updating TrainingEditor when sender is not a checkbox * Add complete state space GUI test for checkboxes * Finish combobox test * Test that form is reset * Remove straggling TODO --------- Co-authored-by: roomrys <38435167+roomrys@users.noreply.github.com> Co-authored-by: jimzers <jimzersml@gmail.com> --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Return trainer from sleap-train and check that trainer configured correctly * Add CLI documentation for website --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Small (final?) revisions and fix test * Revert changes to fixture --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * GenericTableModel/View improvements (#1163) * [dataviews] GenericTableModel/View improvements ... * GenericTableView got a new argument specifying whether the ellipsis for long cell content should be right (old behavior, default) or left useful for long content such as the filenames in the video table. * GenericTableView uses all the space that available to the table. * The model's data function returns the full cell content to be shown as tool tip text. * [gui/app] set the ellipsis to be on the left for long table contents --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Add Skeleton Templates (#1122) * Update docs: change 'M1' to 'Apple Silicon' (#1188) * Bump to 1.3.0a0 (#1189) --------- Co-authored-by: sheridana <asheridan@salk.edu> Co-authored-by: getzze <getzze@gmail.com> Co-authored-by: Talmo Pereira <talmo@salk.edu> Co-authored-by: Jan Grewe <jngrewe@googlemail.com> Co-authored-by: Sean Afshar <84047864+sean-afshar@users.noreply.github.com> Co-authored-by: Jiaying Hsu <72051972+jiayinghsu@users.noreply.github.com> Co-authored-by: Adam Lee <38634441+jimzers@users.noreply.github.com> Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Jan Grewe <jan.grewe@g-node.org> Co-authored-by: Aaditya Prasad <78439225+aaprasad@users.noreply.github.com>

aaprasad added 4 commits January 17, 2023 13:06

Initial Commit.

7f991a4

Added `skeletons` folder containing json for each skeleton from [reference dataset](https://sleap.ai/datasets.html). Changed `name` section to corresponding dataset name

Added initial attempt at skeleton template box

2524e57

fixed fly32 name attribute to correctly match dataset name

Got MVP working:

3b63210

Added dropdown functionality to select from preset templates or custom .json file Changed load button to load template or custom depending on selection

Added Qlabel for templates dropdown

3a2e6cc

added test for template loading

4610ebc

fixed assert function in `test_OpenSkeleton` to not depend on order of `skeleton.nodes` thru comparing set diff

aaprasad marked this pull request as ready for review January 18, 2023 20:55

aaprasad requested review from roomrys and talmo January 18, 2023 20:55

roomrys requested changes Jan 18, 2023

View reviewed changes

tests/gui/test_commands.py Outdated Show resolved Hide resolved

tests/gui/test_commands.py Outdated Show resolved Hide resolved

sleap/gui/commands.py Outdated Show resolved Hide resolved

aaprasad added 2 commits January 18, 2023 13:57

removed print statements and debugging stuff

1a026a9

Fixed style issues

aa608ab

aaprasad requested a review from roomrys January 18, 2023 22:04

roomrys mentioned this pull request Feb 1, 2023

Add comparison table for replacing skeleton #1160

Merged

11 tasks

Aadi/add skeleton template pr 2 (#1145)

2ca8bb6

roomrys requested changes Feb 16, 2023

View reviewed changes

sleap/gui/app.py Outdated Show resolved Hide resolved

sleap/gui/app.py Outdated Show resolved Hide resolved

sleap/util.py Outdated Show resolved Hide resolved

tests/nn/test_viz.py Outdated Show resolved Hide resolved

tests/gui/test_dialogs.py Show resolved Hide resolved

roomrys added 5 commits February 16, 2023 09:36

Merge branch 'develop' into aadi/add-skeleton-template

eda4c5e

Add test from replace skeleton table when no add/delete nodes

2aa3284

Fix docstring in test_viz

0c3a2de

Reorganize imports and fix docstrings in util.py

db68e72

Remove QGroupBox for node/edge tables

d8c1a16

Nit: docstring fix

6b09833

roomrys reviewed Feb 16, 2023

View reviewed changes

sleap/skeleton.py Outdated Show resolved Hide resolved

sleap/nn/viz.py Outdated Show resolved Hide resolved

Last of the nits

2a15615

roomrys approved these changes Feb 16, 2023

View reviewed changes

roomrys added 2 commits February 21, 2023 14:54

Reorganize skeleton dialog

25f913f

Remove extra callbacks

02327b4

roomrys added 7 commits February 21, 2023 18:09

Add collapsible group box

900d1ce

Create MVP of collapsible widget

feab260

Clean-up collapsible widget class

491d726

Fix arrow/content initialization

3e1a795

Style collapsible widget

f77f802

Final? clean-up for collapsible widget

76d5fef

Keep combo closed when toggling visibility

e186d8c

roomrys approved these changes Feb 22, 2023

View reviewed changes

talmo approved these changes Feb 23, 2023

View reviewed changes

Merge branch 'develop' into aadi/add-skeleton-template

3c84be4

roomrys merged commit 8696aa5 into develop Feb 23, 2023

roomrys deleted the aadi/add-skeleton-template branch February 23, 2023 17:32

roomrys mentioned this pull request Feb 28, 2023

Fix typo in Skeleton.__repr__ #1200

Merged

11 tasks

roomrys mentioned this pull request Mar 14, 2023

Ensure data format compatibility #1222

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Skeleton Templates #1122

Add Skeleton Templates #1122

aaprasad commented Jan 17, 2023

codecov bot commented Jan 17, 2023 •

edited

Loading

roomrys left a comment

talmo commented Jan 26, 2023

roomrys commented Feb 16, 2023

roomrys Feb 16, 2023

roomrys left a comment

roomrys commented Feb 22, 2023

talmo left a comment

Add Skeleton Templates #1122

Add Skeleton Templates #1122

Conversation

aaprasad commented Jan 17, 2023

Description

Types of changes

Does this address any currently open issues?

Outside contributors checklist

Thank you for contributing to SLEAP!

codecov bot commented Jan 17, 2023 • edited Loading

Codecov Report

roomrys left a comment

Choose a reason for hiding this comment

talmo commented Jan 26, 2023

roomrys commented Feb 16, 2023

roomrys Feb 16, 2023

Choose a reason for hiding this comment

roomrys left a comment

Choose a reason for hiding this comment

roomrys commented Feb 22, 2023

The new skeleton dialog:

Open

Collapsed

talmo left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 17, 2023 •

edited

Loading