Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement compute_speed and compute_path_length #280

Merged
merged 18 commits into from
Nov 5, 2024

Conversation

niksirbi
Copy link
Member

@niksirbi niksirbi commented Aug 23, 2024

Description

What is this PR

  • Bug fix
  • Addition of a new feature
  • Other

What does this PR do?

It introduces two new functions in the kinematics module:

  • compute_speed: quite straightforward to implement, it just returns the norm of velocity, using existing movement functions
  • compute_path_length: in theory also straightforward, as it should just be the sum of the norms of displacement vectors, but I spent a lot of time debating how to handle missing values. See extensive discussion on zulip.

In the end I opted for providing two alternative nan policies, and emitting a warning if too many values are missing. Expand the dropdown below for more details.

compute_path_length() function signatures and docstring
def compute_path_length(
    data: xr.DataArray,
    start: float | None = None,
    stop: float | None = None,
    nan_policy: Literal["drop", "scale"] = "drop",
    nan_warn_threshold: float = 0.2,
) -> xr.DataArray:
    """Compute the length of a path travelled between two time points.

    The path length is defined as the sum of the norms (magnitudes) of the
    displacement vectors between two time points ``start`` and ``stop``,
    which should be provided in the time units of the data array.
    If not specified, the minimum and maximum time coordinates of the data
    array are used as start and stop times, respectively.

    Parameters
    ----------
    data : xarray.DataArray
        The input data containing position information in Cartesian
        coordinates, with ``time`` and ``space`` among the dimensions.
    start : float, optional
        The time to consider as the path's starting point. If None (default),
        the minimum time coordinate in the data is used.
    stop : float, optional
        The time to consider as the path's end point. If None (default),
        the maximum time coordinate in the data is used.
    nan_policy : Literal["drop", "scale"], optional
        Policy to handle NaN (missing) values. Can be one of the ``"drop"``
        or ``"scale"``. Defaults to ``"drop"``. See Notes for more details
        on the two policies.
    nan_warn_threshold : float, optional
        If more than this proportion of values are missing in any point track,
        a warning will be emitted. Defaults to 0.2 (20%).

    Returns
    -------
    xarray.DataArray
        An xarray DataArray containing the computed path length.
        Will have the same dimensions as the input data, except for ``time``
        and ``space`` which will be removed.

    Notes
    -----
    Choosing ``nan_policy="drop"`` will drop NaN values from each point track
    before computing path length. This equates to assuming that a track
    follows a straight line between two valid points flanking a missing
    segment. Missing segments at the beginning or end of the specified
    time range are not counted. This approach tends to underestimate
    the path length, and the error increases with the number of missing
    values.

    Choosing ``nan_policy="scale"`` will adjust the path length based on the
    the proportion of valid segments per point track. For example, if only
    80% of segments are present, the path length will be computed based on
    these and the result will be divided by 0.8. This approach assumes
    that motion dynamics are similar across present and missing time
    segments, which may not be the case.

    """

I'm in two minds as to whether we need the start and stop arguments, as the user could easily select a time range using ds.position.sel(time=slice(0, 10)), before passing the data array to compute_path_length(). I've chosen to include them for now, but I'm open to counter-arguments, CC @b-peri.

References

Closes #147

How has this PR been tested?

For speed, I used the existing kinematics tests (added "speed" as a parameter) with a bit of modification needed.

Testing for the path length was harder because:

  • this variable lacks both time and space dimensions, so modifying existing kinematics tests to account for it would b complicated
  • With the two aforementioned nan_policies, there are several tricky edge cases, and I tried to cover them all. The uniform linear motion fixture has been instrumental in helping me implement , debug and test these nan policies.
  • I also had to test that the nan warning is emitted in the right scenarios and contains specific messages.

As a result I ended up adding 3 new unit tests to account for all of the above, but suggestion to streamline the tests are welcome.

Is this a breaking change?

No.

Does this PR require an update to the documentation?

API docs should be automatically updated. We might consider mentioning the new functions in some of the examples in the future, but not necessary right now.

Checklist:

  • The code has been tested locally
  • Tests have been added to cover all new functionality
  • The documentation has been updated to reflect any changes
  • The code has been formatted with pre-commit

Copy link

Copy link

codecov bot commented Aug 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.77%. Comparing base (ca4daf2) to head (13e2326).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #280   +/-   ##
=======================================
  Coverage   99.77%   99.77%           
=======================================
  Files          14       14           
  Lines         878      907   +29     
=======================================
+ Hits          876      905   +29     
  Misses          2        2           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@niksirbi niksirbi force-pushed the speed-and-distance-traveled branch from ebae6af to 129e8c1 Compare October 8, 2024 13:09
@niksirbi niksirbi changed the title WIP: implement compute_speed and compute_distance_traveled Implement compute_speed and compute_path_length Oct 8, 2024
@niksirbi niksirbi marked this pull request as ready for review October 10, 2024 16:35
@niksirbi niksirbi requested a review from lochhh October 10, 2024 16:35
@niksirbi niksirbi force-pushed the speed-and-distance-traveled branch from 92e91f0 to d089a9c Compare October 22, 2024 11:37
Copy link
Collaborator

@lochhh lochhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @niksirbi !
To summarise:

  • a few suggestions for wordings, which you're free to take or leave as usual
  • start, stop checks: I'm wondering if we should explicitly check these for slicing. While the checks are much more restrictive than xarray, do we need to do so? xarray, for instance, automatically constraints the slicing to the min, max, regardless of whether the provided start, stop values are within or out of range. if we drop our own checks, the relevant tests can then be much simpler
  • simplify _compute_path_length_drop_nan: hacking with ffill and let xarray's groupby take care of stacking - needs further testing?
  • simplify tests (reduce parametrisation, test only id_1 values in nan test)
  • resolve merge conflicts with main branch

movement/analysis/kinematics.py Outdated Show resolved Hide resolved
movement/analysis/kinematics.py Outdated Show resolved Hide resolved
movement/analysis/kinematics.py Outdated Show resolved Hide resolved
movement/analysis/kinematics.py Outdated Show resolved Hide resolved
movement/analysis/kinematics.py Outdated Show resolved Hide resolved
tests/conftest.py Show resolved Hide resolved
tests/test_unit/test_kinematics.py Outdated Show resolved Hide resolved
tests/test_unit/test_kinematics.py Outdated Show resolved Hide resolved
tests/test_unit/test_kinematics.py Outdated Show resolved Hide resolved
tests/test_unit/test_kinematics.py Outdated Show resolved Hide resolved
@niksirbi niksirbi force-pushed the speed-and-distance-traveled branch from 8bee9b9 to 77fa873 Compare November 4, 2024 09:55
@niksirbi
Copy link
Member Author

niksirbi commented Nov 4, 2024

Thanks for the thorough review @lochhh!

  • I've accepted your suggestions for docstring wording.
  • I dropped the custom start/stop time checks, relying mostly on xarray.sel to handle this. That said, I did choose to still catch one specific edge case (see relevant comment)
  • I adopted and further simplified your ffill approach (instead of drop). It achieves the same result and is much more concise!
  • Simplified test parametrisation, following your suggestions.
  • Rebased on main and dealt with merge conflicts.

@niksirbi niksirbi requested a review from lochhh November 4, 2024 10:01
@lochhh lochhh force-pushed the speed-and-distance-traveled branch from 77fa873 to 13e2326 Compare November 5, 2024 16:54
Copy link

sonarqubecloud bot commented Nov 5, 2024

Copy link
Collaborator

@lochhh lochhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant! 🚀 Thanks for addressing all the comments!

@lochhh lochhh enabled auto-merge November 5, 2024 16:55
@lochhh lochhh added this pull request to the merge queue Nov 5, 2024
Merged via the queue into main with commit a3956c4 Nov 5, 2024
27 checks passed
@niksirbi niksirbi deleted the speed-and-distance-traveled branch November 6, 2024 07:15
niksirbi added a commit that referenced this pull request Nov 19, 2024
* implement compute_speed and compute_path_length functions

* added speed to existing kinematics unit test

* rewrote compute_path_length with various nan policies

* unit test compute_path_length across time ranges

* fixed and refactor compute_path_length and its tests

* fixed docstring for compute_path_length

* Accept suggestion on docstring wording

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Remove print statement from test

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Ensure nan report is printed

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* adapt warning message match in test

* change 'any' to 'all'

* uniform wording across path length docstrings

* (mostly) leave time range validation to xarray slice

* refactored parameters for test across time ranges

* simplified test for path lenght with nans

* replace drop policy with ffill

* remove B905 ruff rule

* make pre-commit happy

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>
niksirbi added a commit that referenced this pull request Nov 20, 2024
* initialise napari plugin development

* Create skeleton for napari plugin with collapsible widgets (#218)

* initialise napari plugin development

* initialise napari plugin development

* create  skeleton for napari plugin with collapsible widgets

* add basic widget smoke tests and allow headless testing

* do not depend on napari from pip

* include napari option in install instructions

* make meta_widget module private

* pin atlasapi version to avoid unnecessary dependencies

* pin napari >= 0.4.19 from conda-forge

* switched to pip install of napari[all]

* seperation of concerns in widget tests

* add pytest-mock dev dependency

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* Added loader widget for poses

* update widget tests

* simplify dependency on brainglobe-utils

* consistent monospace formatting for movement in public docstrings

* get rid of code that's only relevant for displaying Tracks

* enable visibility of napari layer tooltips

* renamed widget to PosesLoader

* make cmap optional in set_color_by method

* wrote unit tests for napari convert module

* wrote unit-tests for the layer styles module

* linkcheck ignore zenodo redirects

* move _sample_colormap out of PointsStyle class

* small refactoring in the loader widget

* Expand tests for loader widget

* added comments and docstrings to napari plugin tests

* refactored all napari tests into separate unit test folder

* added napari-video to dependencies

* replaced deprecated edge_width with border_width

* got rid of widget pytest fixtures

* remove duplicate word from docstring

* remove napari-video dependency

* include napari extras in docs requirements

* add test for _on_browse_clicked method

* getOpenFileName returns tuple, not str

* simplify poses_to_napari_tracks

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* [pre-commit.ci] pre-commit autoupdate (#338)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2)
- [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.13.0](pre-commit/mirrors-mypy@v1.11.2...v1.13.0)
- [github.com/mgedmin/check-manifest: 0.49 → 0.50](mgedmin/check-manifest@0.49...0.50)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Implement `compute_speed` and `compute_path_length` (#280)

* implement compute_speed and compute_path_length functions

* added speed to existing kinematics unit test

* rewrote compute_path_length with various nan policies

* unit test compute_path_length across time ranges

* fixed and refactor compute_path_length and its tests

* fixed docstring for compute_path_length

* Accept suggestion on docstring wording

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Remove print statement from test

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Ensure nan report is printed

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* adapt warning message match in test

* change 'any' to 'all'

* uniform wording across path length docstrings

* (mostly) leave time range validation to xarray slice

* refactored parameters for test across time ranges

* simplified test for path lenght with nans

* replace drop policy with ffill

* remove B905 ruff rule

* make pre-commit happy

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* avoid redefining duplicate attributes in child dataclass

* modify test case to match poses_to_napari_tracks simplification

* expected_log_messages should be a subset of captured messages

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* fix typo

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* use names for Qwidgets

* reorganised test_valid_poses_to_napari_tracks

* parametrised layer style tests

* delet integration test which was reintroduced after conflict resolution

* added test about file filters

* deleted obsolete loader widget file (had snuck back in due to conflict merging)

* combine tests for button callouts

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Simplify test_layer_style_as_kwargs

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
niksirbi added a commit that referenced this pull request Dec 16, 2024
* initialise napari plugin development

* Create skeleton for napari plugin with collapsible widgets (#218)

* initialise napari plugin development

* initialise napari plugin development

* create  skeleton for napari plugin with collapsible widgets

* add basic widget smoke tests and allow headless testing

* do not depend on napari from pip

* include napari option in install instructions

* make meta_widget module private

* pin atlasapi version to avoid unnecessary dependencies

* pin napari >= 0.4.19 from conda-forge

* switched to pip install of napari[all]

* seperation of concerns in widget tests

* add pytest-mock dev dependency

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* Added loader widget for poses

* update widget tests

* simplify dependency on brainglobe-utils

* consistent monospace formatting for movement in public docstrings

* get rid of code that's only relevant for displaying Tracks

* enable visibility of napari layer tooltips

* renamed widget to PosesLoader

* make cmap optional in set_color_by method

* wrote unit tests for napari convert module

* wrote unit-tests for the layer styles module

* linkcheck ignore zenodo redirects

* move _sample_colormap out of PointsStyle class

* small refactoring in the loader widget

* Expand tests for loader widget

* added comments and docstrings to napari plugin tests

* refactored all napari tests into separate unit test folder

* added napari-video to dependencies

* replaced deprecated edge_width with border_width

* got rid of widget pytest fixtures

* remove duplicate word from docstring

* remove napari-video dependency

* include napari extras in docs requirements

* add test for _on_browse_clicked method

* getOpenFileName returns tuple, not str

* simplify poses_to_napari_tracks

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* [pre-commit.ci] pre-commit autoupdate (#338)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2)
- [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.13.0](pre-commit/mirrors-mypy@v1.11.2...v1.13.0)
- [github.com/mgedmin/check-manifest: 0.49 → 0.50](mgedmin/check-manifest@0.49...0.50)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Implement `compute_speed` and `compute_path_length` (#280)

* implement compute_speed and compute_path_length functions

* added speed to existing kinematics unit test

* rewrote compute_path_length with various nan policies

* unit test compute_path_length across time ranges

* fixed and refactor compute_path_length and its tests

* fixed docstring for compute_path_length

* Accept suggestion on docstring wording

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Remove print statement from test

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Ensure nan report is printed

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* adapt warning message match in test

* change 'any' to 'all'

* uniform wording across path length docstrings

* (mostly) leave time range validation to xarray slice

* refactored parameters for test across time ranges

* simplified test for path lenght with nans

* replace drop policy with ffill

* remove B905 ruff rule

* make pre-commit happy

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* avoid redefining duplicate attributes in child dataclass

* modify test case to match poses_to_napari_tracks simplification

* expected_log_messages should be a subset of captured messages

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* fix typo

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* use names for Qwidgets

* reorganised test_valid_poses_to_napari_tracks

* parametrised layer style tests

* delet integration test which was reintroduced after conflict resolution

* added test about file filters

* deleted obsolete loader widget file (had snuck back in due to conflict merging)

* combine tests for button callouts

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Simplify test_layer_style_as_kwargs

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
niksirbi added a commit that referenced this pull request Dec 18, 2024
* initialise napari plugin development

* Create skeleton for napari plugin with collapsible widgets (#218)

* initialise napari plugin development

* initialise napari plugin development

* create  skeleton for napari plugin with collapsible widgets

* add basic widget smoke tests and allow headless testing

* do not depend on napari from pip

* include napari option in install instructions

* make meta_widget module private

* pin atlasapi version to avoid unnecessary dependencies

* pin napari >= 0.4.19 from conda-forge

* switched to pip install of napari[all]

* seperation of concerns in widget tests

* add pytest-mock dev dependency

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* Added loader widget for poses

* update widget tests

* simplify dependency on brainglobe-utils

* consistent monospace formatting for movement in public docstrings

* get rid of code that's only relevant for displaying Tracks

* enable visibility of napari layer tooltips

* renamed widget to PosesLoader

* make cmap optional in set_color_by method

* wrote unit tests for napari convert module

* wrote unit-tests for the layer styles module

* linkcheck ignore zenodo redirects

* move _sample_colormap out of PointsStyle class

* small refactoring in the loader widget

* Expand tests for loader widget

* added comments and docstrings to napari plugin tests

* refactored all napari tests into separate unit test folder

* added napari-video to dependencies

* replaced deprecated edge_width with border_width

* got rid of widget pytest fixtures

* remove duplicate word from docstring

* remove napari-video dependency

* include napari extras in docs requirements

* add test for _on_browse_clicked method

* getOpenFileName returns tuple, not str

* simplify poses_to_napari_tracks

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* [pre-commit.ci] pre-commit autoupdate (#338)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2)
- [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.13.0](pre-commit/mirrors-mypy@v1.11.2...v1.13.0)
- [github.com/mgedmin/check-manifest: 0.49 → 0.50](mgedmin/check-manifest@0.49...0.50)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Implement `compute_speed` and `compute_path_length` (#280)

* implement compute_speed and compute_path_length functions

* added speed to existing kinematics unit test

* rewrote compute_path_length with various nan policies

* unit test compute_path_length across time ranges

* fixed and refactor compute_path_length and its tests

* fixed docstring for compute_path_length

* Accept suggestion on docstring wording

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Remove print statement from test

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Ensure nan report is printed

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* adapt warning message match in test

* change 'any' to 'all'

* uniform wording across path length docstrings

* (mostly) leave time range validation to xarray slice

* refactored parameters for test across time ranges

* simplified test for path lenght with nans

* replace drop policy with ffill

* remove B905 ruff rule

* make pre-commit happy

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* initialise napari plugin development

* avoid redefining duplicate attributes in child dataclass

* modify test case to match poses_to_napari_tracks simplification

* expected_log_messages should be a subset of captured messages

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* fix typo

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* use names for Qwidgets

* reorganised test_valid_poses_to_napari_tracks

* parametrised layer style tests

* delet integration test which was reintroduced after conflict resolution

* added test about file filters

* deleted obsolete loader widget file (had snuck back in due to conflict merging)

* combine tests for button callouts

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

* Simplify test_layer_style_as_kwargs

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>

---------

Co-authored-by: Chang Huan Lo <changhuan.lo@ucl.ac.uk>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider adding a distance and speed property to accessor
2 participants