Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamed Volumes #52

Merged
merged 16 commits into from
Jul 3, 2019
Merged

Streamed Volumes #52

merged 16 commits into from
Jul 3, 2019

Conversation

Enet4
Copy link
Owner

@Enet4 Enet4 commented Jun 27, 2019

This gathers a bunch of additions and changes to the crate so as to provide slice-by-slice reading of volumes. (#44)

A summary of all changes below:

  • Split NiftiVolume trait to have a separate random access API: RandomAccessNiftiVolume
  • Added Dim and Idx type definitions for containing a NIfTI-compliant volume shape and index to volume, provides a nicer abstraction than working with the 8-element array directly.
  • New volume::streamed module contains a volume implementation which reads the voxel data slice by slice.
  • Deprecated InMemNiftiVolume::new_from_stream and all other from_stream associated functions in favor of from_reader, which is more idiomatic and does not conflict with the futures concept of stream
  • Made GenericNiftiObject. InMemoryNiftiObject and StreamedNiftiObject are now aliases to this type.
  • Added information to NiftiError::IncompatibleLength

Enet4 added 10 commits June 27, 2019 16:15
- split NiftiVolume trait to have a separate random access API
- `Dim` construct provides nice and efficient volume shape access
- volume::streamed module containing volumes read slice by slice
- deprecate InMemNiftiVolume::new_from_stream in favor of from_reader
- make Nifti object impl more generic, use type alias to retain API
- provide nifti object for streamed volumes
- add more info to NiftiError::IncompatibleLength
- Improve volume::streamed module documentation
- rename methods in NiftiObject away from "stream", "reader" is better
- deprecate old functions
- "reader" is more appropriate here,
  and prevents confusion with async streams
- move Dim to new module volume::shape
- add Idx to volume::shape (ndimensional index)
- add `indexed` method to streamed volume

- document some methods in util
Copy link
Collaborator

@nilgoyette nilgoyette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting work. Thank you. Did you do some benchmark? Is it a lot slower but necessary on huge file, or as fast and we should use it whenever we can?

src/volume/inmem.rs Outdated Show resolved Hide resolved
src/volume/lazy.rs Outdated Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
src/util.rs Show resolved Hide resolved
src/volume/streamed.rs Show resolved Hide resolved
tests/volume.rs Show resolved Hide resolved
Enet4 added 4 commits June 28, 2019 15:43
staged by mistake
- split FromSource trait to have options invariant to source
- add "_rank" variants of streamed object construction
- move common implementations to generic impl scope
- rustfmt
@Enet4
Copy link
Owner Author

Enet4 commented Jul 1, 2019

Thanks for the initial review, @nilgoyette. A few more changes were pushed just now, some of which reduce redundancy. The subtle differences in each kind of volume make it pretty hard to merge common routines.

I haven't benchmarked this yet, but it would be nice to see some measurements on large volumes. The real benefit comes from using much less memory, at the cost of potentially a bit of extra maintenance overhead (plus more allocations if the idiomatic iterator is used instead of the streaming iterator).

Copy link
Collaborator

@nilgoyette nilgoyette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only minor details, good job.

tests/volume.rs Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
tests/volume.rs Outdated Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
src/object.rs Outdated Show resolved Hide resolved
Enet4 added 2 commits July 2, 2019 22:55
- remove redundant commented code
- reduce redundancy in GenericNiftiObject::from_reader
- rustfmt tests/volume
@Enet4 Enet4 merged commit d29371d into master Jul 3, 2019
@Enet4 Enet4 deleted the new/streamed branch July 3, 2019 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants