Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary batching functionality from functions #718

Closed
4 of 6 tasks
alex-l-kong opened this issue Sep 20, 2022 · 1 comment · Fixed by #803
Closed
4 of 6 tasks

Remove unnecessary batching functionality from functions #718

alex-l-kong opened this issue Sep 20, 2022 · 1 comment · Fixed by #803
Assignees
Labels
enhancement New feature or request

Comments

@alex-l-kong
Copy link
Contributor

alex-l-kong commented Sep 20, 2022

Is your feature request related to a problem? Please describe.

We've found that in some cases, batching exists solely for the purpose of loading more image data in at once. We later loop over each FOV in that batch anyways, which doesn't add any significant optimization.

This additionally causes issues with cohorts with different image sizes, because loading a batch into a 1024x1024 or 2048x2048 array will cause dimension errors.

Describe the solution you'd like

We've already removed batching from generate_deepcell_input, the following functions also need to be modified in this regard:

  • data_utils.generate_and_save_pixel_cluster_masks (and by extension, data_utils.generate_pixel_cluster_mask)
  • data_utils.generate_and_save_cell_cluster_masks (and by extension, data_utils.label_cells_by_cluster)
  • marker_quantification.generate_cell_table
  • spatial_analysis.batch_channel_spatial_enrichment and spatial_analysis.batch_cluster_spatial_enrichment: talked to @ackagel about this, we can condense these to a per-FOV basis with negligible speed difference
  • spatial_analysis.create_neighborhood_matrix (and by extension, the neighborhood analysis notebook): currently, this process requires precomputing all the distance matrices prior to running then function, then an additional per-FOV loop during neighborhood analysis. We should condense all this down to one per-FOV loop in the neighborhood matrix function.
  • spatial_analysis_utils.calc_dist_matrix: after the neighborhood analysis process is updated, the dist matrix function will no longer be receiving a batch of FOVs to process over, so it should be condensed to process just one FOV.
@alex-l-kong alex-l-kong added the enhancement New feature or request label Sep 20, 2022
@alex-l-kong alex-l-kong self-assigned this Sep 20, 2022
@ngreenwald
Copy link
Member

Looks good, free to proceed one at a time as we discussed, or whatever feels like the right amount for one PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants