Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SliceInferer may need to handle conditional-based models #8220

Open
SanBast opened this issue Nov 19, 2024 · 0 comments
Open

SliceInferer may need to handle conditional-based models #8220

SanBast opened this issue Nov 19, 2024 · 0 comments
Labels

Comments

@SanBast
Copy link

SanBast commented Nov 19, 2024

Is your feature request related to a problem? Please describe.
Lately I'm working with LDMs to generate samples conditioned by a specific conditional vector. My model learned to generate samples in an axial-2D fashion. The conditioning embeddings are based on 2D axial slices as well. However, during inference I need to work with entire volumes, so I need to find something to do a proper aggregation of 2D generated slices. Following the tutorial here, I noticed that SliceInferer might come in handy. However, as the documentation suggests, you can just tweak a little bit the network's method you will use for inference, and not the external conditions. What's happening in the SliceInferer is, as the name suggests, passing the input slice-by-slice to the considered network, yet the condition is not processed in the same way.

Describe the solution you'd like
It would be nice to adjust the code to allow models' conditioning. An idea can be to add a "forced" parameter in the SliceInferer (e.g., "condition") that may call another SliceWindowInferer dedicated to the condition.

Additional context
Given the number of axial slices Z and a bottleneck output that matches of my network that matches the dimensionality D of the conditional vector, the main error behind using SliceInferer is that the input model expects a [1, D] but what it got is a [Z,D] of course.

Thanks for helping me out!

@KumoLiu KumoLiu added enhancement New feature or request Contribution wanted labels Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants