You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add an API for estimating the peak VRAM usage of a model. Ideally, this would be something simple like:
classImageModelDescriptor:
...
defestimate_peak_memory(self, size: tuple[int, int]) ->int|None:
""" Returns an estimate for the peak memory usage (VRAM if on the GPU) of the model for the given image size. If no estimate is available, `None` will be returned. """
This would allow us to use better VRAM estimates in chaiNNer, which would allow us to pick better tile sizes.
As for actually finding good estimates. I think we could use a script to determine an estimate automatically. The script would be given an architecture, possible hyperparameters, and a series of image size to test. It would then simply run a model on a random image and measure the peak VRAM usage for all combinations of images sizes and hyperparameters. This gives us some data points to work with. The rest is a curve-fitting exercise, and generating python code for what we could out.
With this script in place, we can just run it and automatically get a good function for estimating the VRAM usage of an architecture.
The text was updated successfully, but these errors were encountered:
Add an API for estimating the peak VRAM usage of a model. Ideally, this would be something simple like:
This would allow us to use better VRAM estimates in chaiNNer, which would allow us to pick better tile sizes.
As for actually finding good estimates. I think we could use a script to determine an estimate automatically. The script would be given an architecture, possible hyperparameters, and a series of image size to test. It would then simply run a model on a random image and measure the peak VRAM usage for all combinations of images sizes and hyperparameters. This gives us some data points to work with. The rest is a curve-fitting exercise, and generating python code for what we could out.
With this script in place, we can just run it and automatically get a good function for estimating the VRAM usage of an architecture.
The text was updated successfully, but these errors were encountered: