Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FP32 fallback support on sd_vae_approx #14046

Merged

Commits on Nov 20, 2023

  1. Add FP32 fallback support on sd_vae_approx

    This tries to execute interpolate with FP32 if it failed.
    
    Background is that
    on some environment such as Mx chip MacOS devices, we get error as follows:
    
    ```
    "torch/nn/functional.py", line 3931, in interpolate
            return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        RuntimeError: "upsample_nearest2d_channels_last" not implemented for 'Half'
    ```
    
    In this case, ```--no-half``` doesn't help to solve. Therefore this commits add the FP32 fallback execution to solve it.
    
    Note that the submodule may require additional modifications. The following is the example modification on the other submodule.
    
    ```repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/openaimodel.py
    
    class Upsample(nn.Module):
    ..snip..
        def forward(self, x):
            assert x.shape[1] == self.channels
            if self.dims == 3:
                x = F.interpolate(
                    x, (x.shape[2], x.shape[3] * 2, x.shape[4] * 2), mode="nearest"
                )
            else:
                try:
                    x = F.interpolate(x, scale_factor=2, mode="nearest")
                except:
                    x = F.interpolate(x.to(th.float32), scale_factor=2, mode="nearest").to(x.dtype)
            if self.use_conv:
                x = self.conv(x)
            return x
    ..snip..
    ```
    
    You can see the FP32 fallback execution as same as sd_vae_approx.py.
    hidenorly committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    58c1954 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. Revert "Add FP32 fallback support on sd_vae_approx"

    This reverts commit 58c1954.
    Since the modification is expected to move to mac_specific.py
    (AUTOMATIC1111#14046 (comment))
    hidenorly committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    39eae9f View commit details
    Browse the repository at this point in the history
  2. Add FP32 fallback support on torch.nn.functional.interpolate

    This tries to execute interpolate with FP32 if it failed.
    
    Background is that
    on some environment such as Mx chip MacOS devices, we get error as follows:
    
    ```
    "torch/nn/functional.py", line 3931, in interpolate
            return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        RuntimeError: "upsample_nearest2d_channels_last" not implemented for 'Half'
    ```
    
    In this case, ```--no-half``` doesn't help to solve. Therefore this commits add the FP32 fallback execution to solve it.
    
    Note that the ```upsample_nearest2d``` is called from ```torch.nn.functional.interpolate```.
    And the fallback for torch.nn.functional.interpolate is necessary at
    ```modules/sd_vae_approx.py``` 's ```VAEApprox.forward```
    ```repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/openaimodel.py``` 's ```Upsample.forward```
    hidenorly committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    a0096c5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    81c0072 View commit details
    Browse the repository at this point in the history