Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--use-sage-attention flag causing CUDA illegal memory access #6125

Open
EthicalCell opened this issue Dec 19, 2024 · 10 comments
Open

--use-sage-attention flag causing CUDA illegal memory access #6125

EthicalCell opened this issue Dec 19, 2024 · 10 comments
Labels
Potential Bug User is reporting a bug. This should be tested.

Comments

@EthicalCell
Copy link

Expected Behavior

Expected Hunyuan video output, but ComfyUI crashes instead.

Actual Behavior

Hunyuan Wrapper from Kijai working fine, but for some reason sageattention isn't working correctly with comfyUI

Steps to Reproduce

Runpod RTX4090 with python 3.11/cu124 template:
Install torch for cu124
Install sage attention 2.0.0
start ComfyUI using "python main.py --use-sage-attention"
Use the ComfyUI Example workflow
ComfyUI crashes at "SamplerCustomAdvanced" node
Screenshot 2024-12-19 at 6 06 48 PM

Debug Logs

/workspace/ComfyUI/sageattention/sageattention/core.py:631: UserWarning: pv_accum_dtype is 'fp32+fp32', smooth_v will be ignored.
  warnings.warn("pv_accum_dtype is 'fp32+fp32', smooth_v will be ignored.")
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x77fefe0b9446 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x77fefe0636e4 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x77fefe1a5a18 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x5e460 (0x77fefe1ab460 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x60184 (0x77fefe1ad184 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x3847bdc (0x77feb6247bdc in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0x389a4c3 (0x77feb629a4c3 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x389ced4 (0x77feb629ced4 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #8: <unknown function> + 0x35e696c (0x77feb5fe696c in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #9: <unknown function> + 0x35e6a3d (0x77feb5fe6a3d in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #10: at::_ops::addmm::call(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::Scalar const&, c10::Scalar const&) + 0x19e (0x77fee9f65efe in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x1d46803 (0x77fee9746803 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::native::linear(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&) + 0x6d3 (0x77fee97476a3 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2fed243 (0x77feea9ed243 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::_ops::linear::call(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&) + 0x18c (0x77fee9f4e05c in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x75d5ac (0x77fefcf5d5ac in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: python() [0x55563b]
<omitting python frames>
frame #19: python() [0x5851f0]
frame #21: python() [0x5851f0]
frame #23: python() [0x5851f0]
frame #28: python() [0x659fdb]
frame #31: python() [0x585af7]
frame #32: python() [0x5852de]
frame #35: python() [0x585af7]
frame #36: python() [0x5852de]
frame #42: python() [0x659fdb]
frame #45: python() [0x585af7]
frame #46: python() [0x5852de]
frame #49: python() [0x585af7]
frame #50: python() [0x5852de]
frame #56: python() [0x659fdb]
frame #59: python() [0x585af7]
frame #60: python() [0x5852de]

Aborted (core dumped)

Other

No response
@EthicalCell EthicalCell added the Potential Bug User is reporting a bug. This should be tested. label Dec 19, 2024
@comfyanonymous
Copy link
Owner

What's your GPU? can you post the full log?

@EthicalCell
Copy link
Author

On RunPod RTX4090, full log:

/workspace/ComfyUI# python main.py --use-sage-attention --port 4444
[START] Security scan
[DONE] Security scan

ComfyUI-Manager: installing dependencies done.

** ComfyUI startup time: 2024-12-20 00:31:07.073952
** Platform: Linux
** Python version: 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 11.4.0]
** Python executable: /workspace/ComfyUI/venv/bin/python
** ComfyUI Path: /workspace/ComfyUI
** Log path: /workspace/ComfyUI/comfyui.log

Prestartup times for custom nodes:
0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
1.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 24118 MB, total RAM 1031845 MB
pytorch version: 2.5.1+cu124
xformers version: 0.0.28.post3
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using sage attention
[Prompt Server] web root: /workspace/ComfyUI/web

Loading: ComfyUI-Manager (V2.55.5)

ComfyUI Version: v0.3.8-4-g3cacd3f | Released on '2024-12-19'

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json

Comfyroll Studio v1.76 : 175 Nodes Loaded

** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki

[Crystools INFO] Crystools version: 1.21.0
[Crystools INFO] CPU: AMD EPYC 7B13 64-Core Processor - Arch: x86_64 - OS: Linux 6.5.0-41-lowlatency
[Crystools INFO] Pynvml (Nvidia) initialized.
[Crystools INFO] GPU/s:
[Crystools INFO] 0) NVIDIA GeForce RTX 4090
[Crystools INFO] NVIDIA Driver: 555.42.02

[rgthree-comfy] Loaded 42 epic nodes. 🎉

[ComfyUI-Easy-Use] server: v1.2.6 Loaded
[ComfyUI-Easy-Use] web root: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use/web_version/v2 Loaded
Total VRAM 24118 MB, total RAM 1031845 MB
pytorch version: 2.5.1+cu124
xformers version: 0.0.28.post3
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync

Loading: ComfyUI-Impact-Pack (V8.1.2)

[Impact Pack] Wildcards loading done.
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/registry.py:4: FutureWarning: Importing from timm.models.registry is deprecated, please import via timm.models
warnings.warn(f"Importing from {name} is deprecated, please import via timm.models", FutureWarning)
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/helpers.py:7: FutureWarning: Importing from timm.models.helpers is deprecated, please import via timm.models
warnings.warn(f"Importing from {name} is deprecated, please import via timm.models", FutureWarning)

Import times for custom nodes:
0.0 seconds: /workspace/ComfyUI/custom_nodes/cg-use-everywhere
0.0 seconds: /workspace/ComfyUI/custom_nodes/Comfyui-StableSR
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyMath
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
0.0 seconds: /workspace/ComfyUI/custom_nodes/Derfuu_ComfyUI_ModdedNodes
0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-GGUF
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_essentials
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper
0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_LayerStyle
0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools
0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Impact-Pack
0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-MMAudio
0.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
0.4 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-GIMM-VFI
0.5 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_Comfyroll_CustomNodes

Starting server

To see the GUI go to: http://127.0.0.1:4444
FETCH DATA from: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json [DONE]
got prompt
Using xformers attention in VAE
Using xformers attention in VAE
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLOW
clip missing: ['text_projection.weight']
Requested to load HunyuanVideoClipModel_
loaded completely 9.5367431640625e+25 7894.8529052734375 True
Requested to load HunyuanVideo
loaded completely 9.5367431640625e+25 12227.070373535156 True
0%| | 0/20 [00:00<?, ?it/s]/workspace/ComfyUI/sageattention/sageattention/core.py:631: UserWarning: pv_accum_dtype is 'fp32+fp32', smooth_v will be ignored.
warnings.warn("pv_accum_dtype is 'fp32+fp32', smooth_v will be ignored.")
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff05236c446 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7ff0523166e4 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7ff0527d2a18 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x5e460 (0x7ff0527d8460 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x60184 (0x7ff0527da184 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x3847bdc (0x7ff00a447bdc in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #6: + 0x389a4c3 (0x7ff00a49a4c3 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x389ced4 (0x7ff00a49ced4 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #8: + 0x35e696c (0x7ff00a1e696c in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #9: + 0x35e6a3d (0x7ff00a1e6a3d in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #10: at::_ops::addmm::call(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::Scalar const&, c10::Scalar const&) + 0x19e (0x7ff03e165efe in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #11: + 0x1d46803 (0x7ff03d946803 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::native::linear(at::Tensor const&, at::Tensor const&, std::optionalat::Tensor const&) + 0x6d3 (0x7ff03d9476a3 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #13: + 0x2fed243 (0x7ff03ebed243 in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::_ops::linear::call(at::Tensor const&, at::Tensor const&, std::optionalat::Tensor const&) + 0x18c (0x7ff03e14e05c in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #15: + 0x75d5ac (0x7ff05115d5ac in /workspace/ComfyUI/venv/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: python() [0x55563b]

frame #19: python() [0x5851f0]
frame #21: python() [0x5851f0]
frame #23: python() [0x5851f0]
frame #28: python() [0x659fdb]
frame #31: python() [0x585af7]
frame #32: python() [0x5852de]
frame #35: python() [0x585af7]
frame #36: python() [0x5852de]
frame #42: python() [0x659fdb]
frame #45: python() [0x585af7]
frame #46: python() [0x5852de]
frame #49: python() [0x585af7]
frame #50: python() [0x5852de]
frame #56: python() [0x659fdb]
frame #59: python() [0x585af7]
frame #60: python() [0x5852de]

Aborted (core dumped)

@deepfree2023
Copy link

Same problem. Python crashes with --use-sage-attention every time. And no problem without it.

@comfyanonymous
Copy link
Owner

Do you get the same crash with sage attention 1?

@EthicalCell
Copy link
Author

Sageattention 1.0.6 works as expected

python main.py --use-sage-attention --port 6006
[START] Security scan
[DONE] Security scan

ComfyUI-Manager: installing dependencies. (GitPython)

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python -m pip install --upgrade pip

ComfyUI-Manager: installing dependencies done.

** ComfyUI startup time: 2024-12-20 13:50:22.608414
** Platform: Linux
** Python version: 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 11.4.0]
** Python executable: /usr/bin/python
** ComfyUI Path: /workspace/ComfyUI
** Log path: /workspace/ComfyUI/comfyui.log

Prestartup times for custom nodes:
0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
6.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager

Traceback (most recent call last):
File "/workspace/ComfyUI/main.py", line 130, in
import comfy.utils
File "/workspace/ComfyUI/comfy/utils.py", line 30, in
from einops import rearrange
ModuleNotFoundError: No module named 'einops'
root@af68b03d527b:/workspace/ComfyUI# source venv/bin/activate
(venv) root@af68b03d527b:/workspace/ComfyUI# python main.py --use-sage-attention --port 6006
[START] Security scan
[DONE] Security scan

ComfyUI-Manager: installing dependencies done.

** ComfyUI startup time: 2024-12-20 13:50:34.786604
** Platform: Linux
** Python version: 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 11.4.0]
** Python executable: /workspace/ComfyUI/venv/bin/python
** ComfyUI Path: /workspace/ComfyUI
** Log path: /workspace/ComfyUI/comfyui.log

Prestartup times for custom nodes:
0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
0.9 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 24111 MB, total RAM 176863 MB
pytorch version: 2.5.1+cu124
xformers version: 0.0.28.post3
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using sage attention
[Prompt Server] web root: /workspace/ComfyUI/web

Loading: ComfyUI-Manager (V2.55.5)

ComfyUI Version: v0.3.8-7-g418eb70 | Released on '2024-12-20'

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json

Comfyroll Studio v1.76 : 175 Nodes Loaded

** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[Crystools INFO] Crystools version: 1.21.0
[Crystools INFO] CPU: AMD EPYC 9254 24-Core Processor - Arch: x86_64 - OS: Linux 6.8.0-40-generic
[Crystools INFO] Pynvml (Nvidia) initialized.
[Crystools INFO] GPU/s:
[Crystools INFO] 0) NVIDIA GeForce RTX 4090
[Crystools INFO] NVIDIA Driver: 560.35.03

[rgthree-comfy] Loaded 42 extraordinary nodes. 🎉

[ComfyUI-Easy-Use] server: v1.2.6 Loaded
[ComfyUI-Easy-Use] web root: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use/web_version/v2 Loaded
Total VRAM 24111 MB, total RAM 176863 MB
pytorch version: 2.5.1+cu124
xformers version: 0.0.28.post3
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync

Loading: ComfyUI-Impact-Pack (V8.1.2)

[Impact Pack] Wildcards loading done.
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/registry.py:4: FutureWarning: Importing from timm.models.registry is deprecated, please import via timm.models
warnings.warn(f"Importing from {name} is deprecated, please import via timm.models", FutureWarning)
/workspace/ComfyUI/venv/lib/python3.11/site-packages/timm/models/helpers.py:7: FutureWarning: Importing from timm.models.helpers is deprecated, please import via timm.models
warnings.warn(f"Importing from {name} is deprecated, please import via timm.models", FutureWarning)

Import times for custom nodes:
0.0 seconds: /workspace/ComfyUI/custom_nodes/websocket_image_save.py
0.0 seconds: /workspace/ComfyUI/custom_nodes/cg-use-everywhere
0.0 seconds: /workspace/ComfyUI/custom_nodes/Comfyui-StableSR
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyMath
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
0.0 seconds: /workspace/ComfyUI/custom_nodes/Derfuu_ComfyUI_ModdedNodes
0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-GGUF
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_essentials
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Impact-Pack
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_LayerStyle
0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools
0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-MMAudio
0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-GIMM-VFI
0.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
0.4 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_Comfyroll_CustomNodes

Starting server

To see the GUI go to: http://127.0.0.1:6006
FETCH DATA from: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json [DONE]
got prompt
Using xformers attention in VAE
Using xformers attention in VAE
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLOW
clip missing: ['text_projection.weight']
Requested to load HunyuanVideoClipModel_
loaded completely 9.5367431640625e+25 7894.8529052734375 True
Requested to load HunyuanVideo
loaded completely 9.5367431640625e+25 12227.070373535156 True
100%|███████████████████████████████████████████| 20/20 [00:07<00:00, 2.81it/s]
Requested to load AutoencoderKL
loaded completely 9.5367431640625e+25 470.1210079193115 True
Prompt executed in 28.57 seconds

@deepfree2023
Copy link

Do you get the same crash with sage attention 1?

I'm using sage attention 2.0 too. This might be the reason?

@EthicalCell
Copy link
Author

Do you get the same crash with sage attention 1?

I'm using sage attention 2.0 too. This might be the reason?

sageattention 1 works, but sageattention2 brings speed improvements, so hoping this issue is resolvable for 2.

@deepfree2023
Copy link

deepfree2023 commented Dec 20, 2024 •
edited
Loading

Yes, sageattention 1 works, I've got a speed improvement from about 3.3s/it to 2.75s/it. hope for 2 support.

@Djanghost
Copy link

Same here, work well with the Kijai's wrapper, hope for v2 support in comfy

@GalaxyTimeMachine
Copy link

GalaxyTimeMachine commented Dec 23, 2024 •
edited
Loading

I have sageattention installed, and can create clear videos using the Kijai nodes, but if I try to use comfy nodes I always get a very blurry image. It doesn't crash or error, but I'm using the same model/clip/vae in both workflows.

Edit: I just had the same crash and error with Kijai's nodes after bumping up the number of frames from 97 to 145. Could this be a memory related issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Potential Bug User is reporting a bug. This should be tested.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@GalaxyTimeMachine @EthicalCell @comfyanonymous @Djanghost @deepfree2023