Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

可以支持cuda12.1吗 #25

Open
zixian-wu opened this issue Jun 27, 2024 · 4 comments
Open

可以支持cuda12.1吗 #25

zixian-wu opened this issue Jun 27, 2024 · 4 comments

Comments

@zixian-wu
Copy link

No description provided.

@MyNiuuu
Copy link
Owner

MyNiuuu commented Jul 2, 2024

My CUDA version is 12.1.

1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.

To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.

Now, the program should run correctly.

Originally posted by @laishujie in #16 (comment)

抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？

@zishen-ucap
Copy link

My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.

Originally posted by @laishujie in #16 (comment)

抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？

我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。
此外，用cuda12.1执行
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1

@zishen-ucap
Copy link

My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.

Originally posted by @laishujie in #16 (comment)

抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？

我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行 pip install "git+https://github.com/facebookresearch/pytorch3d.git" 会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1

报错如下

Exception: None
You selected None at [397, 137] from image
[[[397, 137]]]
You selected None at [381, 175] from image
[[[397, 137], [381, 175]]]
torch.Size([1, 24, 2, 512, 512])
torch.Size([1, 24, 2, 512, 512])
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1721800906.491101  545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.712256  545675 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.713922  545391 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W0000 00:00:1721800906.759709  545677 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1721800906.786435  545690 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
I0000 00:00:1721800906.793379  545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.849270  545737 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.859003  545742 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
  warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
  0%|                                                                                                                | 0/25 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
    outputs = self.forward_sample(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
    val_output = self.pipeline(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
    down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
    warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
    warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype)  # [b, c, w, h]
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
    tenOut = softsplat_func.apply(tenIn, tenFlow)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
    cuda_launch(cuda_kernel('softsplat_out', '''
  File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
    return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
  File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
  File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
    return _compile_with_cache_cuda(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
    compiled_obj, mapping = prog.compile(options, log_stream)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression

Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
    outputs = self.forward_sample(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
    val_output = self.pipeline(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
    down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
    warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
    warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype)  # [b, c, w, h]
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
    tenOut = softsplat_func.apply(tenIn, tenFlow)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
    cuda_launch(cuda_kernel('softsplat_out', '''
  File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
    return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
  File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
  File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
    return _compile_with_cache_cuda(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
    compiled_obj, mapping = prog.compile(options, log_stream)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression

Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

@zishen-ucap
Copy link

My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.

Originally posted by @laishujie in #16 (comment)

抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？

我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行 pip install "git+https://github.com/facebookresearch/pytorch3d.git" 会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1

报错如下

Exception: None
You selected None at [397, 137] from image
[[[397, 137]]]
You selected None at [381, 175] from image
[[[397, 137], [381, 175]]]
torch.Size([1, 24, 2, 512, 512])
torch.Size([1, 24, 2, 512, 512])
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1721800906.491101  545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.712256  545675 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.713922  545391 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W0000 00:00:1721800906.759709  545677 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1721800906.786435  545690 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
I0000 00:00:1721800906.793379  545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.849270  545737 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.859003  545742 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
  warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
  0%|                                                                                                                | 0/25 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
    outputs = self.forward_sample(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
    val_output = self.pipeline(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
    down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
    warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
    warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype)  # [b, c, w, h]
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
    tenOut = softsplat_func.apply(tenIn, tenFlow)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
    cuda_launch(cuda_kernel('softsplat_out', '''
  File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
    return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
  File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
  File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
    return _compile_with_cache_cuda(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
    compiled_obj, mapping = prog.compile(options, log_stream)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression

Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
    outputs = self.forward_sample(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
    val_output = self.pipeline(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
    down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
    warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
    warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype)  # [b, c, w, h]
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
    tenOut = softsplat_func.apply(tenIn, tenFlow)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
    cuda_launch(cuda_kernel('softsplat_out', '''
  File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
    return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
  File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
  File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
    return _compile_with_cache_cuda(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
    compiled_obj, mapping = prog.compile(options, log_stream)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error

/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression

/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"

/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined

/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression

Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.

Originally posted by @laishujie in #16 (comment)

抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？

我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行 pip install "git+https://github.com/facebookresearch/pytorch3d.git" 会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1

我的可以参考下
我最终最终解决方法是

  1. 安装torch+cu118
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install
  1. 接着安装相关依赖
pip install -r requirements.txt
pip install opencv-python-headless
  1. 确保你此时的cuda是11.8，接着执行下面命令
    pip install "git+https://github.com/facebookresearch/pytorch3d.git"
  2. 接着卸载cupy-cuda117
    pip uninstall cupy-cuda117
  3. 切换cuda12.1后，再执行下面命令
    pip install cupy
  4. 最后修改脚本代码models/softsplat.py 第225行
    return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MyNiuuu @zixian-wu @zishen-ucap