how to get mixed precision to work (on 16gb *4080)

[rikabi89]

(StyleTTS2) H:\StyleTTS2>accelerate launch --mixed_precision=fp16 train_first.py --config_path ./Configs/config_ft.yml The following values were not passed to accelerate launchand had defaults used instead:--num_processeswas set to a value of1 --num_machineswas set to a value of1 --dynamo_backendwas set to a value of'no'To avoid this warning pass in values for each of the problematic parameters or runaccelerate config`.
Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init_.py", line 116, in getattr
return object.getattribute(self, k)
AttributeError: 'Munch' object has no attribute 'TMA_epoch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init_.py", line 119, in getattr
return self[k]
KeyError: 'TMA_epoch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_first.py", line 444, in
main()
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 783, in invoke
return _callback(*args, **kwargs)
File "train_first.py", line 131, in main
TMA_epoch = loss_params.TMA_epoch
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init.py", line 121, in getattr
raise AttributeError(k)
AttributeError: TMA_epoch
Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "H:\anaconda3\envs\StyleTTS2\Scripts\accelerate.exe_main.py", line 7, in
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
args.func(args)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 941, in launch_command
simple_launcher(args)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 603, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['H:\anaconda3\envs\StyleTTS2\python.exe', 'train_first.py', '--config_path', './Configs/config_ft.yml']' returned non-zero exit status 1.`

So it works without mixed precision but its extremely slow, in fact 1 epoch didnt even finish after 6 hours but this was a big dataset of around 2hours of wavs. Anything I can do optimise vram other than reduce batch and max_lens size? Win11 Anaconda

[rikabi89]

Also I tried FP8 and got :

(StyleTTS2) H:\StyleTTS2>accelerate launch --mixed_precision=fp8 train_first.py --config_path ./Configs/config_ft1.yml
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `1`
        `--num_machines` was set to a value of `1`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
Traceback (most recent call last):
  File "train_first.py", line 445, in <module>
    main()
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "train_first.py", line 50, in main
    accelerator = Accelerator(project_dir=log_dir, split_batches=True, kwargs_handlers=[ddp_kwargs])
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\accelerator.py", line 353, in __init__
    **kwargs,
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\state.py", line 691, in __init__
    raise ValueError("Using `fp8` precision requires `transformer_engine` to be installed.")
ValueError: Using `fp8` precision requires `transformer_engine` to be installed.
Traceback (most recent call last):
  File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "H:\anaconda3\envs\StyleTTS2\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
    args.func(args)
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 941, in launch_command
    simple_launcher(args)
  File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 603, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['H:\\anaconda3\\envs\\StyleTTS2\\python.exe', 'train_first.py', '--config_path', './Configs/config_ft1.yml']' returned non-zero exit status 1.

But I assume this does not work on Windows

[yl4579]

I think the error you got was not related to mixed precision, because it’s more like issues of Windows. Have you checked #81 for tips on Windows?

[rikabi89]

I think the error you got was not related to mixed precision, because it’s more like issues of Windows. Have you checked #81 for tips on Windows?

Hi yl4579,

Yes I had a look at this helpful thread. #81 and it helped me start fine-tuning using the standard command : python train_finetune.py --config_path ./Configs/config_ft.yml and setting set PYTHONUTF8=1

But it is mega slow. I was not able finish 1 epoch in 6 hours on a 2hr dataset.

So when I include : accelerate launch --mixed_precision=fp16 train_first.py --config_path ./Configs/config_ft.yml this is when I get the above errors. With out MP it does work techically but extremely slow and not viable while using full VRAM at 16gb.

[yl4579]

I think you did it wrong. You can’t finetune with train_first.py, and currently finetuning doesn’t support accelerate due to #7

[rikabi89]

ah ok :/

So this is not correct as well?

I run : accelerate launch --mixed_precision=fp8 train_finetune.py --config_path ./Configs/config_ft.yml

[yl4579]

No you can’t finetune using train_first.py. Since train_finetune.py uses DP, I think implementing mixed precision using native pytorch library should work. If someone can help with it it’d be great.

[yl4579]

@rikabi89 wait, I think you can actually change the DP to accelerate if you only have one GPU and do mixed_precision. You only need to do these changes: https://huggingface.co/docs/accelerate/package_reference/accelerator

[rikabi89]

@rikabi89 wait, I think you can actually change the DP to accelerate if you only have one GPU and do mixed_precision. You only need to do these changes: https://huggingface.co/docs/accelerate/package_reference/accelerator

I have no idea how to imeplent this unfortunately. But I saw your other post, hopefully someone knowledgeable can sort it.