Rough Accelerate implementation, and updated finetuning colab

[korakoe]

dont mind the commit messages regarding the colab

[yl4579]

Can you make sure you do not replace the original files, because it won't work for multi-GPU? Just add a new file called train_finetune_accelerate.py would be fine.

[korakoe]

Can you make sure you do not replace the original files, because it won't work for multi-GPU? Just add a new file called train_finetune_accelerate.py would be fine.

Sure, will do

[korakoe]

@yl4579 done

[yl4579]

I think the current state is too messy that I can't fix it. Can you just submit a PR with only train_finetune_accelerate.py? I can do that too but I want you to be credited.

[korakoe]

I think the current state is too messy that I can't fix it. Can you just submit a PR with only train_finetune_accelerate.py? I can do that too but I want you to be credited.

will do

[korakoe]

@yl4579 feel free to add it yourself, Trying to pr just that one file is proving difficult right now

[yl4579]

@korakoe Thanks I have credited you in README.

[vishalsantoshi]

Thank you for doing this. Running on a 4 GPU ( p3.8xlarge AWS ) , both this script and the non single GPU, throws this error.

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/StyleTTS2/Colab/StyleTTS2/Modules/hifigan.py", line 458, in forward
    F0 = self.F0_conv(F0_curve.unsqueeze(1))
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1128, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 302, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 298, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [1, 1, 3], expected input[1, 199, 1] to have 1 channels, but got 199 channels instead ```

[yl4579]

@vishalsantoshi See #126

[vishalsantoshi]

@yl4579 thanks, saw the linked issue. Please note that the batch_size was 2 and the the batch_percentage was not touched ( remains 0.5 ). The same setup works with a single GPU instance. This is straight up execution of the tuning notebook. Will test it again and confirm.

config['batch_size'] = 2 # not enough RAM
config['max_len'] = 100 # not enough RAM

The config

{'log_dir': 'Models/LJSpeech',
 'save_freq': 5,
 'log_interval': 10,
 'device': 'cuda',
 'epochs': 50,
 'batch_size': 2,
 'max_len': 800,
 'pretrained_model': 'Models/LibriTTS/epochs_2nd_00020.pth',
 'second_stage_load_pretrained': True,
 'load_only_params': True,
 'F0_path': 'Utils/JDC/bst.t7',
 'ASR_config': 'Utils/ASR/config.yml',
 'ASR_path': 'Utils/ASR/epoch_00080.pth',
 'PLBERT_dir': 'Utils/PLBERT/',
 'data_params': {'train_data': 'Data/train_list.txt',
  'val_data': 'Data/val_list.txt',
  'root_path': 'Data/wavs',
  'OOD_data': 'Data/OOD_texts.txt',
  'min_length': 50},
 'preprocess_params': {'sr': 24000,
  'spect_params': {'n_fft': 2048, 'win_length': 1200, 'hop_length': 300}},
 'model_params': {'multispeaker': True,
  'dim_in': 64,
  'hidden_dim': 512,
  'max_conv_dim': 512,
  'n_layer': 3,
  'n_mels': 80,
  'n_token': 178,
  'max_dur': 50,
  'style_dim': 128,
  'dropout': 0.2,
  'decoder': {'type': 'hifigan',
   'resblock_kernel_sizes': [3, 7, 11],
   'upsample_rates': [10, 5, 3, 2],
   'upsample_initial_channel': 512,
   'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]],
   'upsample_kernel_sizes': [20, 10, 6, 4]},
  'slm': {'model': 'microsoft/wavlm-base-plus',
   'sr': 16000,
   'hidden': 768,
   'nlayers': 13,
   'initial_channel': 64},
  'diffusion': {'embedding_mask_proba': 0.1,
   'transformer': {'num_layers': 3,
    'num_heads': 8,
    'head_features': 64,
    'multiplier': 2},
   'dist': {'sigma_data': 0.2,
    'estimate_sigma_data': True,
    'mean': -3.0,
    'std': 1.0}}},
 'loss_params': {'lambda_mel': 5.0,
  'lambda_gen': 1.0,
  'lambda_slm': 1.0,
  'lambda_mono': 1.0,
  'lambda_s2s': 1.0,
  'lambda_F0': 1.0,
  'lambda_norm': 1.0,
  'lambda_dur': 1.0,
  'lambda_ce': 20.0,
  'lambda_sty': 1.0,
  'lambda_diff': 1.0,
  'diff_epoch': 10,
  'joint_epoch': 30},
 'optimizer_params': {'lr': 0.0001, 'bert_lr': 1e-05, 'ft_lr': 0.0001},
 'slmadv_params': {'min_len': 400,
  'max_len': 500,
  'batch_percentage': 0.5,
  'iter': 10,
  'thresh': 5,
  'scale': 0.01,
  'sig': 1.5}}

The nvidia -smi is that helps

Wed Dec  6 19:11:11 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   37C    P0    50W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   32C    P0    50W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   32C    P0    52W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   37C    P0    56W / 300W |      0MiB / 16160MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                             
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

SOLVED: The batches should be >=2, based on this math number of GPUs/ batch_size >=2 as mentioned in in #81. Was missing each GPU assertion.

[martinambrus]

I won't pretend I know anything about this kind of acceleration - but since this finetuning script can now be run through accelerate, saving memory and time - and the original 2nd stage one can't... wouldn't something like this work on 2nd stage training file as well for people with a single GPU?

[martinambrus]

not sure whether comments from closed pulls give you guys notifications, so pardon me for mentioning you explicitly - @yl4579 @korakoe

[yl4579]

@martinambrus Yes it works too, but since it is too slow to train it from scratch using a single GPU, I don't think it's worth implementing.

[korakoe]

@martinambrus Yes it works too, but since it is too slow to train it from scratch using a single GPU, I don't think it's worth implementing.

it might resolve the DDP issue if done properly, but I believe someone ended up fixing the DDP. here's the message they sent me as I don't have access to their fork

Basically, I place operators write to memory that has to be shared amongst all processes
So if they are all written at once
By different things and synced
Causes problems

not sure how useful this ends up being

[martinambrus]

@yl4579 if you understand the comment from @korakoe, do you think this could solve the DDP issue with stage 2 as well?