accelerator problem? #70

postyork · 2022-05-20T13:31:01Z

postyork
May 20, 2022

Hi all,

Forgive the potentially very noob question. I'm using a windows machine and attempting to train Rave, got all setup as recommended using conda, checked GPU, etc. Started training, and after I got to 100% I got the following message. Any ideas on how to proceed? And when I do, do I delete the massive .mdb file in the temporary folder or ...? thanks!

(rave) PS C:\Users\posty\RAVE> python train_rave.py -c default --name test1 --wav C:\Users\posty\OneDrive\Desktop\2022\ravetraining --preprocessed C:\Users\posty\OneDrive\Desktop\2022\rave\test1\rave
Recursive search in C:\Users\posty\OneDrive\Desktop\2022\ravetraining
ROLLER COASTER Small Steel Coaster MECHANICS 03 Lap Bar Restraints. Open,Clank,Metal.wav: 100%|███████████████████████████████████████████| 563/563 [29:07<00:00,  3.10s/it]
Traceback (most recent call last):
  File "C:\Users\posty\RAVE\train_rave.py", line 156, in <module>
    trainer = pl.Trainer(
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\utilities\argparse.py", line 339, in insert_env_defaults
    return fn(self, **kwargs)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 483, in __init__
    self._accelerator_connector = AcceleratorConnector(
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\connectors\accelerator_connector.py", line 196, in __init__
    self._set_parallel_devices_and_init_accelerator()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\connectors\accelerator_connector.py", line 514, in _set_parallel_devices_and_init_accelerator
    raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: GPUAccelerator can not run on your system since the accelerator is not available. The following accelerator(s) is available and can be passed into `accelerator` argument of `Trainer`: ['cpu'].

oh, and my GPU info is here (when I checked nvidia-smi)

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 471.51       Driver Version: 471.51       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   35C    P8    12W /  N/A |    164MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |

caillonantoine · 2022-05-20T13:46:37Z

caillonantoine
May 20, 2022
Collaborator

it looks like torch doesn't see your gpu... can you check the output of python -c "import torch; print(torch.cuda.is_available())" ?

3 replies

postyork May 20, 2022
Author

I get a "false" when I input that, so that does seem to be my problem!

caillonantoine May 20, 2022
Collaborator

maybe re check your torch install then

postyork May 20, 2022
Author

Thanks so much for the fast responses. I checked torch install and managed to get the response to your earlier suggestion to yield "true". I've now tried again, with the following error. Apologies if these are fairly basic questions.

(rave) PS C:\Users\posty\RAVE> python train_rave.py -c default --name test1 --wav C:\Users\posty\OneDrive\Desktop\2022\ravetraining --preprocessed C:\Users\posty\OneDrive\Desktop\2022\rave\test1\rave
Recursive search in C:\Users\posty\OneDrive\Desktop\2022\ravetraining
ROLLER COASTER Small Steel Coaster MECHANICS 03 Lap Bar Restraints. Open,Clank,Metal.wav: 100%|██████████████████████████████████████████████████████████████████████████████| 563/563 [26:58<00:00,  2.88s/it]
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: runs\test1\rave
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name          | Type                | Params
------------------------------------------------------
0 | pqmf          | CachedPQMF          | 16.7 K
1 | loudness      | Loudness            | 0
2 | encoder       | Encoder             | 4.8 M
3 | decoder       | Generator           | 12.8 M
4 | discriminator | StackDiscriminators | 16.9 M
------------------------------------------------------
34.6 M    Trainable params
0         Non-trainable params
34.6 M    Total params
138.202   Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "C:\Users\posty\RAVE\train_rave.py", line 171, in <module>
    trainer.fit(model, train, val, ckpt_path=run)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1234, in _run
    results = self._run_stage()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1343, in _run_train
    self._run_sanity_check()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1411, in _run_sanity_check
    val_loop.run()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\loops\dataloader\evaluation_loop.py", line 153, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\loops\base.py", line 199, in run
    self.on_run_start(*args, **kwargs)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\loops\epoch\evaluation_epoch_loop.py", line 87, in on_run_start
    self._data_fetcher = iter(data_fetcher)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 178, in __iter__
    self.dataloader_iter = iter(self.dataloader)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\torch\utils\data\dataloader.py", line 368, in __iter__
    return self._get_iterator()
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\torch\utils\data\dataloader.py", line 314, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\posty\anaconda3\envs\rave\lib\site-packages\torch\utils\data\dataloader.py", line 927, in __init__
    w.start()
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object
(rave) PS C:\Users\posty\RAVE> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\posty\anaconda3\envs\rave\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

accelerator problem? #70

{{title}}

Replies: 1 comment 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

accelerator problem? #70

postyork May 20, 2022

Replies: 1 comment · 3 replies

caillonantoine May 20, 2022 Collaborator

postyork May 20, 2022 Author

caillonantoine May 20, 2022 Collaborator

postyork May 20, 2022 Author

postyork
May 20, 2022

Replies: 1 comment 3 replies

caillonantoine
May 20, 2022
Collaborator

postyork May 20, 2022
Author

caillonantoine May 20, 2022
Collaborator

postyork May 20, 2022
Author