ERROR in run_hp_search_optuna when trying to use multi-GPU

[sstoia]

System Info

• transformers version: 4.28.1
• Platform: Linux-3.10.0-1160.95.1.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.9.16
• Huggingface_hub version: 0.14.1
• Safetensors version: not installed
• PyTorch version (GPU?): 1.13.1 (False)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The problem appears when using run_hp_search_optuna method from transformers/integrations.py . This method is called when trying to perform an hyperparameter search with the Trainer.hyperparameter_search method:

best_trial = trainer.hyperparameter_search(
     direction='maximize',
     backend='optuna',
     hp_space=optuna_hp_space,
     n_trials=10,
)

The error obtained is the next one:

Traceback (most recent call last): File "/mnt/beegfs/sstoia/proyectos/LLM_finetuning_stratified_multiclass_optuna.py", line 266, in <module> best_trial = trainer.hyperparameter_search( File "/mnt/beegfs/sstoia/.conda/envs/env/lib/python3.9/site-packages/transformers/trainer.py", line 2592, in hyperparameter_search best_run = backend_dict[backend](self, n_trials, direction, **kwargs) File "/mnt/beegfs/sstoia/.conda/envs/env/lib/python3.9/site-packages/transformers/integrations.py", line 218, in run_hp_search_optuna args = pickle.loads(bytes(args_main_rank)) _pickle.UnpicklingError: pickle data was truncated

Expected behavior

It should work, as the same function without multi-GPU works fine. I guess the problem comes from a parallelization error, as both GPUs may write on the same file.

[amyeroberts]

cc @muellerzr @pacman100

[muellerzr]

Sorry for the delay, will be looking into it over this week!

[linhdvu14]

I'm running into the same issue. Any updates on this please?

[amyeroberts]

Gentle ping @muellerzr @pacman100

[amyeroberts]

Another ping @muellerzr @pacman100

[NishchalPrasad]

Running into the same issue. Using the latest version of transformers (4.40.1) and python: 3.11

[tomaarsen]

Having the same issue with my Trainer subclass when doing HPO with DDP and optuna.

[amyeroberts]

Gentle ping @muellerzr

[svduplessis]

Same issue here. Also trying to run hyperparameter search with DDP (accelerate launch) using Trainer and Optuna as the backend.

The following error is returned:

[rank2]: Traceback (most recent call last):
[rank2]:   File "/scratch-small-local/303700.hpc1.hpc/run_hpo_optuna.py", line 1054, in <module>
[rank2]:     main()
[rank2]:   File "/scratch-small-local/303700.hpc1.hpc/run_hpo_optuna.py", line 956, in main
[rank2]:     best_trial = trainer.hyperparameter_search(
[rank2]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/trainer.py", line 3206, in hyperparameter_search
[rank2]:     best_run = backend_obj.run(self, n_trials, direction, **kwargs)
[rank2]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/hyperparameter_search.py", line 72, in run
[rank2]:     return run_hp_search_optuna(trainer, n_trials, direction, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/integrations/integration_utils.py", line 237, in run_hp_search_optuna
[rank2]:     args = pickle.loads(bytes(args_main_rank))
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: _pickle.UnpicklingError: pickle data was truncated
[rank1]: Traceback (most recent call last):
[rank1]:   File "/scratch-small-local/303700.hpc1.hpc/run_hpo_optuna.py", line 1054, in <module>
[rank1]:     main()
[rank1]:   File "/scratch-small-local/303700.hpc1.hpc/run_hpo_optuna.py", line 956, in main
[rank1]:     best_trial = trainer.hyperparameter_search(
[rank1]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/trainer.py", line 3206, in hyperparameter_search
[rank1]:     best_run = backend_obj.run(self, n_trials, direction, **kwargs)
[rank1]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/hyperparameter_search.py", line 72, in run
[rank1]:     return run_hp_search_optuna(trainer, n_trials, direction, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/2x22/mlm/miniconda/envs/thesis/lib/python3.11/site-packages/transformers/integrations/integration_utils.py", line 237, in run_hp_search_optuna
[rank1]:     args = pickle.loads(bytes(args_main_rank))
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: _pickle.UnpicklingError: pickle data was truncated

[amyeroberts]

Another ping @muellerzr @SunMarc

[SunMarc]

Hey @sstoia and @tomaarsen and everyone who has this issue ! I was able to reproduce the error and fix with the PR above! Let me know if this works on your side !

[aakash0017]

Is the library updated? I'm still having the same issue

[SunMarc]

Please install the latest version of transformers that was release today and let us know if this is fixed !