Fix model checkpoint saving issue when using PEFT

[gioannides]

What does this PR do?

This PR fixes an issue in the model checkpoint saving process when using Parameter-Efficient Fine-Tuning (PEFT) in distributed training environments. Previously, if the directory for saving adapters already existed, an error would occur, particularly in distributed scenarios where multiple processes might try to create the directory simultaneously. This PR ensures that the directory creation is handled safely by using the exist_ok=True flag.

Error Example

In distributed training environments, the following error could occur if the directory already existed:

FileExistsError: [Errno 17] File exists: '/working_dir/output/checkpoint-150/adapter_shards/model'

This error is now handled by using the exist_ok=True parameter when creating directories, preventing failures if the directory already exists.

How does this PR fix the problem?

This PR introduces the exist_ok=True flag in the directory creation step to handle the case where the directory already exists. The relevant code snippet is:

adapter_shards_dir_model = os.path.join(output_dir, "adapter_shards", "model")
if not os.path.isdir(adapter_shards_dir_model):
    os.makedirs(adapter_shards_dir_model, exist_ok=True)

By using exist_ok=True, the os.makedirs call will not raise an error if the directory already exists, making it safe for distributed environments.

Motivation and Context

In distributed training environments, multiple processes might attempt to save model checkpoints to the same directory. If the directory already exists, this can lead to an interruption in the training process. This PR resolves the issue by ensuring that the directory creation step is non-blocking and error-free.

Impact

This change makes the checkpoint-saving mechanism more robust when using PEFT in distributed training scenarios, eliminating errors related to existing directories and improving the reliability of the training process.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gioannides]

@michaelbenayoun just pinging to have a look