Timed out initializing process group in store based....

[EvanZ]

Does anyone know what this error means?

Exception has occurred: RuntimeError

Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=8, worker_count=7, timeout=0:30:00)

[YiranJing]

Hey EvanZ,

I have met the same error in single-node multi-GPU training (DDP).

In DDP, the trainer won't start until all workers join the process group (i.e. initializing process group) and there is a timeout for operations executed against the process group. The default value equals 30 minutes.

Sometimes, the process group initialization could take quite a long time and exceed 30 mins (timeout)

The solution is to increase the timeout of DDPStrategy.

trainer = pl.Trainer(
   # set timeout to 1hr, the default timeout is 1800 sec (30 mins)
   strategy=pl.strategies.DDPStrategy(timeout=datetime.timedelta(seconds=3600)),
   ...
 )
trainer.fit(...)

Please let me know if this way can fix your issue.

[bjaeger1]

Hi @YiranJing,
I also ran into the same error and will try to increase the timeout for ddp as you described.
Do you know if the dataset size influences the initializing process somehow?
I am able to train a smaller dataset with 4 GPU's (ddp) but the same training procedure gets stuck (timeout because the initialization takes too long) when using a larger dataset. However when I reduce the number of GPU's it works again.
Its not clear to me how the dataset size can influence the initialization time? Or is it maybe related to the num_workers of the dataloader?

[bxrjmfh]

same problem..

你好@YiranJing， 我也遇到了同样的错误，并将尝试按照您的描述增加 ddp 的超时时间。 您知道数据集大小是否会以某种方式影响初始化过程吗？ 我能够使用 4 个 GPU (ddp) 训练较小的数据集，但在使用较大的数据集时，相同的训练过程会卡住（超时，因为初始化时间太长）。然而，当我减少 GPU 的数量时，它又可以工作了。 我不清楚数据集大小如何影响初始化时间？或者它可能与num_workers数据加载器有关？

[fengza]

Hello @bxrjmfh , is your problem solved? I encountered the same problem as you.

[bxrjmfh]

set CUDA_VISBLE_DEVICE=0 if not necessary for multiple device.

[HaiCLi]

I met this problem on Multi-nodes Multi-gpus Type. Did anyone has solutions?