Training stuck at 0% after few epochs while training with DDP

[HareshKarnan]

🐛 Bug

I recently updated to pytorch_lightning 1.1.7 and noticed that after a few epochs of training, the training % is stuck at 0% and never progresses. When I switched back to 1.1.4, this strange behavior does not occur. I do not know the root cause of this issue.

• PyTorch Version (e.g., 1.0): 1.1.7
• OS (e.g., Linux): Linux, U18
• How you installed PyTorch (conda, pip, source): pip install
• Build command you used (if compiling from source):
• Python version: 3.6
• CUDA/cuDNN version: 10.0
• GPU models and configuration: RTX 2080 x3
• Any other relevant information:

[ndrplz]

Related? I'm facing a similar issue, not sure if those might help

• Code stuck on "initalizing ddp" when using more than one gpu  #4612
• NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8 pytorch/pytorch#38702

[matyushinleonid]

Hi, @HareshKarnan @ndrplz. Folks, may I ask you to run your pipelines with NCCL_ASYNC_ERROR_HANDLING=1?
Seems like we have the same problem. I use 1.2.0.dev0 lightning version, so this issue may migrate to the new release... My error trace:

https://pastebin.com/LNxG2JF6

[HareshKarnan]

@ndrplz i dont know if it is related, but the problem here is that training does happen for first few epochs - in my case, it ran for 13 epochs and then gets stuck at 0% by epoch 14.

[HareshKarnan]

https://pastebin.com/QhanSNrK

Here is my output. It gets stuck at 0% while training in epoch 16

[matyushinleonid]

https://pastebin.com/QhanSNrK

Here is my output. It gets stuck at 0% while training in epoch 16

run your script with NCCL_ASYNC_ERROR_HANDLING=1 flag. Like NCCL_ASYNC_ERROR_HANDLING=1 python train.py. Without this flag, I do not get any error like you since, by default, this async error does not raise an Exception (at least, for me). If we have the same error traces, it will help community to fix the bug.

[HareshKarnan]

I ran the script again as NCCL_ASYNC_ERROR_HANDLING=1 python train.py and got no difference in the output :

https://pastebin.com/jtLyuXTN

This time it got stuck at epoch 11

[genghisun]

same problem here, stuck at 0% at epoch 18

[Borda]

Mind share code ideally in Colab to reproduce?

[genghisun]

In my case, it stuck at 0% at epoch 18 with 2 gpus ddp before.
Then I try to use only 1 gpu, currently trained for 100+ epochs without any problem.

[stillwalker1234]

have the same issue after updating to 1.1.8, will try with 1.2.0.dev0 to see if it has the same error

pytorch 1.7
2x 3090
cuda 11.2

[stillwalker1234]

1.2.0rc1 also has the issue, 1.1.6 does not

[SeanNaren]

Would you be able to try master? We've recently consolidated the branches back to master!

[tchaton]

Hey everyone,

Could it be related to this issue and solved by this PR: #6004

@HareshKarnan,

I have seen val_loss in your logs. I think it might be related. Mind trying the fix ?

Best,
T.C

[stillwalker1234]

Started a run with master + @tchaton 's patch, will see how it goes.

UPDATE:

run stalled at epoch 6 :(

[Borda]

@tchaton any update here?

[talolard]

Also having this problem.
Always getting stuck on epoch 2, if I have checkpoints enabled when training with ddp on single machine

[talolard]

Wanted to add some details.
We ran some different code with copy paste and the problem reoccured. We had the checkpoint set up to monitor train_loss and once we took that out everything worked fine.
In other words, I suspect this has something to do with monitor on ddp .

[edenlightning]

Thanks! Will take a look and try to resolve it soon.

[edenlightning]

@HareshKarnan, @talolard, @stillwalker1234 or @genghisun can any of you please provide a reproducible script/colab?

Do you checkpoint based on a value that is not coming from lightning metrics and can be different on different processes? Probably related to #5604 (comment).

[JonasFrey96]

I face the same issue and the problem was resolved by changing the number of workers to 0. This is not an acceptable fix workaround. I assume we have some deadlock when something goes wrong with spaning dataloaders.

[tchaton]

Dear @JonasFrey96, @taltalim,@HareshKarnan

Would it be possible for you to work on a reproducible script using the BoringModel.

Best,
T.C

[jgbos]

I'm having the same issue that goes away when using the default ModelCheckpoint settings. Seems to be a problem with setting monitor and save_top_k. I have not been able to achieve the same issue with BoringModel.

edit: just wanted to add that I am logging the loss as self.log("Val/Loss", loss), monitor="Val/Loss", and save_top_k=1

[Anjum48]

I've been running into this issue with 1.18 & 1.2.1 the last few days with ddp and 2 GPUs

Things tried:

• Upgrade from Pytorch 1.7, to 1.8
• Upgrade drivers from 450 to 460
• Upgrade from CUDA 10.2 to 11.2
• Upgrade NCCL to latest
• Rebuild entire conda env
• Remove all loggers (in case of wandb multithreading)
• os.environ["WANDB_START_METHOD"] = "fork"
• cv2.setNumThreads(0)
• DDPPlugin(find_unused_parameters=True)
• sync batch norm True/False
• Reduced num_workers (zero is too slow, but went down to 2)
• Roll back Linux kernel in Ubuntu (I think a saw a meme about this the other day so I thought I'd give it a try haha)
• Nothing unusual in dmesg

The epoch seems to be random (might stop after first, or after several), but always just after validation and on the start of the next epoch (0%). No error messages. Both GPUs locked at "100%" but the data being sent to GPU 0 (RX in nvtop) is 0 MB/s and the temperatures show that the GPUs are not working hard. 2 CPU cores locked at 100%.

[Anjum48]

I think I've isolated the issue from the discussion in #5604 (comment).

This issue started when I switched the ModelCheckpoint monitor from AUC (using PL metrics and dist_sync_on_step=True) to val_loss. val_loss was not being synced between the 2 GPUs, so adding this line:

val_loss = torch.mean(self.all_gather(val_loss))

In validation_epoch_end fixed the issue for me. Model has been successfully training overnight and still running :)

[ifsheldon]

I can see the same issue when running my training script. My pl is 1.2.1. Still trying out which approaches mentioned above is useful in my case.

[Anjum48]

Edit: response to a deleted question ¯_(ツ)_/¯

self.all_gather will return a tensor of shape [num_gpus, x] where x is the result from a single GPU. If you're only syncing a scalar, then you can just take a mean (or min/max/std) like in my example.

If for example, x is not a scalar (e.g. you want to calculate IoU or something), you can combine them using something like:

def gather_and_squash(self, x):
    # The reshape goes from [n_gpus, bs, N, N] to [n_gpus * bs, N, N]
    return torch.reshape(self.all_gather(x), [-1] + list(x.shape)[1:])

At the moment I'm working on a joint segmentation & classification problem and using valid_clf as my monitor. Here's what my validation_epoch_end looks like:

def validation_epoch_end(self, outputs):
    loss_val = torch.stack([x["val_loss"] for x in outputs]).mean()
    loss_seg = torch.stack([x["loss_seg"] for x in outputs]).mean()
    loss_clf = torch.stack([x["loss_clf"] for x in outputs]).mean()
    
    log = {
        "loss/valid_seg": torch.mean(self.all_gather(loss_seg)),
        "loss/valid_clf": torch.mean(self.all_gather(loss_clf)),
    }
           
    # AUC calculation etc...
    y_true = torch.cat([x["y_true"] for x in outputs])
    y_pred = torch.cat([x["y_pred"] for x in outputs])
    # etc ...

    self.log_dict(log)
    self.log_dict(
        {"loss/valid": loss_val, "auc/overall": log["metric"]}, prog_bar=True,
    )

[jgbos]

I'm also noticing that there are logger files (tfeevents) for each process now. I wonder if #6364 and this issue are related.

[taltalim]

At the moment I can't seem to reproduce using BoringModel. I will look into that in the next days.

[senarvi]

I encountered a similar issue, training hanging at the end of a validation epoch, when a custom metric is being synced between processes. I used cat as dist_reduce_fx. One would think that in the tensors produced by different processes, the first dimension - along which the tensors are going to be concatenated - doesn't have to match. However, gather_all_tensors uses torch.distributed.all_gather to gather the tensors from different processes, and all_gather requires the tensors to be correctly sized. gather_all_tensors assumes that all processes create identically-sized tensors. If the sizes don't match, syncing will hang indefinitely.

This is probably not the issue for most people, but I just wanted to point out that this can cause some confusion too. I guess there's no way to check that all processes produce identically-sized tensors. Maybe the Metric API documentation could be updated to note that all state variables must have identical shape across processes.

[jgbos]

@Anjum48 did implementing those reduction in val_epoch_end fix your issue with training hanging? My feeling is there's an uncaught error in the syncing of tensors like @senarvi found. I can't seem to find the error though.

[jgbos]

I hope this helps: I implemented all the reduction and logging in validation_epoch_end.

Looks like PL attempts to do a model checkpoint before running validation_epoch_end, the trainer crashes because it was unable to find the monitor metric. I can verify that validation_step_end is called before the crash but not validation_epoch_end. Unfortunately having troubles find the code to check this behavior.

[TysonYu]

I have the same problem when using version==1.2.3

[TysonYu]

Hi, @ifsheldon I am facing the same problem, may I ask how do you solve this problem?

[jgbos]

An initial test with today's master seems to show this issue is fixed for me

[thiyagu145]

looks like this issue is fixed in version 1.2.4

[jgbos]

@thiyagu145 oh really? I'll double check, I thought it required some changes not in 1.2.4. But that would be great if 1.2.4 fixes it.

[thiyagu145]

yea, training completed without any issues.

[TysonYu]

Yeah, now I try 1.2.4 and there is no issue anymore.

[taltalim]

I can confirm 1.2.4 fixes the issue. I'm asking myself which PR fixed this - possibly #6410 ?

[luozhouyang]

Same issue.
pytorch-lightning version 1.3.7post0

[JusperLee]

同样的问题。
pytorch-闪电版本1.3.7post0

You can downgrade to 1.2.4

[yinrong]

I'm stuck too using lightning=1.4.9
the numbers "96% 4260/4435" keeps the same for forever.

[HareshKarnan]

same issue with 1.8.3.post1