Why is my hybrid RNN-T - CTC model training hanging at training logging time?

[riqiang-dp]

Very similar to this issue from years ago: #532

In the sense that, I tried to trace where the hanging happens, and I also got actions.py and backward() when it's stuck and try to make the process print out a traceback.

It also happens at "evaluation time" - it happens when the steps of log_every_n_steps happens, so the trainer is decoding some training samples and print out the hypothesis, compute loss and WER.

When it hangs, GPU usage goes to 100% but VRAM is not maxed out. It then times out after a few dozen minutes.

When I change the data, it happens at different times during training, for example when I removed some data < 1s from the dataset, it happened at 400 steps instead of 200 steps (where my log_every_n_steps = 200), so it successfully evaluated one batch of data but not the next one. So initially I assumed it had to do with data, but then I tried CTC only training and it was going smoothly. Also I noticed that with RNNT+CTC training it prints out more samples during this evaluation / logging step? In CTC training only one sample is printed out where RNNT+CTC prints multiple.

Also, I'm training with NeMo toolkit 2.0.0rc1, Pytorch 2.2.2, (numpy < 2.0), multiple GPUs with ddp, same number of CPUs as GPUs for dataloader workers.

I'm not sure what are some other relevant information needed to debug this, please let me know.

[nithinraok]

Hi,
This got fixed with #10787. Can you please pull these changes and re-try?

-Nithin

[riqiang-dp]

Hi thanks for the update. I tried it, but it still hangs at eval time...(the logging time is not hanging anymore)

[nithinraok]

Hi @riqiang-dp,

I hope you are trying with r2.0.0 and container: nvcr.io/nvidia/nemo:24.07 or nvcr.io/nvidia/nemo:dev . If not could you please try with above environment. We are not seeing this issue on evaluation, so it would be helpful to know more about your environment and training configuration

[riqiang-dp]

I see! I'll try it thanks

[riqiang-dp]

I made it work by removing the if blocks in

NeMo/nemo/collections/asr/modules/rnnt.py

Lines 1494 to 1496 in db7169d

	if self.training:
	original_sync = self.wer._to_sync
	self.wer._to_sync = False

and

NeMo/nemo/collections/asr/modules/rnnt.py

Lines 1508 to 1509 in db7169d

	if self.training:
	self.wer._to_sync = original_sync

Is this the right way to go?

[nithinraok]

That was interesting, I placed it to handle the hanging issue during training. Can you share your environment state?
Container and nemo version.

[riqiang-dp]

I use poetry to handle the environment:

[tool.poetry.dependencies]
python = "^3.11.8"
voiceai-tooling-mlflow-utils = "^0.2.3"
cython = "^3.0.10"
nemo-toolkit = {git = "https://github.com/NVIDIA/NeMo.git", rev = "r2.0.0", extras = ["asr"]}
torch = { version = "^2.2.2", source = "pytorch" }
torchaudio = { version = "^2.2.2", source = "pytorch" }
torchmetrics = { version = "^1.3.2", source = "pytorch" }
torchvision = { version = "^0.17.2", source = "pytorch" }
voiceai-asr-textproc = "11"
plotly = "^5.22.0"
nbformat = "^5.10.4"
numpy = "<2.0"
numba = "0.59.1"
cuda-python = "^12.6.0"


[[tool.poetry.source]]
name = "voiceai-pypi"
url = "https://us-python.pkg.dev/voiceai-infra/voiceai-pypi/simple"
priority = "supplemental"

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
priority = "explicit"

Plus, with the hanging gone, model (this RNN-T model but also CTC models) can't train past certain epochs because the CPU memory usage keeps increasing epoch after epoch and eventually OOM kills the dataloader.