cpu (ubuntu-18.04, 3.9, oldest, stable) fails with segmentation fault

[akihironitta]

🐛 Bug

Observed that cpu (ubuntu-18.04, 3.9, oldest, stable) quite often fails with segmentation fault after the actual tests had passed as the following:

=== 2218 passed, 532 skipped, 7 xfailed, 1470 warnings in 634.06s (0:10:34) ====
/home/runner/work/_temp/50029d2b-e6ee-470c-bd34-b5acf8834c42.sh: line 2:  3747 Segmentation fault      (core dumped) coverage run --source pytorch_lightning -m pytest pytorch_lightning tests -v --durations=50 --junitxml=junit/test-results-Linux-py3.9-oldest-stable.xml

To Reproduce

• https://github.com/PyTorchLightning/pytorch-lightning/runs/4587925567?check_suite_focus=true#step:16:2825
• https://github.com/PyTorchLightning/pytorch-lightning/runs/4587519066?check_suite_focus=true#step:16:2825
• https://github.com/PyTorchLightning/pytorch-lightning/runs/4587417079?check_suite_focus=true#step:16:2825
• https://github.com/PyTorchLightning/pytorch-lightning/runs/4559402636?check_suite_focus=true#step:16:2821

Expected behavior

No segmentation fault 🍏

Environment

• PyTorch Lightning Version (e.g., 1.5.0):
• PyTorch Version (e.g., 1.10):
• Python version (e.g., 3.9):
• OS (e.g., Linux):
• CUDA/cuDNN version:
• GPU models and configuration:
• How you installed PyTorch (conda, pip, source):
• If compiling from source, the output of torch.__config__.show():
• Any other relevant information:

Additional context

cc @carmocca @akihironitta @Borda

[carmocca]

Hi @akihironitta, thanks for writing this.

This has been failing for a while now. I don't know what would be the cause but I'd say it's either caused by coverage or the destruction of a resource when testing finishes.

So I would try removing coverage to check or removing tests to see if it still segfaults.

[carmocca]

Just saw some PRs with the job passing, so it is also flaky 😵

[daniellepintz]

Is there any progress on resolving this issue? Can we just remove this job? I believe it is causing more harm than good by making our CI red :/

[carmocca]

I'll leave the decision to @akihironitta as he's the one trying to save us

[akihironitta]

Hi @daniellepintz, sorry for the delay. I'm now working on it with Carlos's help but not sure if it'll be resolved in a reasonably short period of time.

Can we just remove this job?

IMHO, I'd keep it because it segfaults after the actual testing part finishes in the job, and thus we can still see its result in the log (although "the job" is marked red). If we just remove the job, no one is able to catch any bugs/behaviours with the environment.

[Borda]

btw, I would not try to install py3.9 on ubuntu 18.04, rater tries ubuntu 20.04

[akihironitta]

@Borda I thought about that, too, and I'll definitely give it a try with a newer version of the OS, but why not install py3.9 on ubuntu18.04? Is there any known issue or something?

[Borda]

@Borda I thought about that, too, and I'll definitely give it a try with a newer version of the OS, but why not install py3.9 on ubuntu18.04? Is there any known issue or something?

I do not know about any, but why waste time on very unlikely configuration 🐰

[guotuofeng]

It seems like a coverage bug on Ubuntu for python 3.9.9 like nedbat/coveragepy#1294? I tried locally in Ubuntu 18.04 with python 3.9.7, the coverage works fine.

Could we try python with other versions like 3.9.7?

[akihironitta]

Hi @guotuofeng, thanks for your support! It seems like the problem is pytest-dev/pytest#8841 in our case as commented by @\carmocca in the linked PR: #11217 (comment)