Double all timeouts on flutter_tools integration tests #20872
Conversation
Due to CPU contention we've seen these go really slow on Cirrus (see #19542 (comment)), and there's also a chance our flakes are timeouts due to running slow rather than hanging.
|
@fkorotkov the Windows bot here seemed to time out after all the tests finished, which is a new kind of failure mode... |
|
@Hixie, sorry about that. That is very weird. Do you know if running tests involve creation daemon processes? I found a few Go issues that show the same behaviour. Investigating... |
|
We have had issues in the past where we didn't terminate processes correctly, though normally it's on failure (and it seems there weren't any here). Are you doing something that would wait for background processes and not just waiting for the main script to end? Restarting this one to make it green before merging; I believe the failed run logs can still be accessed if required. |
|
Yeah we could definitely have runaway daemon processes. Is it possible to dump all running processes to the logs on termination, or some such? |
|
@Hixie @DanTup seems might be related to golang/go#13155. I've added some extra checks to workaround it. Will deploy shortly. Update: deployed. |
|
@fkorotkov I think I just saw the same again here: https://cirrus-ci.com/task/5300252680650752 If you think it's us leaving processes around, let me know and maybe we can dump the list at top/bottom of the list to debug (maybe it'd be a neat feature if Cirrus failed with "script finished but these new processes were left hanging around:.."!). |
|
@DanTup weird thing is that it's happening only on Windows so far 🤔Will try to get some debug information. |
|
Great; we'll let you know if we see it again 👍 |
Due to CPU contention we've seen these go pretty slow on Cirrus (see #19542 (comment)) and there's also a chance our flakes are timeouts due to running slow rather than hanging.