integration.modules.test_saltutil tests failing on windows sometimes

[Ch3LL]

Description
I'm seeing these tests fail on windows periodically:

integration.modules.test_saltutil.SaltUtilSyncModuleTest.test_sync_all
integration.modules.test_saltutil.SaltUtilSyncModuleTest.test_sync_all_blacklist
integration.modules.test_saltutil.SaltUtilSyncModuleTest.test_sync_all_whitelist

Here they fail on a PR test: https://jenkinsci.saltstack.com/job/pr-windows2019-py3/job/PR-56408/6/

and on branch tests: https://jenkinsci.saltstack.com/job/pr-windows2019-py3/job/master/737/

[xeacott]

Looking at this now...

[xeacott]

They seem to fail very sporadically with no real paper trail. There is a PR out right now with Windows2019 being built and the tests pass there:

[2020-04-20T17:48:16.371Z]          test_sync_all (integration.modules.test_saltutil.SaltUtilSyncModuleTest)
[2020-04-20T17:49:03.695Z]        [CPU:0.0%|MEM:61.1%]  ... ok (47.939s)
[2020-04-20T17:49:03.695Z]          test_sync_all_blacklist (integration.modules.test_saltutil.SaltUtilSyncModuleTest)
[2020-04-20T17:49:52.830Z]        [CPU:0.0%|MEM:61.1%]  ... ok (49.093s)
[2020-04-20T17:49:52.830Z]          test_sync_all_blacklist_and_whitelist (integration.modules.test_saltutil.SaltUtilSyncModuleTest)
[2020-04-20T17:50:41.872Z]        [CPU:0.0%|MEM:61.2%]  ... ok (49.111s)
[2020-04-20T17:50:41.872Z]          test_sync_all_whitelist (integration.modules.test_saltutil.SaltUtilSyncModuleTest)
[2020-04-20T17:51:31.767Z]        [CPU:0.0%|MEM:61.2%]  ... ok (48.783s)
[2020-04-20T17:51:31.767Z]          test_pillar_refresh (integration.modules.test_saltutil.SaltUtilSyncPillarTest)
[2020-04-20T17:51:31.767Z]        [CPU:0.0%|MEM:61.2%]  ... skip (0.000s)
[2020-04-20T17:51:31.767Z]          test_sync_grains (integration.modules.test_saltutil.SyncGrainsTest)
[2020-04-20T17:51:45.763Z]        [CPU:0.0%|MEM:61.2%]  ... ok (14.328s)

So immediate thinking is put sleeps in the test or skip them for now until a full test run can be run locally with the log file investigated. Otherwise can't repro locally and cant preproduce on a Windows 2019 VM.

[xeacott]

Link to that PR #56637

[Ch3LL]

You should be able to look at the full log file on jenkins where the test fails in the build artifacts. Also if you see any way to make this test less flaky that is an approach you can take as well.

Also did you try running the full test suite to try to reproduce? It seems to be happening pretty consistently across tests. I want to guess 1 out of 3-5 runs.

[xeacott]

I may look at making the test less flakey. I have a VM running nightly that is running the full suite, so im investigating those logs :)