EFCore.SqlServer.FunctionalTest hanging on Linux VM (Ubuntu 16.04 LTS and 18.10)

[Gary-Zh]

When I was running EFCore.SqlServer.FunctionalTest on Linux VM, the test hangs frequently.
The test runs but does not finish. dotnet process looks hanging as well, it consumes 0 CPU usage but does not terminate.
When it doesn't hang, the test finishes within 10 mins on my VM, but it could hang hours and I had to manually terminate the test execution.

Here's what I did:
Create a completely clean Ubuntu VM (I've tried version 18.10 and 16.04).
Install git and clone EFCore repo.
Go to EFCore root folder and run ./build.sh -c Release -v d

export Test__SqlServer__DefaultConnection="Data Source=192.xxx.xxx.xxx;DataBase=master;UID={id};PWD={pwd};Connect Timeout=60;ConnectRetryCount=0;" (in order not to skip tests)

./test.sh -projects /test/EFCore.SqlServer.FunctionalTests/EFCore.SqlServer.FunctionalTests.csproj -configuration Release -v d

Sometimes the test finishes successfully and gives me the test result file. Sometimes it just hangs.
Here's the log file being generated when it hangs:
EFCore.SqlServer.FunctionalTests_netcoreapp3.0_x64.log

The screenshot shows the dotnet process is hanging:

Further technical details

EF Core version: latest in master branch
Operating system: Ubuntu 16.04 LTS and 18.0.

[Gary-Zh]

It does not hang on Windows

[ajcvickers]

@Gary-Zh How many logical CPUs are you using when you run the tests on Linux?

[Gary-Zh]

@ajcvickers I have 3 logical CPUs on my VM.

[ajcvickers]

@roji to try running locally on Linux.

[roji]

Tested locally - everything seems OK, no deadlocks...

(ran tests on daeb665, on Ubuntu 19.04, against SQL Server running locally, 6 physical cores).

[smitpatel]

It does not hang on Windows

@Gary-Zh - How many logical CPUs on your windows machine? The current implementation on query translation and concurrent running on tests can cause deadlock on machines with low number of cores regardless of OS.

[Gary-Zh]

@smitpatel I have 8 logical CPUs on my windows machine.

Do you suggest me to add more cores to the VM and try reproducing the issue again?

[divega]

@Gary-Zh yes, that is something we believe is worth trying.

[ajcvickers]

Closing this as we have been able to run on Linux without issues, and until the known deadlock issues in query are fixed there is nothing more to try here for now.

[Gary-Zh]

Hi @divega , it's been a while since we last time running EFCore tests.
So I've tried to increase the core number to 6 this time, same thing happens, please see the pictures below:

It still hangs on my Linux VM.

[Gary-Zh]

Can we re-open this issue and investigate more?

[roji]

@Gary-Zh the problem is that this occurs neither in our continuous builds (which run all the time), nor on my personal build machine. So whatever issue you're running into seems to be somehow tied to your specific environment...

Any chance you could try on a physical Linux box (non-VM) and report?

[divega]

@roji would it possible to break into the running tests with a debugger to figure out where the execution is at then the tests stall?

[roji]

Sure, but I haven't actually seen the tests stall on my Linux machine at any point... I just ran the SqlServer functional test suite, once on 2.2.5 and once on master and everything was fine.

@Gary-Zh have you had a chance to try on a physical Linux box, ideally against the same SQL Server instance?