Handle node allocation errors gracefully, log details, and exit on failure

[TaekyungHeo]

Summary

This is a bug fix for https://redmine.mellanox.com/issues/4100971.

• Handle node allocation errors gracefully, log details, and exit on failure
• Enhance exception message in allocate_nodes

Test Plan

CI passes.

[TaekyungHeo]

Please find the updated code, @amaslenn .

[amaslenn]

Could you please explain how this scenario will work:

1. test1 has enough nodes
2. test2 doesn't have enough nodes
3. test3 has enough nodes
   ?

What will user see as the result? Which test will run?

[TaekyungHeo]

@amaslenn CloudAI can run test 1, but when it tries to run test 2, it will fail and raise an exception. Test 3 will not run as CloudAI has failed.

[TaekyungHeo]

@amaslenn I recall that we had a similar discussion with Jeff and Srivatsan earlier. This is an important topic, and we should make a decision, but it is outside the scope of this PR.

[amaslenn]

..., but it is outside the scope of this PR.

Right, I asked this because we do not raise an exception now and do not propagate it, so I wasn't sure that mentioned behaviour remains. But if you confirm that, no issues, we are good.