Retry GPU devices check during env vars load if instance supports GPU

[danehlim]

Summary

Current behavior of ecs-init during loading environment variables is that if environment variable with key config.GPUSupportEnvVar has value true, then check if NVIDIA GPU devices are present on the instance. If they are not yet present at the time of the check, then ecs-init does not load in the aforementioned environment variable and effectively later gives up on attempting any PRESTART GPU setup (such as NVIDIA GPU info file creation).

Although under usual circumstances NVIDIA GPU devices are present by the time ecs-init checks for the same on a container instance that supports GPU workloads, there is nothing in place that guarantees this. Thus, there could be a race condition where NVIDIA GPU devices are not yet present on the instance at the time that ecs-init checks (perhaps due to delays in device initialization), but then do become present shortly afterwards. In such a case, ecs-init GPU setup is skipped and ECS Agent later runs into error when attempting to initialize its NVIDIA GPU manager.

Implement a retry mechanism for checking NVIDIA GPU device availability during the time ecs-init loads environment variables in the event that the underlying container instance supports GPU workloads. This ensures that ecs-init at least waits a reasonable amount of time for NVIDIA GPU devices to be present in this case before continuing. If NVIDIA GPU devices still are not yet present after the maximum number of retries, then ecs-init will give up and continue as per the current behavior. This shall not impact existing scenarios where NVIDIA GPU device is already available by the time ecs-init checks for the first time.

Implementation details

If environment variable with key config.GPUSupportEnvVar has value true, it is expected that NVIDIA GPU devices should eventually be present on the instance. Thus, do not give up and continue right away in the event that NVIDIA GPU devices are not yet present.

Define nvidiaGPUDevicesPresentWithRetries() to be called in this case which will retry and wait for some time for NVIDIA GPU devices to be present before continuing. The retry logic is configured to:

• retry on failure
• every 3 seconds
• for a maximum of 10 attempts

This results in 30 seconds of retry in the worst case scenario.

Also, rename some existing constants in the docker package of ecs-init to improve readability.

Testing

New tests cover the changes: yes

Also manually test using a custom ecs-init built with the changes in this pull request against an internal repro environment that reliably reproduces the race condition when using the current ecs-init. With the custom ecs-init, race condition behavior is no longer observed.

Partial ecs-init logs for the above:

./ecs-init.log:level=warn time=2024-10-07T23:27:18Z msg="NVIDIA GPU devices are not yet present, retrying (attempt 1/10) in 3000000000 nanoseconds"
./ecs-init.log:level=info time=2024-10-07T23:27:21Z msg="pre-start: setting up GPUs"
./ecs-init.log:level=info time=2024-10-07T23:27:21Z msg="By using the GPU Optimized AMI, you agree to Nvidia’s End User License Agreement: https://www.nvidia.com/en-us/about-nvidia/eula-agreement/"
./ecs-init.log:level=info time=2024-10-07T23:27:21Z msg="pre-start: done setting up GPUs"

Description for the changelog

Bugfix: Retry GPU devices check during env vars load if instance supports GPU

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions?

No

Does this PR include the addition of new environment variables in the README?

No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[singholt]

thank you Dane! 🚀