Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Run NVIDIA GPU tests on A10 per PR #1544

Conversation

shintaro-iwasaki
Copy link
Contributor

Problem

The OSS side does not run any NVIDIA-GPU unit test, so we cannot catch any OSS-specific issues before merging it; we run these NVIDIA-GPU tests only when a nightly build is pushed to PyPI. This sometimes breaks our nightly build; our customers get unhappy.

Solution

This PR adds a new GitHub Action job that runs unit tests on an AWS NVIDIA A10 machine "per PR". This job creates a wheel file and tests it as the nightly job does, so it should be able to detect some wheel-related issues too (e.g., a build procedure was updated but the nightly script is not updated) though it is not comprehensive (see below).

Note:

  • This job shouldn't lengthen the CI time. This job takes 1-2 hours, which i s shorter than other CUDA-build jobs using GHA-native runners.
  • This job covers only Python 3.10 + CUDA 11.7 + A10 + OS that our Docker script uses. We might want different combinations (e.g., different Python/CUDA/GCC versions and/or Volta GPU) in the future, but anyway this addition should be better than nothing.
  • If you need a thorough nightly/release script check, please add a label (e.g., test_wheel_nightly) to your PR, which runs the real wheel-creation scripts with upload_pypi disabled.

@netlify
Copy link

netlify bot commented Jan 12, 2023

Deploy Preview for pytorch-fbgemm-docs canceled.

Name Link
🔨 Latest commit c5bec2b
🔍 Latest deploy log https://app.netlify.com/sites/pytorch-fbgemm-docs/deploys/63c0b40e50bdf600081b9254

@shintaro-iwasaki shintaro-iwasaki force-pushed the siwasaki/pr/test_nvidia_gpu branch from b738817 to a24d745 Compare January 12, 2023 23:39
@facebook-github-bot
Copy link
Contributor

@shintaro-iwasaki has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@shintaro-iwasaki shintaro-iwasaki force-pushed the siwasaki/pr/test_nvidia_gpu branch from a24d745 to d642bde Compare January 13, 2023 00:25
@facebook-github-bot
Copy link
Contributor

@shintaro-iwasaki has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@shintaro-iwasaki shintaro-iwasaki force-pushed the siwasaki/pr/test_nvidia_gpu branch from d642bde to e6a0e50 Compare January 13, 2023 00:32
@facebook-github-bot
Copy link
Contributor

@shintaro-iwasaki has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@shintaro-iwasaki shintaro-iwasaki force-pushed the siwasaki/pr/test_nvidia_gpu branch from e6a0e50 to c5bec2b Compare January 13, 2023 01:29
@facebook-github-bot
Copy link
Contributor

@shintaro-iwasaki has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@shintaro-iwasaki merged this pull request in 3848313.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants