Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Set Device when kernel be applied into Multiple GPUs. #155

Merged
merged 6 commits into from
Aug 28, 2024

Conversation

LeiWang1999
Copy link
Contributor

This pull request includes several changes to improve device compatibility and streamline the weight transformation process in the bitblas module. The most important changes involve modifying various functions to ensure they correctly handle the device context for GPU operations.

Device Compatibility Improvements:

Code Simplification:

@LeiWang1999
Copy link
Contributor Author

Performance:

1x8192x8192

Lib + SetDevice

kernel only: 0.011571199999999999
Profile mixed-precision matrix multiplication
op:
num_repeats = 28149
0.021342752822495984

Lib only

kernel only: 0.0113664
Profile mixed-precision matrix multiplication
num_repeats = 42799
op: 0.011026431735468434

Torch Func

kernel only: 0.0113664
Profile mixed-precision matrix multiplication
op:

num_repeats = 19784
0.03557460924126193

@LeiWang1999
Copy link
Contributor Author

it's a bit cost..

@LeiWang1999
Copy link
Contributor Author

We suggest performing the setDevice operation outside the kernel, as in multi-GPU environments, multiple contexts need to be switched, not just BitBLAS.

@LeiWang1999 LeiWang1999 merged commit 393c53e into microsoft:main Aug 28, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant