-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
faiss::gpu::runMatrixMult ... cublas failed (13): (1024, 12) x (256, 12)' = (1024, 256) gemm params m 256 n 1024 k 12 trA T trB N lda 12 ldb 12 ldc 256 #2064
Comments
What is your CUDA version? If it is >=11.2, have you tried on CUDA 10? |
@xzyaoi My CUDA version is 10.1. |
What kind of GPU are you using? 40 GiB makes me think of A100, which really should require CUDA 11? |
Hi @wickedfoo My cuda is 11 but still showing the error |
The same error, have you solved it now? |
I have the same error with RTX3090 |
I changed the version of faiss gpu and it worked
…On Wed, Mar 2, 2022, 08:48 tengteng-Lin ***@***.***> wrote:
I have the same error with RTX3090
—
Reply to this email directly, view it on GitHub
<#2064 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG46Y6NKW2VMSL76Q6ROPDLU53MXXANCNFSM5EUH2UCA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
@MotiBaadror Can you tell us what CUDA, faiss, faiss-gpu, etc. versions were when you finally managed to get it to work? Were you using A100 GPUs? |
I have the same error with RTX3090, please help me ??? |
I solved it by changing the version, and the current version is 1.7.2
At 2022-03-21 17:39:36, "zhoujianch" ***@***.***> wrote:
I have the same error with RTX3090, please help me ???
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you commented.Message ID: ***@***.***>
|
@tengteng-Lin thanks for your reply. |
Seeing this with A100 / CUDA 11.5 / faiss-gpu=1.7.2 |
Seeing this with A100 / CUDA 11.1 / faiss-gpu=1.7.2. The error occurs at the search step of a flat index. |
I am running into the same issue on RTX3090. Ubuntu, Driver 510.73.05; Cuda: 11.6 |
showing me this error with cuda 11.6 rtx3090 faiss-gpu=1.7.2 |
Seeing this with cuda 11.1 rtx3090 faiss-gpu=1.7.2 |
Same error with:
Trying to reinstall from scratch, upgrade or downgrade |
Same here: |
Same with |
Update: The error occurs when I use the faiss-gpu PIP package from https://github.com/kyamagu/faiss-wheels (in Rocky Linux 9 with Python 3.9 and CUDA 11.7). If I use Anaconda3 with Python 3.8 and install the faiss-gpu from pytorch conda repo with cuda 11.3 (which is the officially supported manner), the error no longer appears. Perhaps this should have been an issue in that repo instead. |
Thank you so much. I also fix this issue on A100 GPU following your suggestion. My environment is python==3.8, cuda==11.3, faiss-gpu==1.7.2, torch==1.9.1+cu111. |
Met also in the env above, but haven't tried the solution to downgrade the python version. After downgrading the python version to py38 and follow #2064 (comment) said, it works!!! |
It helped me to install a specific wheel with faiss-gpu==1.7.3:
|
I am getting the same error using |
Please install faiss-gpu with conda. pip is not supported by our team at this time. |
uninstall faiss, and print follow code: |
Summary
I'm trying to train an IVFPQ index for 100000 768-dimensional embeddings on an NVIDIA GPU with 40537MiB of memory. The code fails at
index.train()
with the following error message:Platform
OS: Ubuntu 20.04
Faiss version: faiss-gpu 1.7.1.post2
Installed from: anaconda (pip install faiss-gpu)
Faiss compilation options: Nothing explicitly
Running on:
Interface:
Reproduction instructions
The text was updated successfully, but these errors were encountered: