-
Notifications
You must be signed in to change notification settings - Fork 467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[torchbench] moco
fails to run.
#6083
Comments
Is this one still relevant? |
Yes. I got it in the last benchmark run. |
I think I finally understood what was going wrong, here. By setting
At first, I thought that there was not dispatch registered for I think that the way to make it work is to actually run the fallback on CUDA, since that was the device we initialized |
This sounds reasonable to me. We can try to land fallbacks to CUDA first and then revisit this one after that |
..hmm why is this model actually uses c10d.. is it doing multi device training? |
I guess that's a fair question. I'm not sure why it was implemented like that, since they set the |
On second thoughts, maybe this is a case we should change the benchmark code for initializing the |
yea that's worth a try.. through my impression is still that torchbench only do single process benchmark which made me wonder why there is even distributed stuff involved. |
🐛 Bug
Running the upstreamed benchmarking scripts with the following command results in an unexpected error.
python xla/benchmarks/experiment_runner.py \ --suite-name torchbench \ --accelerator cuda \ --xla PJRT --xla None \ --dynamo openxla --dynamo None \ --test eval --test train \ --repeat 30 --iterations-per-run 5 \ --print-subprocess \ --no-resume -k moco
Environment
The text was updated successfully, but these errors were encountered: