Multi-GPU inference

[shivangsharma1]

Hi Team,

I have trained a t5/mt5 hugging face model, I am looking for a way to to inference 1Million examples on multiple GPU. I was able to inference using single GPU but I want a way to load the pretrained saved huggingface model and do multi-GPU inference and save it at last.

Can someone please share a script to do the process?

[pacman100]

Hello @shivangsharma1, please go through this resolution of same issue: #754 (comment)

[shivangsharma1]

Hi @pacman100 ,
I have reviewed that code, I didn't find much help from the script. Would it be possible for you share a small script, as I am new to this Accelerator Api.

[ZhiyuanChen]

Basically you just need to call accelerator.prepare on your model and dataloader, and everything else is set up for you

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.