Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

all_gather for TPU doesn't support backward gradients. #6295

Closed
tchaton opened this issue Mar 2, 2021 · 5 comments · Fixed by #15003
Closed

all_gather for TPU doesn't support backward gradients. #6295

tchaton opened this issue Mar 2, 2021 · 5 comments · Fixed by #15003
Labels
accelerator: tpu Tensor Processing Unit feature Is an improvement or enhancement good first issue Good for newcomers help wanted Open to be worked on won't fix This will not be worked on

Comments

@tchaton
Copy link
Contributor

tchaton commented Mar 2, 2021 •
edited by edenlightning
Loading

Currently, we rely on AllGatherGrad to compute gather for GPUs.

TODO:

  • [] Extend this class to support TPU
  • [] Add tests
@tchaton tchaton added bug Something isn't working help wanted Open to be worked on good first issue Good for newcomers labels Mar 2, 2021
@edenlightning edenlightning added the accelerator: tpu Tensor Processing Unit label Mar 2, 2021
This was referenced Mar 18, 2021
Closed
Merged
@edenlightning edenlightning added the duplicate This issue or pull request already exists label Mar 19, 2021
@ethanwharris
Copy link
Member

Although all_gather will now work, it still doesn't support grad. so re-opening :)

@ethanwharris ethanwharris reopened this Mar 19, 2021
@ethanwharris ethanwharris removed the duplicate This issue or pull request already exists label Mar 19, 2021
@edenlightning edenlightning added this to the v1.3 milestone Apr 27, 2021
@edenlightning edenlightning added priority: 2 Low priority task feature Is an improvement or enhancement and removed bug Something isn't working priority: 2 Low priority task labels Apr 27, 2021
@edenlightning edenlightning modified the milestones: v1.3, v1.4 Apr 28, 2021
@vatch123
Copy link
Contributor

Is anyone looking into this? I would like to take it up but I would need some help fixing this.

@edenlightning
Copy link
Contributor

sorry for the late reply, @vatch123 would be great if you can tackle this! @kaushikb11 can prob help.

@edenlightning edenlightning removed this from the v1.4 milestone Jul 1, 2021
@vatch123
Copy link
Contributor

Ya sorry for the delayed response. Would love to start on this if someone can describe the issue a little more. Like what is the current state and what do we want to achieve? Thanks

@stale
Copy link

stale bot commented Aug 17, 2021

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

@stale stale bot added the won't fix This will not be worked on label Aug 17, 2021
@stale stale bot closed this as completed Aug 24, 2021
@stekiri stekiri mentioned this issue Oct 5, 2022
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accelerator: tpu Tensor Processing Unit feature Is an improvement or enhancement good first issue Good for newcomers help wanted Open to be worked on won't fix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make gradients available for all_gather on TPU
5 participants
@ethanwharris @tchaton @vatch123 @kaushikb11 @edenlightning