Change embedding backward op output data format setup #1143
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pmarkovicTT
requested review from
nvukobratTT,
pilkicTT and
dgolubovicTT
as code owners
|
||||||||||||||
|
||||||||||||||
pilkicTT
requested changes
Jan 30, 2025
|
||||||||||||||
|
||||||||||||||
pilkicTT
approved these changes
Jan 31, 2025
|
||||||||||||||
|
||||||||||||||
|
One more thing, please change the title so it reflects that this is only related to |
pmarkovicTT
changed the title
vwellsTT
pushed a commit
that referenced
this pull request
Feb 4, 2025
### Ticket Close #1142 ### Problem description The current logic for setting up output data format for backward ops selects data format of the first input operand. Reference: `void OpNode::set_output_df_from_operands(const Graph *graph)` in the `node_types.cpp`. This is problematic since different ops can require output to match data format of some other input operand. For instance, the embedding backward op should match data format of embedding weights (the second operand). ### What's changed Changed the way embedding backward op is created (explicitly setting up output data format through autogradContext). ### Checklist - [x] Add set_output_df method to autogradContext - [x] Updated backward method for embedding op (`embedding.py`) - [x] Test new logic (test_llama_backward) - note here, couldn't find explicit tests for adding new functions so test with existing one.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Close #1142
Problem description
The current logic for setting up output data format for backward ops selects data format of the first input operand.
Reference:
void OpNode::set_output_df_from_operands(const Graph *graph)in thenode_types.cpp.This is problematic since different ops can require output to match data format of some other input operand. For instance, the embedding backward op should match data format of embedding weights (the second operand).
What's changed
Changed the way embedding backward op is created (explicitly setting up output data format through autogradContext).
Checklist
embedding.py)