Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Legacy checkpoint read, fix checkpoint re-training issue #434

Merged
merged 2 commits into from
Oct 6, 2022
Merged

Legacy checkpoint read, fix checkpoint re-training issue #434

merged 2 commits into from
Oct 6, 2022

Conversation

maljoras
Copy link
Collaborator

@maljoras maljoras commented Oct 6, 2022 •
edited
Loading

Related issues

  • Legacy checkpoints loading did not load correctly with the new mapping scales in case of InferenceRPUConfig
  • Re-training would not update the learning of the out_scaling factors

Description

Checkpoint load modified the parameter in the tile reconstruction, which however caused a mismatch in the module._parameters dictionary, which meant that model.parameters() picked the wrong references.

This caused the model to not train after checkpoint loading, in case when out_scaling_alpha were set to be trained

Now the parameters are re-registered correctly for each load.

Details

  • tests added to check out_scaling_alpha loading
  • small fixes regarding the runners and experiments
  • bump version to 0.6.2

@maljoras maljoras requested a review from Fabio-83 October 6, 2022 14:31
@maljoras maljoras added the bug Something isn't working label Oct 6, 2022
Fabio-83
Fabio-83 approved these changes Oct 6, 2022
View reviewed changes
Copy link
Collaborator

@Fabio-83 Fabio-83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @maljoras

@maljoras maljoras merged commit 647efaa into IBM:master Oct 6, 2022
@maljoras maljoras deleted the checkpointing branch October 6, 2022 15:42
HCY-11 pushed a commit to HCY-11/aihwkit that referenced this pull request Dec 6, 2022
@maljoras @HCY-11
Legacy checkpoint read, fix checkpoint re-training issue (IBM#434)
3668e4b 
Signed-off-by: Henry Ye <yehenry11@gmail.com>
HCY-11 pushed a commit to HCY-11/aihwkit that referenced this pull request Dec 7, 2022
@maljoras @HCY-11
Legacy checkpoint read, fix checkpoint re-training issue (IBM#434)
f9d7c22 
Signed-off-by: Henry Ye <yehenry11@gmail.com>
HCY-11 pushed a commit to HCY-11/aihwkit that referenced this pull request Dec 7, 2022
@maljoras @HCY-11
Legacy checkpoint read, fix checkpoint re-training issue (IBM#434)
eb9d38b 
Signed-off-by: Henry Ye <yehenry11@gmail.com>
HCY-11 pushed a commit to HCY-11/aihwkit that referenced this pull request Dec 7, 2022
@maljoras @HCY-11
Legacy checkpoint read, fix checkpoint re-training issue (IBM#434)
c1ccc01 
Signed-off-by: Henry Ye <yehenry11@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Fabio-83 Fabio-83 Fabio-83 approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maljoras @Fabio-83