Update learning_rate type to float in learning_rate_scheduler's logs #649
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This merge request updates the
learning_rate
type tofloat
in thelearning_rate_scheduler
's logs for the following reasons:Reason 1:
Currently, when the
learning_rate
is not converted to floats, the history logs show that we are using the same parameter value, resulting in a display like this:'learning_rate': [Parameter containing:
tensor(3.9063e-04),
Parameter containing:
tensor(3.9063e-04),
Parameter containing:
tensor(3.9063e-04),
...
Parameter containing:
tensor(3.9063e-04)]
This issue occurs we are keeping the parameter type throughout the epochs. Consequently, the parameter type remains the same, but its value changes dynamically as the training progresses. As a result, only the last value is visible, and we lose the history of the learning rate.
Reason 2:
The current parameter type is not serializable, which can cause issues when using certain callbacks, such as WandbMetricsLogger callback, which requires serializable data. By updating the
learning_rate
type tofloat
, we resolve this serialization issue and ensure smooth integration with such callbacks.Proposed Solution:
By converting the
learning_rate
to floats in the logs, we preserve the entire history of the learning rate, making it easier to track and analyze changes. Additionally, we address the serialization problem, enabling seamless integration with callbacks that require serializable data.Impact and Testing:
These changes do not impact the functionality of the learning rate scheduler itself. The update simply ensures better logging and serialization. I have added a unit test to ensure that we have the learning_rate in history as expected.