You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
generate_artifacts fails when creating training artifacts for a model using external data and MSELoss.
The use of a global base model when creating new training Blocks and onnx.save destroying any external data means any loss block (e.g. MSELoss) that builds more than one sub-Block will fail validation due to missing external data and raise an exception.
Fix
Saving using a deep copy of the global model circumvents this at the cost of holding 2x the model size in memory.
Other Implementations
An alternative approach using less memory would load the on-disk external data before it is deleted in Block::__del__ and insert the appropriate fields into the global ModelProto.
This seems a bit brittle due to the coupling to the specific way external data is destructively accessed in onnx.save. If there exists a non-modifying save in the onnx repo it would be ideal to use that in Block::__call__ instead.
The use of a global base model when creating new training `Blocks`
and `onnx.save` destroying any external data meant any loss block
(e.g. `MSELoss`) that builds more than one sub-`Block` will fail
validation due to missing external data.
Saving using a deep copy of the global model circumvents this.
Fixesmicrosoft#22411
/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline
/azp run Windows CPU CI Pipeline, Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline
trainingissues related to ONNX Runtime training; typically submitted using template
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
snnn
added
the
training
Description
generate_artifactsfails when creating training artifacts for a model using external data andMSELoss.The use of a global base model when creating new training
Blocksandonnx.savedestroying any external data means any loss block (e.g.MSELoss) that builds more than one sub-Blockwill fail validation due to missing external data and raise an exception.Fix
Saving using a deep copy of the global model circumvents this at the cost of holding 2x the model size in memory.
Other Implementations
An alternative approach using less memory would load the on-disk external data before it is deleted in
Block::__del__and insert the appropriate fields into the globalModelProto.This seems a bit brittle due to the coupling to the specific way external data is destructively accessed in
onnx.save. If there exists a non-modifying save in theonnxrepo it would be ideal to use that inBlock::__call__instead.Motivation and Context
Fixes
generate_artifactsbug reported in #22411