Add TorchScriptWrapper_v1 #802
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PyTorch `Module`s can be converted to TorchScript. TorchScript has the advantage that the model is serialized with the model parameters in a portable manner. Deserialization (in contrast to pickling) does not require certain Python types to be available. In fact, a TorchScript module can be loaded in C++ without a Python interpreter. For Thinc and spaCy, supporting TorchScript has two main benefits: 1. Since the model is serialized, we don't run into the issue that we have with the PyTorch wrapper where we have to construct the PyTorch model before deserializing its parameters but we can only know the model shapes through deserialization. 2. When a model is rewritten for e.g. quantization, it is unwieldy to construct the rewritten model by hand. So, with the TorchScript wrapper we would first have to construct the original model and then reapply the graph transformation. This is quite bad for transformations that reduce the model size, since we end up temporarily allocating all parameters. This is not an issue with TorchScript, since we serialize the rewritten model.
shadeMe
approved these changes
Nov 16, 2022
|
@danieldk : can you have a look at the conflicts? |
Fixed. |
svlandeg
added
enhancement
svlandeg
reviewed
Nov 22, 2022
|
I put the PR in draft for the moment, since I am still wondering if it's best to pass |
As discussed, I'd keep it as is. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PyTorch
Modules can be converted to TorchScript. TorchScript has the advantage that the model is serialized with the model parameters in a portable manner. Deserialization (in contrast to pickling) does not require certain Python types to be available. In fact, a TorchScript module can be loaded in C++ without a Python interpreter.For Thinc and spaCy, supporting TorchScript has two main benefits:
Since the model is serialized, we don't run into the issue that we have with the PyTorch wrapper where we have to construct the PyTorch model before deserializing its parameters but we can only know the model shapes through deserialization.
When a model is rewritten for e.g. quantization, it is unwieldy to construct the rewritten model by hand. So, with the TorchScript wrapper we would first have to construct the original model and then reapply the graph transformation. This is quite bad for transformations that reduce the model size, since we end up temporarily allocating all parameters. This is not an issue with TorchScript, since we serialize the rewritten model.