Fix edge cases in (de)serialize_torch_tensor #591
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mryab
approved these changes
Sep 5, 2023
mryab
changed the title
Codecov Report
@@ Coverage Diff @@
## master #591 +/- ##
==========================================
+ Coverage 85.20% 85.37% +0.17%
==========================================
Files 81 81
Lines 8009 8022 +13
==========================================
+ Hits 6824 6849 +25
+ Misses 1185 1173 -12
|
justheuristic
changed the title
mryab
approved these changes
Sep 5, 2023
hivemind/compression/floating.py
Outdated
| @@ -12,22 +12,28 @@ class Float16Compression(CompressionBase): | |||
| FP16_MIN, FP16_MAX = torch.finfo(torch.float16).min, torch.finfo(torch.float16).max | |||
|
|
|||
| def compress(self, tensor: torch.Tensor, info: CompressionInfo, allow_inplace: bool = False) -> runtime_pb2.Tensor: | |||
| assert torch.is_floating_point(tensor) and tensor.dtype != torch.bfloat16 | |||
There was a problem hiding this comment.
Is there a reason why we should fail with an error in case of bf16 inputs? It is indeed not sensible, but if the user wants to do so, it's probably better to issue a warning instead of flat out refusing to pass that through quantization
There was a problem hiding this comment.
added ValueError with a more user-legible reason
hivemind/compression/quantization.py
Outdated
| @@ -135,14 +138,15 @@ def quantize( | |||
| except ImportError: | |||
| raise ImportError(BNB_MISSING_MESSAGE) | |||
|
|
|||
| quantized, (absmax, codebook) = quantize_blockwise(tensor) | |||
| quantized, (absmax, codebook, *extra_params) = quantize_blockwise(tensor, blocksize=4096, nested=False) | |||
| assert tuple(extra_params) == (4096, False, tensor.dtype, None, None) # blocksize, nested, dtype, offset, s2 | |||
There was a problem hiding this comment.
Maybe we can make that tuple on the right a module-level constant? It's used twice in the code, better to make it clear we're using some predefined values
There was a problem hiding this comment.
done, thanks for the suggestion
jmikedupont2
pushed a commit
to meta-introspector/hivemind
that referenced
this pull request
Mar 29, 2024
* serialize with requires_grad * ensure that all compression methods return tensor of the original dtype * test that all compression methods preserve dtype and requires_grad --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: Max Ryabinin <mryabinin0@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During an earlier patch, we lost the requires_grad property during serialize_torch_tensor. This PR adds it back.