Fix endianness in array of float/double #11
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@irmen Is it possible to have an bugfix release soon? |
|
Haven't had the ability yet to verify this issue. What platform are you running on that gave you problems? |
|
In Mac OS X 10.9, Python 3.4, The array('f', [16711938.0]) is pickled with little endian, it cannot be unpickled by Pyrolite. >>> struct.pack('<f', 16711938.0)
b'\x02\x01\x7fK'
>>> struct.pack('f', 16711938.0)
b'\x02\x01\x7fK'
>>> struct.pack('f', 16711938.0) in pickle.dumps(array.array('f',[16711938.0]))
True |
irmen
added a commit
that referenced
this pull request
Sep 15, 2014
Fix endianness in array of float/double
|
Thanks for the pull request |
guavuslabs-builder
pushed a commit
to ThalesGroup/spark
that referenced
this pull request
Sep 16, 2014
Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. I had send an PR to Pyrolite to fix it: irmen/Pyrolite#11 Author: Davies Liu <davies.liu@gmail.com> Closes apache#2365 from davies/pickle and squashes the following commits: f44f771 [Davies Liu] enable tests about array 3908f5c [Davies Liu] Merge branch 'master' into pickle c77c87b [Davies Liu] cleanup debugging code 60e4e2f [Davies Liu] support unpickle array.array for Python 2.6
davies
added a commit
to davies/spark
that referenced
this pull request
Dec 10, 2014
Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. I had send an PR to Pyrolite to fix it: irmen/Pyrolite#11 Author: Davies Liu <davies.liu@gmail.com> Closes apache#2365 from davies/pickle and squashes the following commits: f44f771 [Davies Liu] enable tests about array 3908f5c [Davies Liu] Merge branch 'master' into pickle c77c87b [Davies Liu] cleanup debugging code 60e4e2f [Davies Liu] support unpickle array.array for Python 2.6 Conflicts: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala python/pyspark/tests.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PickleUtils.bytes_to_double() and PickleUtils.bytes_to_float() assume that the bytes is in big endian order, so it should reverse the bytes only when machine code is little endian order.
The serialized result of array of float/double used in unit tests is reversed, they had been corrected in this patch.