[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 #2365
Conversation
|
QA tests have started for PR 2365 at commit
|
davies
changed the title
|
QA tests have started for PR 2365 at commit
|
|
QA tests have finished for PR 2365 at commit
|
|
QA tests have finished for PR 2365 at commit
|
|
Maybe we should wait a couple of days to hear back from the Pyrolite folks and see if they will cut a new release. |
| } else if (args.length == 2 && args(1).isInstanceOf[String]) { | ||
| val typecode = args(0).asInstanceOf[String].charAt(0) | ||
| val data: String = args(1).asInstanceOf[String] | ||
| println(typecode, machineCodes(typecode), data.length, data.toList) |
There was a problem hiding this comment.
Can you remove this debugging statement?
|
QA tests have started for PR 2365 at commit
|
|
QA tests have finished for PR 2365 at commit
|
+1 unless there's agreed upon urgency, it'd be better to fix the dep instead of working around it |
|
where's the message that was sent to the pyrolite folks? it looks like SPARK-2378 s targetted for 1.2, so it has a bit of time |
|
if you do end up merging, what do you think about logging an issue for fixing up the workaround once pyrolite is update? |
|
I had created https://issues.apache.org/jira/browse/SPARK-3524 to track this. |
|
thanks, +1, lgtm |
|
LGTM Ran relevant python unit tests with no problems. |
|
I'm going to merge this now. As a reference / side note, http://bugs.python.org/issue2389 provides some good context for why Python 2.6's array pickling exposes the machine's endianness; Python 2.7 uses a different pickling strategy that doesn't go through this code path. |
Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. I had send an PR to Pyrolite to fix it: irmen/Pyrolite#11 Author: Davies Liu <davies.liu@gmail.com> Closes apache#2365 from davies/pickle and squashes the following commits: f44f771 [Davies Liu] enable tests about array 3908f5c [Davies Liu] Merge branch 'master' into pickle c77c87b [Davies Liu] cleanup debugging code 60e4e2f [Davies Liu] support unpickle array.array for Python 2.6 Conflicts: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala python/pyspark/tests.py
Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.
There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.
I had send an PR to Pyrolite to fix it: irmen/Pyrolite#11