DOC: Serialization recommendation is deprecated

[chrisroat]

Location of the documentation

https://pandas.pydata.org/pandas-docs/dev/user_guide/io.html#io-msgpack

Documentation problem

Since the deprecation of msgpack for on-the-wire transmission, it is recommended to use pyarrow serialization/deserialization. However, since pyarrow 2.0, this has been deprecated for arbitrary objects. A deprecation message is emitted when using the documented code snippet.

$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.__version__
'3.0.0'
>>> pa.default_serialization_context()
<stdin>:1: DeprecationWarning: 'pyarrow.default_serialization_context' is deprecated as of 2.0.0 and will be removed in a future version. Use pickle or the pyarrow IPC functionality instead.
<pyarrow.lib.SerializationContext object at 0x7f23dfa36940>
>>> 

Suggested fix for documentation

Would pickle be next in line for a recommended on-the-wire format?

[jreback]

no just need to update to the renamed pyarrow format

[chrisroat]

I may not understand the situation fully -- what is renamed? The deprecation message in the pyarrow docs linked above recommends pickle for non-pyarrow objects. One can convert a dataframe to/from pyarrow table, but it may not be fully compatible.

[jorisvandenbossche]

@chrisroat thanks for the report! We should indeed have updated our docs after pyarrow deprecated the serialization functionality.

The most appropriate alternative will depend on your exact use case, but in general I think we can indeed refer users to use pickle instead.

[simonjayhawkins]

removing 1.3 milestone.

[jorisvandenbossche]

It's important to fix this, as our docs are simply pointing to a (soon) no-longer existing alternative. Since this is arrow-related, will look into it one of the next days (not crucial for the RC of course)

[jorisvandenbossche]

Opened a PR for this at #41899

[stochastic-thread]

Does anyone have any information as to why this was deprecated?
Pandas deprecated to_msgpack
Now PyArrow plans to deprecate a simple and useful serialization option.
Kind of annoying if I'm being honest.

[Neltherion]

Using Pickle5 (as suggested) doesn't seem to have the same performance as PyArrow's deprecated Serialization method. Is there ANY proper replacements for pa.serialize() and pa.deserialize() ? I hate this kind of deprecation....

[jorisvandenbossche]

@Neltherion can you show some example code that illustrates the performance difference? That might help finding out the reason / how this can be improved.

[Neltherion]

@jorisvandenbossche Here's a simplified code that compares the difference between PyArrow & Pickle when Serializing/Deserializing:

import time

import numpy as np
import pickle5
import pyarrow as pa


class Person:
    def __init__(self, Thumbnail: np.ndarray = None):
        if Thumbnail is not None:
            self.Thumbnail: np.ndarray = Thumbnail
        else:
            self.Thumbnail: np.ndarray = np.random.rand(256, 256, 3)


def serialize_Person(person):
    return {'Thumbnail': person.Thumbnail}


def deserialize_Person(person):
    return Person(person['Thumbnail'])


context = pa.SerializationContext()
context.register_type(Person, 'Person', custom_serializer=serialize_Person, custom_deserializer=deserialize_Person)

PERSONS = [Person() for i in range(100)]

"""
PyArrow
"""
t1 = time.time()
persons_serialized = pa.serialize(PERSONS, context=context).to_buffer()
persons_deserialized = pa.deserialize(persons_serialized, context=context)
t2 = time.time()
print(f'PyArrow Time => {t2 - t1}')

"""
Pickle
"""
t1 = time.time()
persons_pickled = pickle5.dumps(PERSONS, protocol=5)
persons_depickled = pickle5.loads(persons_pickled)
t2 = time.time()
print(f'Pickle Time => {t2 - t1}')

The outputs on my system are:

PyArrow Time => 0.04499983787536621
Pickle Time => 0.2220008373260498

[Neltherion]

@jorisvandenbossche Did the example help?

[jorisvandenbossche]

@Neltherion I answered at apache/arrow#11239