Unexpected to_dict output after upgrading to 0.24.1

[atlasstrategic]

Code to illustrate 0.23.4 versus 0.24.1

pandas 0.23.4

import pandas as pd

columns = ["A", "B"]
pd.DataFrame(columns=columns).transpose().to_dict(orient="split")
# output
# {'index': ['A', 'B'], 'columns': [], 'data': [[], []]}

pd.DataFrame(columns=columns).transpose().values.tolist()
# output
# [[], []]

pandas 0.24.1

import pandas as pd

columns = ["A", "B"]
pd.DataFrame(columns=columns).transpose().to_dict(orient="split")
# output not consistent with 0.23.4
# {'index': ['A', 'B'], 'columns': [], 'data': []}

pd.DataFrame(columns=columns).transpose().values.tolist()
# output is consistent with 0.23.4
# [[], []]

Problem description

After upgrading to Pandas 0.24.1, the expected output when doing a transpose, followed by a to_dict has changed. Before, an empty DataFrame was returning an empty list for each column (similar to values.tolist()) e.g. 'data': [[]. []]. Now the output is 'data': [].

Expected Output

Refer to code and output for 0.23.4 above.

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_ZA.UTF-8
LOCALE: en_ZA.UTF-8

pandas: 0.24.1
pytest: None
pip: 19.0.2
setuptools: 40.4.3
Cython: None
numpy: 1.14.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: 2.5.12
xlrd: None
xlwt: None
xlsxwriter: 1.1.2
lxml.etree: None
bs4: 4.7.1
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.7 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[jorisvandenbossche]

Thanks for the report!

We had a regression in to_dict in 0.24.0, which should be fixed in 0.24.1 (#24965). But apparently there are some more corner cases broken as well (although this is certainly much more a corner case where the correct answer is less obvious I would say that the other ones we have fixed).

This is another consequence of the usage of itertuples:

In [13]: df = pd.DataFrame(columns=['A', 'B']).transpose()                                                                                                                      

In [14]: list(df.itertuples(name=None, index=False))                                                                                                                            
Out[14]: []

In [15]: list(df.itertuples(name=None, index=True))                                                                                                                             
Out[15]: [('A',), ('B',)]

You could say that the output [] should actually be [(), ()] (which would solve the to_dict problem as well).

[MarcoGorelli]

take

[MarcoGorelli]

You could say that the output [] should actually be [(), ()]

What if name is not None? Would we expect

[Pandas(_0=()), Pandas(_0=())]

?