BUG: Regression on DataFrame.from_records

[felixdivo]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

from numpy import empty, array
from pandas import DataFrame

structured_dtype = [('prop', int)]

# Does NOT work any more
result = empty((0, len(structured_dtype)))
structured_array = array(result, dtype=structured_dtype)
DataFrame.from_records(structured_array)

Triggers: ValueError: If using all scalar values, you must pass an index.

Problem description

This previously worked on Pandas version 1.2.5.

Expected Output

No exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-77-generic
Version : #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.3.0
numpy : 1.21.0
pytz : 2021.1
dateutil : 2.8.1
pip : 20.0.2
setuptools : 57.0.0
Cython : None
pytest : 6.2.4
hypothesis : 6.14.0
sphinx : 4.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.7.0
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mzeitlin11]

Thanks for reporting this change @felixdivo! Bisection indicates #40121 as causing this change, doesn't look intended from a brief glance. Further investigations welcome!

[felixdivo]

@jbrockmendel Do you have an idea (as the author of #40121)?

[jbrockmendel]

Tentatively looks like in _extract_index we might want the is_list_like(val) and getattr(val, "ndim", 1) == 1 condition to allow ndim >= 1

[phofl]

This supresses the error currently raised but causes another later on

Traceback (most recent call last):
  File "/home/developer/.config/JetBrains/PyCharm2021.1/scratches/scratch.py", line 13, in <module>
    pd.DataFrame.from_records(structured_array)
  File "/home/developer/PycharmProjects/pandas/pandas/core/frame.py", line 2118, in from_records
    mgr = arrays_to_mgr(arrays, arr_columns, result_index, columns, typ=manager)
  File "/home/developer/PycharmProjects/pandas/pandas/core/internals/construction.py", line 124, in arrays_to_mgr
    arrays = _homogenize(arrays, index, dtype)
  File "/home/developer/PycharmProjects/pandas/pandas/core/internals/construction.py", line 584, in _homogenize
    val = sanitize_array(
  File "/home/developer/PycharmProjects/pandas/pandas/core/construction.py", line 577, in sanitize_array
    subarr = _sanitize_ndim(subarr, data, dtype, index, allow_2d=allow_2d)
  File "/home/developer/PycharmProjects/pandas/pandas/core/construction.py", line 628, in _sanitize_ndim
    raise ValueError("Data must be 1-dimensional")
ValueError: Data must be 1-dimensional

Process finished with exit code 1