BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47319

Kelvingandhi · 2022-06-12T18:27:07Z

closes BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47285(Replace xxxx with the Github issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

…ssue pandas-dev#47285

…rable issue pandas-dev#47285

pep8speaks · 2022-06-12T18:27:12Z

Hello @Kelvingandhi! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-06-24 20:23:11 UTC

simonjayhawkins · 2022-06-13T11:27:15Z

cc @phofl (from issue)

doc/source/whatsnew/v1.5.0.rst

phofl · 2022-06-13T11:33:13Z

pandas/core/frame.py

+                    if isinstance(index, str) or not hasattr(index, "__iter__"):
+                        if index not in columns:
+                            result_index = [index]
+                            index = None


This swallows KeyErrors when the iterable is populated but the column is not present.

data = [ {"col_1": 3, "col_2": "a"}, {"col_1": 2, "col_2": "b"}, {"col_1": 1, "col_2": "c"}, {"col_1": 0, "col_2": "d"}, ] result = pd.DataFrame.from_records(data, index="col_14") print(result)

This should still raise, I think.

cc @simonjayhawkins

I think that maybe is another bug, see #47285 (comment) where we get NaN in the index for a missing key.

On main, this raises a KeyError right now, which is fine I think?

i.e. if we do have data (not the empty list case) we should probably raise on missing keys

Yep I think so too.

i.e. if we do have data (not the empty list case) we should probably raise on missing keys

Yeah agree with you.
Furthermore, when we have EMPTY list and we pass some list index then currently on Main, it returns that list index as output with empty list. We want same behavior for string index as well for EMPTY list instead of raising KeyError.

This swallows KeyErrors when the iterable is populated but the column is not present.

data = [ {"col_1": 3, "col_2": "a"}, {"col_1": 2, "col_2": "b"}, {"col_1": 1, "col_2": "c"}, {"col_1": 0, "col_2": "d"}, ] result = pd.DataFrame.from_records(data, index="col_14") print(result)

This should still raise, I think.

cc @simonjayhawkins

As @simonjayhawkins mentioned, and I also tested it, it generates ValueError: Length of values (4) does not match length of index (1) when we've iterable populated. And if it should raise KeyError then that is another issue.

Here, in my PR, I tried to address actual issue when we have empty list ([]) and we pass as string index, It gives KeyError. But Ideally in output we want Empty list along with passed string as index.

We want Exact same behaviour when we have EMPTY list and we pass some list index then currently on Main, it returns that list index as output with empty list.

Code change I put here, it will swallow KeyError only when we have EMPTY list passed with string index. Otherwise it won't get executed.

But please correct me if I misunderstood any of your concerns in discussion. Thanks.

My code example should still raise the same error. We do not want to change this behavior.

You have to fix the bug without breaking anything else.

Please also add my example as a test with the KeyError as expected

Ohk. I apologize, earlier I misunderstood it. I fixed this issue and now it is behaving as expected (exactly what currently in main this raises a KeyError). I have added your example as a test as well. Thank you.

Kelvingandhi · 2022-06-19T20:50:44Z

Hello @simonjayhawkins , Just wanted to follow up on this PR. I'm not sure, is it waiting for another approval or is it being checked or need more discussion. If you could provide your feedback, that would be appreciated.

phofl · 2022-06-19T20:52:35Z

Did you address our comments?

Kelvingandhi · 2022-06-19T22:09:48Z

Did you address our comments?

Please check my response above.

#47319 (comment)

…ror as expected

…ror, with code formatting

simonjayhawkins · 2022-06-20T12:40:45Z

@github-actions pre-commit

pandas/tests/io/formats/test_info.py

doc/source/whatsnew/v1.5.0.rst

phofl · 2022-06-24T14:01:04Z

pandas/tests/io/formats/test_info.py

@@ -491,3 +491,19 @@ def test_info_int_columns():
        """
    )
    assert result == expected
+
+
+def test_info_from_rec():


Could you please add the test we discussed where the behavior changed initially?

Also pre-commit is failing

After our discussion, I fixed that issue and also added that test in test_info.py (please check lines 502-509) plus in that test case, I made correction as well suggested by @simonjayhawkins (#47319 (comment)).

Furthermore, I already have initial test case included as well (lines 498-500).

I'm unsure what else is required.

On pre-commit failure, I'm beginner here and not sure what causing it to fail (when last time it passed for earlier commits). Any help to get this resolved would be appreciated.

Ah sorry, missed that. Could you split that into two tests? Since technically they are testing different things.

You can run pre-commit locally and check the log. This should show the errors

Okay. I will do that. Thank you for the response.

Kelvingandhi · 2022-06-24T21:57:30Z

All checks are successful except these 2 which are keep getting canceled after 90 min timeout. Not sure what action is needed now.

Windows-MacOS / windows-latest actions-38.yaml (pull_request)
Windows-MacOS / windows-latest actions-310.yaml (pull_request)

Kelvingandhi · 2022-07-10T18:11:43Z

@phofl @simonjayhawkins Have you gotten a chance to look at this 2 checks getting cancelled issue. #47319 (comment)

yuanx749 · 2022-08-01T17:15:18Z

pandas/core/frame.py

+
+                if len(arrays) == 0:
+                    if index is not None:
+                        if isinstance(index, str) or not hasattr(index, "__iter__"):
+                            if index not in columns:
+                                # error: Incompatible types in assignment
+                                # type "List[Any]"; expected "Optional[Index]"
+                                result_index = [index]  # type: ignore[assignment]
+                                index = None
+


I would use only one if here to be consistent with other code.

Suggested change

if len(arrays) == 0:

if index is not None:

if isinstance(index, str) or not hasattr(index, "__iter__"):

if index not in columns:

# error: Incompatible types in assignment

# type "List[Any]"; expected "Optional[Index]"

result_index = [index] # type: ignore[assignment]

index = None

if (

len(arrays) == 0

and index is not None

and (isinstance(index, str) or not hasattr(index, "__iter__"))

and index not in columns

):

# error: Incompatible types in assignment

# type "List[Any]"; expected "Optional[Index]"

result_index = [index] # type: ignore[assignment]

index = None

I did check that if condition and earlier I had both conditions with one if but it was failing for other case where we pass this pd.DataFrame.from_records([]) (it was failing for if index is not None condition)

Basically we can’t add these conditions in one if to filter out specific edge case.

github-actions · 2022-09-04T00:06:05Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

mroeschke · 2022-09-21T18:06:24Z

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

Kelvingandhi added 2 commits June 12, 2022 14:22

fix bug in dataframe.from_records with str index and empty iterable i…

8c273a1

…ssue pandas-dev#47285

fix bug in dataframe.from_records method with str index and empty ite…

48960d4

…rable issue pandas-dev#47285

fix blank lines PEP 8 issue in test_info.py

149bb59

simonjayhawkins changed the title ~~Bug/47285 from records key error~~ BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable Jun 13, 2022

simonjayhawkins added Bug Constructors Series/DataFrame/Index/pd.array Constructors labels Jun 13, 2022

simonjayhawkins reviewed Jun 13, 2022

View reviewed changes

doc/source/whatsnew/v1.5.0.rst Outdated Show resolved Hide resolved

phofl reviewed Jun 13, 2022

View reviewed changes

Kelvingandhi added 2 commits June 19, 2022 20:27

fix behavior on iterable with data and wrong index to raise the KeyEr…

cb91d1d

…ror as expected

fix behavior on iterable with data and wrong index to raise the KeyEr…

a87ef23

…ror, with code formatting

Kelvingandhi and others added 2 commits June 20, 2022 12:42

Fixes from pre-commit [automated commit]

17b4efc

Update doc/source/whatsnew/v1.5.0.rst

1651372

simonjayhawkins reviewed Jun 20, 2022

View reviewed changes

pandas/tests/io/formats/test_info.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed Jun 20, 2022

View reviewed changes

doc/source/whatsnew/v1.5.0.rst Outdated Show resolved Hide resolved

fix tests with suggestions and typos in doc

2b4b0d4

phofl reviewed Jun 24, 2022

View reviewed changes

Kelvingandhi added 2 commits June 24, 2022 12:57

Split test cases and fixed pre-commit failure

8337706

Fixed Typing Validation Error in frame.py

0abc57a

yuanx749 reviewed Aug 1, 2022

View reviewed changes

github-actions bot added the Stale label Sep 4, 2022

mroeschke closed this Sep 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47319

BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47319

Kelvingandhi commented Jun 12, 2022 •

edited by simonjayhawkins

Loading

pep8speaks commented Jun 12, 2022 •

edited

Loading

simonjayhawkins commented Jun 13, 2022

phofl Jun 13, 2022

simonjayhawkins Jun 13, 2022

phofl Jun 13, 2022

simonjayhawkins Jun 13, 2022

phofl Jun 13, 2022

Kelvingandhi Jun 13, 2022 •

edited

Loading

Kelvingandhi Jun 19, 2022 •

edited

Loading

phofl Jun 19, 2022

phofl Jun 19, 2022

Kelvingandhi Jun 20, 2022 •

edited

Loading

Kelvingandhi commented Jun 19, 2022 •

edited

Loading

phofl commented Jun 19, 2022

Kelvingandhi commented Jun 19, 2022

simonjayhawkins commented Jun 20, 2022

phofl Jun 24, 2022

Kelvingandhi Jun 24, 2022 •

edited

Loading

phofl Jun 24, 2022

Kelvingandhi Jun 24, 2022

Kelvingandhi commented Jun 24, 2022 •

edited

Loading

Kelvingandhi commented Jul 10, 2022

yuanx749 Aug 1, 2022

Kelvingandhi Aug 3, 2022 •

edited

Loading

github-actions bot commented Sep 4, 2022

mroeschke commented Sep 21, 2022

BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47319

BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable #47319

Conversation

Kelvingandhi commented Jun 12, 2022 • edited by simonjayhawkins Loading

pep8speaks commented Jun 12, 2022 • edited Loading

Comment last updated at 2022-06-24 20:23:11 UTC

simonjayhawkins commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kelvingandhi Jun 13, 2022 • edited Loading

Choose a reason for hiding this comment

Kelvingandhi Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kelvingandhi Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

Kelvingandhi commented Jun 19, 2022 • edited Loading

phofl commented Jun 19, 2022

Kelvingandhi commented Jun 19, 2022

simonjayhawkins commented Jun 20, 2022

Choose a reason for hiding this comment

Kelvingandhi Jun 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kelvingandhi commented Jun 24, 2022 • edited Loading

Kelvingandhi commented Jul 10, 2022

Choose a reason for hiding this comment

Kelvingandhi Aug 3, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 4, 2022

mroeschke commented Sep 21, 2022

Kelvingandhi commented Jun 12, 2022 •

edited by simonjayhawkins

Loading

pep8speaks commented Jun 12, 2022 •

edited

Loading

Kelvingandhi Jun 13, 2022 •

edited

Loading

Kelvingandhi Jun 19, 2022 •

edited

Loading

Kelvingandhi Jun 20, 2022 •

edited

Loading

Kelvingandhi commented Jun 19, 2022 •

edited

Loading

Kelvingandhi Jun 24, 2022 •

edited

Loading

Kelvingandhi commented Jun 24, 2022 •

edited

Loading

Kelvingandhi Aug 3, 2022 •

edited

Loading