FIX? df.__repr__ takes for ever with big data

[dengemann]

Hi all,

after having updated my local branch with the current dev master ( pd.version == '0.11.0rc1' -- compiling under Mac OS 10.8 went smoothly) some of my scripts in which I handle large data frames (2.6 million rows, 10 columns) stopped working.
It turned out that accessing the df's repr method but also looking up docstrings seems to make ipython unresponsive and creates the impression as if it takes forever.

This little snippet reproduces the error with fake data ('tested' with EPD Python and IPython 7.2 (32-bit)):

import numpy as np
import pandas as pd

data  = np.random.random((3000000, 10))
df = pd.DataFrame(data)

print df  # does the same as `df.__repr__()` or `df?`

Can anyone confirm this?
It must have been some of the more recent changes as I did not have trouble with this earlier.
Any pointers and comments are highly welcome.

Cheers,
Denis

[ghost]

see #3326 (comment)
should be fixed by the time 0.11.0 final is out.

In the meantime, git reset --hard 7db1af4^ will get you practically everythin new
in 0.11.0.

Thanks for reporting it, wil leave open till it's sorted.

[ghost]

Can you provide a testcase for the docstring issue you mention? that's new.

[dengemann]

Thanks for your quick reply. Unfortunately I cannot give you a pure Python example as DataFrame instances don't employ But doing df? or df?? in IPython does the 'trick'. Maybe that's the point, IPython will certainly employ some inspection magic to find out about the doc string. Would it make sense to move the doc string from under DataFrame.init to DataFrame, so it's possible to referred to it as DataFrame.doc ?

[ghost]

The IPython docstring problem seems to be part of the same issue, so
there's probably no need for that, they'll both get fixed together.

[dengemann]

Yes, makes sense. Just out of curiosity, wouldn't it be more consistent with other scientific / data analysis packages in Python to have non-empty .doc attributes? Or is this already being addressed somewhere? (sorry, I'm quite new to tracking the dev version of pandas and it's not so easy to get an overview on the open issues)

[ghost]

I guess we all use ?? rather then doc , and IPython context help finds
it as well, so I never noticed that before.

Sure. PR welcome.

[dengemann]

I guess we all use ?? rather then doc , and IPython context help finds it as well, so I never noticed that before.

I agree, neither did I. When using the debugger however, IPython features aren't available and:

print foo.doc

is a helpful idiom.

Sure. PR welcome.

ok

[dengemann]

@y-p I will look for this 'issue' in the entire library. where ever I see pep8 violations I also would address those. (?)

[ghost]

Have a read on CONTRIBUTING.md for various notes, including plus and minuses for
"PEP8 storms".

[dengemann]

I just found that many violations are systematic and rather appear as
'dialects', thanks for the pointers, I'll take look.

On Sat, Apr 13, 2013 at 3:01 PM, y-p notifications@github.com wrote:

Have a read on CONTRIBUTING.md for various notes, including plus and
minuses for
"PEP8 storms".

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3337#issuecomment-16332706
.

[hayd]

@y-p I always feel like a PEP8 storm... :)

[ghost]

I guess it's not bad to hit the 'reset' button once per release, but more then that
becomes disruptive. I use emacs and dangling whitespace from lesser editors
is a constant annoyance too, so "formatting storms" are not all bad.

[dengemann]

After taking a tour I've learned ended up wondering about a few things.
It looks like local style to not separate args in a function and not to drop white spaces around kwargs.
also the typical 2 lines between functions rule is not complied with consistently.
Most importantly on many many instances lines are far beyond 80 characters.
I think this calls for a separate PR.
It would be good for new contributors to have a more detailed style guideline including a minimum agreed-upon set of pep8 rules. This might also keep down pep8 storms. If e.g. the 2 lines between functions rule is not among the core style rules then may can simply ignore it (despite eyes bleeding from the editor's pep8 mode).

Wdyt?

[ghost]

So CONTRIBUTING.md didn't convince you. oh well. :)

[dengemann]

So CONTRIBUTING.md didn't convince you. oh well. :)

Admittedly not, was more of a deterrent to me ;-)

[dengemann]

(with regard to pep8 interventions)

[dengemann]

Btw. a few exceptions to pep8 which seem to have the status of a local pandas convention are:

1. White spaces in list-comprehensions and expressions

• [ k for k in my_list ]
• [ v ]

see e. g. : https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L558

2. adding white spaces to equalize column count across a series of assignments belonging together (config file style)

e. g. https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L840

These patterns seem to emerge from time to time, but too frequently to look like unwanted.
So I wouldn't be sure whether to touch them or not in a pep8 storm ;-)

Without complicating things one could address this CONTRIBUTING.md, given there is exceptions one likes to keep un-pep8-ted.

[lodagro]

The DataFrame repr behavior is now such that, conscise formats are only used in interactive mode (see also #3326, discussion not yet closed). Interactive mode can be enforced by using pd.options.mode.sim_interactive = True.

(pandas)[lodagro@ubuntu][1193][n] cat issue_3337.py                                                  [~/projects/pandas/issues]
import numpy as np
import pandas as pd

pd.options.mode.sim_interactive = True

data  = np.random.random((3000, 10))
df = pd.DataFrame(data)

print df
(pandas)[lodagro@ubuntu][1194][n] python issue_3337.py                                               [~/projects/pandas/issues]
<class 'pandas.core.frame.DataFrame'>
Int64Index: 3000 entries, 0 to 2999
Data columns (total 10 columns):
0    3000  non-null values
1    3000  non-null values
2    3000  non-null values
3    3000  non-null values
4    3000  non-null values
5    3000  non-null values
6    3000  non-null values
7    3000  non-null values
8    3000  non-null values
9    3000  non-null values
dtypes: float64(10)
(pandas)[lodagro@ubuntu][1195][n]    

[ghost]

I really don't think the change in behaviour makes sense.
repr() is not a report-generation function. a repr() should be bounded in length,
with the user having some control over the bound. IMO, this part should be rolled back.

Even if it were, it's now established behaviour so this change will break people's scripts.
it already has.

sim_interactive is stricly for testing purposes only, people should not use it for anything else.

[wesm]

Status?

[ghost]

broken.

[ghost]

fixed by b9fa04a