Shorter truncated Series/DataFrame repr: introduce min_rows #27095

jorisvandenbossche · 2019-06-28T14:44:47Z

(didn't yet add tests, docs or whatsnew, but I think the implementation is already more or less complete)

jreback · 2019-06-28T14:45:57Z

pandas/core/frame.py

@@ -665,8 +667,8 @@ def _repr_html_(self):
    def to_string(self, buf=None, columns=None, col_space=None, header=True,
                  index=True, na_rep='NaN', formatters=None, float_format=None,
                  sparsify=None, index_names=True, justify=None,
-                  max_rows=None, max_cols=None, show_dimensions=False,
-                  decimal='.', line_width=None):
+                  min_rows=None, max_rows=None, max_cols=None,


can you add types on new args (and old ones if you can)

pandas/io/formats/format.py

TomAugspurger · 2019-07-02T21:01:01Z

@jorisvandenbossche think you'll be able to update in the next 24 hours?

codecov · 2019-07-03T03:40:22Z

Codecov Report

Merging #27095 into master will decrease coverage by 49.9%.
The diff coverage is 53.33%.

@@             Coverage Diff             @@
##           master   #27095       +/-   ##
===========================================
- Coverage   91.79%   41.89%   -49.91%     
===========================================
  Files         180      180               
  Lines       50934    50803      -131     
===========================================
- Hits        46757    21286    -25471     
- Misses       4177    29517    +25340

Flag	Coverage Δ
#multiple	`?`
#single	`41.89% <53.33%> (-0.22%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/frame.py	`34.99% <ø> (-61.94%)`	⬇️
pandas/core/series.py	`45.45% <ø> (-47.06%)`	⬇️
pandas/core/config_init.py	`80.68% <100%> (-15.39%)`	⬇️
pandas/io/formats/format.py	`50.57% <46.15%> (-47.35%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/io/gcs.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
... and 142 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ab9ff5...98e7d43. Read the comment docs.

codecov · 2019-07-03T03:40:23Z

Codecov Report

Merging #27095 into master will decrease coverage by 49.9%.
The diff coverage is 53.33%.

@@             Coverage Diff             @@
##           master   #27095       +/-   ##
===========================================
- Coverage   91.79%   41.89%   -49.91%     
===========================================
  Files         180      180               
  Lines       50934    50803      -131     
===========================================
- Hits        46757    21286    -25471     
- Misses       4177    29517    +25340

Flag	Coverage Δ
#multiple	`?`
#single	`41.89% <53.33%> (-0.22%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/frame.py	`34.99% <ø> (-61.94%)`	⬇️
pandas/core/series.py	`45.45% <ø> (-47.06%)`	⬇️
pandas/core/config_init.py	`80.68% <100%> (-15.39%)`	⬇️
pandas/io/formats/format.py	`50.57% <46.15%> (-47.35%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/io/gcs.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
... and 142 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ab9ff5...98e7d43. Read the comment docs.

jorisvandenbossche · 2019-07-03T03:49:50Z

I added some tests. Further to do:

Still need to add a whatsnew note / docs
We can decide to not add the min_rows argument to to_string, but only in the Series/DataFrameFormatter classes if we want (cc @simonjayhawkins ), by using those Formatter classes directly in __repr__ instead of calling to_string (to_string is only brief wrapper around the Formatter classes, so it would not add much code duplication)

simonjayhawkins

-1 on adding keywords to public methods. see #27000 (comment)

simonjayhawkins · 2019-07-03T09:22:53Z

pandas/core/frame.py

        max_cols = get_option("display.max_columns")
        show_dimensions = get_option("display.show_dimensions")
        if get_option("display.expand_frame_repr"):
            width, _ = console.get_console_size()
        else:
            width = None
-        self.to_string(buf=buf, max_rows=max_rows, max_cols=max_cols,
-                       line_width=width, show_dimensions=show_dimensions)
+        self.to_string(buf=buf, max_rows=max_rows, min_rows=min_rows,


can the logic be added before this call to .to_string (isn't it as simple as just set max_rows to min_rows if len(self.frame) > max_rows?)

For the simple cases, yes. And I agree that could be a nice option. But there are some corner cases that are only handled within the DataFrameFormatter class (eg if max_rows = 0, we check the terminal size)

Although, looking at it more closely, that special case is only if max_rows is 0 or None, so that could be easily checked. So yes, this would in principle be possible (and probably even give easier code). But that means that we move some of the logic of the repr out of the Formatter class.

But that means that we move some of the logic of the repr out of the Formatter class.

i agree. but this logic is related to display options. i'd prefer to see all the display option handling removed from the Formatter classes entirely and only appear in __repr__ and _repr_html_

it appears from the comments that adding arguments to to_string is not considered undesirable. so this topic can be addressed another day.

TomAugspurger

Changes look good. A release note would be nice if you have a chance.

-1 on adding keywords to public methods. see #27000 (comment)

I don't have a strong opinion here, but I can see them being useful.

jreback · 2019-07-03T11:48:05Z

pandas/io/formats/format.py

        max_rows = self.max_rows
+        # if min_rows is None, follow value of max_rows


pandas/io/formats/format.py

jorisvandenbossche · 2019-07-03T12:35:17Z

-1 on adding keywords to public methods

As I said above, it is easy to avoid that, if we want (with only a little bit of code duplication, just a call to the Formatter class. But it also gives some consistency since max_rows and max_cols are there as well.

jreback · 2019-07-03T12:43:20Z

-1 on adding keywords to public methods

As I said above, it is easy to avoid that, if we want (with only a little bit of code duplication, just a call to the Formatter class. But it also gives some consistency since max_rows and max_cols are there as well.

I like the consistency argument here, +1 on adding min_rows to to_string

TomAugspurger · 2019-07-03T19:55:38Z

@jorisvandenbossche doc failure: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=13946&view=logs&jobId=7e620c85-24a8-5ffa-8b1f-642bc9b1fc36&taskId=2febcae8-7f6a-51bb-45aa-d6e72c9ea7e2&lineStart=2268&lineEnd=2269&colStart=1&colEnd=1

/home/vsts/work/1/s/doc/source/whatsnew/v0.25.0.rst:164: WARNING: Inline literal start-string without end-string.

jorisvandenbossche · 2019-07-03T20:03:27Z

Ah, this annoying check we have in our CI ;-)

doc/source/whatsnew/v0.25.0.rst

jreback · 2019-07-03T21:35:22Z

pandas/core/series.py

@@ -1522,6 +1525,9 @@ def to_string(self, buf=None, na_rep='NaN', float_format=None, header=True,
        max_rows : int, optional
            Maximum number of rows to show before truncating. If None, show
            all.
+        min_rows : int, optional


in theory should add a versionaded here, but not worth repushing

jreback · 2019-07-03T21:35:41Z

pandas/io/formats/format.py

@@ -79,6 +79,9 @@
            * unset.
        max_rows : int, optional
            Maximum number of rows to display in the console.
+        min_rows : int, optional


jreback · 2019-07-03T21:39:28Z

this is green. any objections?

jreback · 2019-07-03T21:59:59Z

very nice @jorisvandenbossche

Shorter truncated Series/DataFrame repr: introduce min_rows

1361fa4

jorisvandenbossche added the Output-Formatting __repr__ of pandas objects, to_string label Jun 28, 2019

jreback reviewed Jun 28, 2019

View reviewed changes

jorisvandenbossche added 3 commits July 2, 2019 22:44

Merge remote-tracking branch 'upstream/master' into shorter-repr

d62407a

add tests

b8f483e

add some comments

98e7d43

simonjayhawkins requested changes Jul 3, 2019

View reviewed changes

TomAugspurger reviewed Jul 3, 2019

View reviewed changes

jreback added this to the 0.25.0 milestone Jul 3, 2019

jreback reviewed Jul 3, 2019

View reviewed changes

jorisvandenbossche added 2 commits July 3, 2019 08:52

simplify

e4b2144

add whatsnew + docs

8060faf

doc fix

41c8543

add to Formatter / to_string docstring

577f078

TomAugspurger reviewed Jul 3, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

fixup

577c5cf

TomAugspurger mentioned this pull request Jul 3, 2019

RLS: 0.25.0 #24950

Closed

jreback reviewed Jul 3, 2019

View reviewed changes

jreback approved these changes Jul 3, 2019

View reviewed changes

simonjayhawkins approved these changes Jul 3, 2019

View reviewed changes

jreback merged commit 96c7ab5 into pandas-dev:master Jul 3, 2019

jorisvandenbossche deleted the shorter-repr branch July 4, 2019 00:44

jorisvandenbossche mentioned this pull request Aug 18, 2019

DataFrame html repr: also follow min_rows setting #27991

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shorter truncated Series/DataFrame repr: introduce min_rows #27095

Shorter truncated Series/DataFrame repr: introduce min_rows #27095

jorisvandenbossche commented Jun 28, 2019 •

edited

Loading

jreback Jun 28, 2019

TomAugspurger commented Jul 2, 2019

codecov bot commented Jul 3, 2019

codecov bot commented Jul 3, 2019

jorisvandenbossche commented Jul 3, 2019

simonjayhawkins left a comment

simonjayhawkins Jul 3, 2019

jorisvandenbossche Jul 3, 2019

jorisvandenbossche Jul 3, 2019

simonjayhawkins Jul 3, 2019

TomAugspurger left a comment

jreback Jul 3, 2019

jorisvandenbossche commented Jul 3, 2019

jreback commented Jul 3, 2019

TomAugspurger commented Jul 3, 2019

jorisvandenbossche commented Jul 3, 2019

jreback Jul 3, 2019 •

edited

Loading

jreback Jul 3, 2019

jreback commented Jul 3, 2019

jreback commented Jul 3, 2019

		max_rows = self.max_rows
		# if min_rows is None, follow value of max_rows

Shorter truncated Series/DataFrame repr: introduce min_rows #27095

Shorter truncated Series/DataFrame repr: introduce min_rows #27095

Conversation

jorisvandenbossche commented Jun 28, 2019 • edited Loading

jreback Jun 28, 2019

Choose a reason for hiding this comment

TomAugspurger commented Jul 2, 2019

codecov bot commented Jul 3, 2019

Codecov Report

codecov bot commented Jul 3, 2019

Codecov Report

jorisvandenbossche commented Jul 3, 2019

simonjayhawkins left a comment

Choose a reason for hiding this comment

simonjayhawkins Jul 3, 2019

Choose a reason for hiding this comment

jorisvandenbossche Jul 3, 2019

Choose a reason for hiding this comment

jorisvandenbossche Jul 3, 2019

Choose a reason for hiding this comment

simonjayhawkins Jul 3, 2019

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback Jul 3, 2019

Choose a reason for hiding this comment

jorisvandenbossche commented Jul 3, 2019

jreback commented Jul 3, 2019

TomAugspurger commented Jul 3, 2019

jorisvandenbossche commented Jul 3, 2019

jreback Jul 3, 2019 • edited Loading

Choose a reason for hiding this comment

jreback Jul 3, 2019

Choose a reason for hiding this comment

jreback commented Jul 3, 2019

jreback commented Jul 3, 2019

jorisvandenbossche commented Jun 28, 2019 •

edited

Loading

jreback Jul 3, 2019 •

edited

Loading