Undocumented feature: partial string slicing

[tdpetrou]

Code Sample, a copy-pastable example if possible

>>> index = ['abe', 'adam', 'andrew', 'ben', 'brad', 'cal', 'chad', 'dan']
>>> data = [0] * len(index)
>>> df = pd.DataFrame(index=index, data=data, columns=['col'])
>>> df
        col
abe       0
adam      0
andrew    0
ben       0
brad      0
cal       0
chad      0
dan       0

>>> df.loc['ac':'d']
        col
adam      0
andrew    0
ben       0
brad      0
cal       0
chad      0

Problem description

Partial string slicing is documented for datetimeindexes but is nowhere to be found for string indexes. The index must be ordered for it to work. I couldn't find a single example of this anywhere online. Is this type of slicing encouraged? Should it be documented?

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Darwin OS-release: 15.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: 3.0.7
pip: 9.0.1
setuptools: 35.0.2
Cython: 0.25.2
numpy: 1.13.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: 1.5.5
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.3.0.post

[gfyoung]

Admittedly, this behavior is a little less intuitive because Python string comparisons aren't as easily comprehensible of datetime objects. That being said, I'm indifferent about allowing (in which case, document) or disallowing it (in which case, forbid).

[toobaz]

Maybe I'm missing something, but I see no "partial" string slicing: I just see slicing with start/stop bounds not present in the index, not differently from what happens with

In [11]: df.loc['academia':'dadaism']
Out[11]: 
        col
adam      0
andrew    0
ben       0
brad      0
cal       0
chad      0

This is something perfectly natural when you are dealing with datetimes or ints, a bit less with strings, as @gfyoung rightly suggests; still, I don't think there is anything it makes sense to forbid. The docs, by the way, never state that start and stop bounds must be present, except maybe (implicitly) when they say that they are included in the results. So unless I'm missing something we should just change the sentence "When slicing, the start bound is included, AND the stop bound is included" with "When slicing, the start bound, AND the stop bound are included, if present.".

Analogously, few lines later, "(note that contrary to usual python slices, both the start and the stop are included!)" should become "(note that contrary to usual python slices, both the start and the stop are included, if present!)"

[gfyoung]

@toobaz : it's a matter of perspective. In the examples provided by @tdpetrou , "ac" and "d" can be viewed as partial strings because they are all shorter than any of the string indices provided.

That being said, given my indifference, do either of you (@toobaz and @tdpetrou ) have any preference? This is really up to users at this point, unless other maintainers have strong opinions about this.

[toobaz]

@toobaz : it's a matter of perspective. In the examples provided by @tdpetrou , "ac" and "d" can be viewed as partial strings because they are all shorter than any of the string indices provided.

Sure, sorry, I wasn't clear :-) I see @tdpetrou 's perspective, I was just suggesting that this is not panda's perspective, and that there is nothing really unexpected going on.

It's like saying "positional indexing with prime numbers is undocumented" - positional indexing just works with any integer number, and you only want to document the general case.

[gfyoung]

@toobaz : No worries. It seems like you have no issues with allowing this behavior, so long as it's documented. Let's see what @tdpetrou has to say about it as well. Otherwise, either one of you is more than welcome to document (and test) this behavior in a PR!

[tdpetrou]

I like this behavior and think it should remain. I disagree with @toobaz and think this behavior is completely unexpected. There is nowhere in the documentation that partial strings (or inexact strings or however you want to term them) work in this manner. This specific behavior only works when the index is sorted and fails with a KeyError when not.

The normal behavior of slicing with .loc is to select all indexes from start to stop and include stop. This is done without regard to lexicographic ordering. If either start or stop is not in the index raise a KeyError.

This specific behavior that I brought up works as such. If index is ordered, either increasing or decreasing, then select all indexes lexicographically greater than or equal to start and less than or equal to stop (or vice versa if monotonic decreasing). Do not raise KeyError if either is not in index.

[toobaz]

I like this behavior and think it should remain. I disagree with @toobaz and think this behavior is completely unexpected. There is nowhere in the documentation that partial strings (or inexact strings or however you want to term them) work in this manner. This specific behavior only works when the index is sorted and fails with a KeyError when not.

Nothing specific to strings (and "inexact" is not a better term than "partial"). Compare pd.Series(index=[1,3,2,5]).loc[0:6] (KeyError) to pd.Series(index=[1,2,3,5]).loc[0:6] (works).

But maybe you are right that this is not documented, and in this case, a PR to the docs would probably be a very good thing.

[tdpetrou]

This is a very specific and different behavior than normally expected which I already outlined above. It would be helpful to the users to see an example. I am using the term 'lexicographic slicing' in my book.

[toobaz]

I don't follow you - I can't see what "lexicographic" means when there aren't multiple levels.

[gfyoung]

@tdpetrou @toobaz : We can worry about exact semantics later in the PR. Maintainers will have the last word in the end 😄

@tdpetrou : if you're interested, you should put up a PR describing this behavior both in the docstring + the documentation (under the doc/ folder). Also, tests for this behavior will be needed.

[tdpetrou]

@gfyoung I'll see if I have time to do this over the weekend but anyone else is welcome to take it.

[toobaz]

OK, just to summarize: my understanding is that .loc[start,stop]

1. returns everything between start and stop in the index if start and stop are in the index
2. return everything sorting between start and stop if the index is sorted (and start and stop can be compared against its elements)
3. fails otherwise

Part 2. is probably undocumented/untested.

[gfyoung]

Yep, @toobaz , that sounds about right.

[jreback]

This is accidentally working somewhat. I don't find this a particularly useful feature, since we have full support via the .str accessor.

[gfyoung]

@jreback : Should it be disallowed or documented then?

[jreback]

it could just be documented. though a bit inclined to disallow as it only partially works. (IOW its sorting order dependent), so a bad UX.

[gfyoung]

That's fair. How about a deprecation cycle?

[toobaz]

This is accidentally working somewhat. I don't find this a particularly useful feature, since we have full support via the .str accessor.

Wait... what "feature" are you talking about?! I hoped I had convinced everybody that this is nothing specific to strings (or if I'm wrong, that someone would convince me the converse is true).

[gfyoung]

Wait... what "feature" are you talking about?!

@toobaz : This is partially why I wanted to wait until I got further input from other maintainers given my indifference. Given that string indexing is somewhat unusual, it is open to interpretation.

@tdpetrou : any response to what @jreback has said? I only recommended a deprecation cycle above because if we have full support via another way, I have no problems picking that way instead of this one with "partial" string indexing (or whatever we want to call it).

[tdpetrou]

@jreback There is no clean way to do this with .str accessor. I personally find it useful for exploration but did not find one instance of it being used anywhere online. I looked for quite some time too. I doubt I'm the only one who's ever done this but it can't be very common.

I actually think deprecation is a good idea as the indexers are still overloaded in my opinion.

[toobaz]

Given that string indexing is somewhat unusual

... why?! I use string indexing all the time, and never expected anything more nor less than from "normal" (?) indexing.

Take my previous comment. It describes a general behaviour which, as far as I can tell, is always honoured by pandas, regardess of the dtype of the index. Maybe undocumented, but easy to document (and makes a lot of sense, and I'm sure lots of lines of code rely on it).

What do you all want/not want to deprecate exactly?! Is it clear to everybody that the title of this issue is plain wrong?

[gfyoung]

@toobaz : The specific application to strings is a little unusual, which you even said yourself above.

That being said, I think we can all agree that documentation + testing of this functionality is needed. Let's start with that (@tdpetrou @toobaz feel free to work on this), and we can address what we want to do with this "feature" (or whatever you want to call it in the future) afterwards.

How does that sound?

[tdpetrou]

This is going to veer a bit off topic but the str accessor could gain a between method to handle this situation. Also, if this gets deprecated, maybe we can just deprecate the indexing operator itself completely and force iloc and loc on everyone...

[jreback]

This is not a feature, rather an accidental impl detail. So while you can document something that only sometimes / partially works, this is bad UX. If someone wants to fully test & document great. Otherwise it should be prohibited. A partial 'working' is bad either way.

[toobaz]

This is not a feature, rather an accidental impl detail.

"This" = ?

Are you talking about point 2. in my comment above? So not specific to strings?

How can you say it is accidental? You mean pd.Series(index=[1,2,3,5]).loc[0:6] wasn't supposed to work? Or is it just "accidental" that indexing works in a same coherent and intuitive way across all dtypes? Would you find an indexing mechanism which applies to all dtypes but strings for no obvious reason a "good UX"?! What about indexes with some strings and some non-strings?! What about MultiIndexes with only one string level? What about categoricals indexes which happen to have strings as values?!

If this functionality (2 in my comment above) is indeed untested/undocumented, then this is true regardless of the dtype.

[gfyoung]

@toobaz : "This" = the partial string indexing. I think @jreback is referring to the fact that we generally index with numbers, dates, and complete strings, not partial ones.

Also, I don't believe @jreback is calling the actual behavior bad UX but the fact that it is not complete to be bad UX. He's say that rather than encourage using an incomplete feature, we should be using a complete feature (which is via .str according to him, though @tdpetrou begs to differ).

[gfyoung]

This is going to veer a bit off topic but the str accessor could gain a between method to handle this situation.

@tdpetrou : Possibly. We should investigate that after we document and test this functionality that you found in the issue.

[toobaz]

@toobaz : "This" = the partial string indexing. I think @jreback is referring to the fact that we generally index with numbers, dates, and complete strings, not partial ones.

OK, this is becoming crazy. There is no such thing as "partial string indexing": everyone is free to see it from his/her preferred perspective, but in the pandas logics (and docs, when it will be documented) it is just "indexing with labels which do not appear in the index", regardless of the dtype. And as long as it is correctly described, I see no need to special-case the docs, because it would be more complicated to understand.

Disabling the behaviour only when start and/or (?) stop are strings and happen to be parts of strings which appear in the index would be a nightmare both to code and to document.

[gfyoung]

Disabling the behaviour only when start and/or (?) stop are strings and happen to be parts of strings which appear in the index would be a nightmare both to code and to document.

Well, to be fair, we haven't actually decided to remove this functionality yet.

We have decided, however, that this functionality should be documented as is and tested. That, I think would work for everyone at least.

[toobaz]

OK, I don't have time for tests now but here's a PR for the docs.

[gfyoung]

@toobaz : I think it may be okay if you just do documentation, but ideally, we would have tests + doc in the same PR. @tdpetrou feel free to jump if you have good examples (like the one in your issue).

[toobaz]

@toobaz : I think it may be okay if you just do documentation, but ideally, we would have tests + doc in the same PR

Time constraints are indeed only one of the many aspects which make me non-ideal

[toobaz]

By the way: you are asking me to spend time testing a feature which you/@jreback will then decide whether to drop or not. This is plain ridiculous. Why does everybody who has commit rights on pandas automatically assume that everybody else is just looking desperately for ways to waste time?!

[jreback]

By the way: you are asking me to spend time testing a feature which you/@jreback will then decide whether to drop or not. This is plain ridiculous. Why does everybody who has commit rights on pandas automatically assume that everybody else is just looking desperately for ways to waste time?!

@toobaz pls be civil and respect others time. Your PR's are always given consideration. adding code / documentation does require discussions. You of course don't need to do any work until / unless things are decided to be kept or not.

[gfyoung]

Why does everybody who has commit rights on pandas automatically assume that everybody else is just looking desperately for ways to waste time?!

It's actually not a waste of time. One, we haven't decided to deprecate yet. Second, it is always good practice to test any and all functionality, both intended and unintended because that way you can better handle user issues and not be taken off guard by problems / surprises that you hadn't bother to explore.

[toobaz]

It's actually not a waste of time. One, we haven't decided to deprecate yet.

You of course don't need to do any work until / unless things are decided to be kept or not.

Let's say I prefer @jreback 's version.

So, sorry if by mentioning once again my frustration in contributing to pandas I deviated your attention from the real topic of this issue.

Just to understand if we are on the some page: does any of you maintainers not agree that the topic is "Undocumented feature: slicing with bounds not in index" rather than "Undocumented feature: partial string slicing"?

[gfyoung]

So, sorry if by mentioning once again my frustration in contributing to pandas

I've had similar frustrations when I began contributing to open-source projects, so I understand where you're coming from. This is just the nature of library development and maintenance. On the one hand, we need to ensure that the existing code is well-documented and comprehensive. However, we also have to look ahead and make decisions as to where we want to the library to go.

Documentation and testing works towards that first point. It just so happens that in those discussions, we also want (maybe) to move forward on the second point too with a deprecation. It doesn't happen all the time, but it does from time to time.

[toobaz]

so I understand where you're coming from

Thanks for your kind words - I'm not convinced by them, but we can discuss in private if you want. Can we go back to discussing the issue?

[gfyoung]

Thanks for your kind words - I'm not convinced by them, but we can discuss in private if you want.

No need to discuss if you're done 😄

[gfyoung]

"Undocumented feature: slicing with bounds not in index" is a little vague IMO because we're focusing specifically on the context of strings. The latter title is a little more specific to what this issue is focusing on here. Thus, of the two, I would prefer the latter title.

FYI, this seems related to your PR, so I would move the conversation over there.

[toobaz]

No need to discuss if you're done 😄

we can discuss in private if you want.