[FEA] Support regular expression patterns for string split

[beckernick]

I'd like to be able to pass a regular expression pattern for string splitting, like I could do in base Python or pandas. This is relevant for tasks like trying to do sentence tokenization with regular expressions, as sentences can end with more than one type of punctuation (for example).

import pandas
import cudf
import re
​
print(s)
0    this is. an? example
dtype: object

s = cudf.Series(['this is. an? example'])
ps = s.to_pandas()
​
print(ps.str.split('[.?!] ', expand=True))
         0   1        2
0  this is  an  example

print(re.split('[.?!] ', ps.iloc[0]))
['this is', 'an', 'example']

# cudf
print(s.str.split('[.?!] '))
                      0
0  this is. an? example

[randerzander]

@beckernick we created nvstrings.replace_multi to support a performance way to make many string substitutions in a single pass (as opposed to passing a lengthy regex string with multiple patterns).

I wonder if a similarly behaved nvstrings,split_multi would suffice? I suspect it would be more performant than a regex based split.

@davidwendt thoughts?

[davidwendt]

I would certainly recommend implementing a split_multi over adding regex to split.

[beckernick]

I agree with both of you. However, I'm curious to get your thoughts on the following hypothetical:

What if I wanted to split on my own regular expression based definition of "End of Sentence", but slightly more complex than the toy example above. It would be difficult to enumerate all the possible "states" explicitly, and if the dependency is more than 1 character long (i.e., more than just a punctuation mark) the state space actually explodes.

replace_multi appears to support regex . If split_multi would also support multiple regex patterns, that would likely allow this behavior. But, at that point, I'm not sure of the additional value compared to standard split + regex in the above scenario.