closing pipes, like reduce, are currently not supported

[jerabaul29]

Moved from the initial discussion in #67 ; from the author:

For reduce I don't think pipe can handle it, I deprecated "closing pipes" a year or so ago:

What I mean by a "closing pipe" is a pipe that does not return an iterable, but a value, so itself cannot be on the left hand side of a pipe, "breaking" the pipe, example:

>>> range(100) | filter(lambda x: x % 2 == 0) | sum 

would be readable, OK, but sum does not return an iterable, so no further | can be used. I deprecated this in favor of the even shorter and standard:

>>> sum(range(100) | filter(lambda x: x % 2 == 0)) 

I think reduce enters this category of finalizing pipes so it can't be added, as it would be better written as:

>>> from functools import reduce
>>> reduce(lambda x, y: x + y, range(100) | filter(lambda x: x % 2 == 0))

I understand the point of the author, but I would like to disagree on this point :) . This is maybe mostly aesthetics, but to my eyes this:

res = ( range(100) | filter(lambda x: x % 2 == 0)
                   | reduce(lambda x, y: x + y) )

looks nicer and more readable than this:

>>> from functools import reduce
>>> reduce(lambda x, y: x + y, range(100) | filter(lambda x: x % 2 == 0))

Because in the first case, I can just "follow the logics" as it flows and as my brain expects it, but in the second case, I have to force my brain to remember that while most of the expression flows from left to right, the final step is actually an exception to this rule as it is at the far left... Also, this breaks my habits from other places where I see similarly formatted "functional programming" expressions, like rust et. co., that would look much more like the first way of formatting it.

Would there be a way to enable a syntax that looks like the first case, but without the worries about closing pipes that are raised by the author? For example, using a separate, special "closing pipe" class that would allow to perform checks and to issue meaningful error messages if closing pipes are used at the wrong place in the expression?

[JulienPalard]

Would there be a way to enable a syntax that looks like the first case, but without the worries about closing pipes that are raised by the author?

Yes, as I see the lib like a tool (the Pipe class) with examples (the various @pipe implementations).

So up to you to implement it as needed, like:

 @Pipe
 def reduce(iterable, function):
     return functools.reduce(function, iterable)

if the arguments were in the same order it could even be implemented as:

reduce = Pipe(itertools.reduce)

Sadly the arguments are reversed, but for some other functions, like sum it would work:

>>> from pipe import Pipe
>>> sum = Pipe(sum)
>>> range(10) | sum
45

(although it's a very bad idea to hide the sum builtin!)

Should I clarify this in the readme (that I encourage people to write their own pipe functions as needed instead of trying to create them all in the lib?)

[jerabaul29]

Should I clarify this in the readme (that I encourage people to write their own pipe functions as needed instead of trying to create them all in the lib?)

Why not? :) . I just do not see a major reason for not having it in the library, but I may be wrong, and it is your library :) . I think that many users will end up having some boilerplate code coming back again and again for implementing reduce and a couple similar "closing pipes".

[JulienPalard]

Sorry for the late reply.

I found most of the "closing pipes" to be of a very low usefullness and I found they were misleading to some users (not easily understanding why they could not "continue to use pipes after a closing pipe" to caricature, having two distinct semantics with the same syntax was misleading and I think having a single one is cleaner).

But you may be right, maybe many project would end up sharing a few common @Pipe, but I'm not sure it's an issue: if they do so they understand what they're doing and it's OK to me as long as it's 2 or 3 lines of code. (Copy-pasting 40 lines of code between 3000 projects would be horrible, I agree).

Sadly I have no real statistics about pipe usage. Looks like it's used, according to pypistats, but search for "pipe" on github is a bit noisy...

There may be a solution: what about adding a "Recipes" section on the README? Like the cpython itertools module does: https://docs.python.org/3/library/itertools.html#itertools-recipes? So if one needs "reduce" it's easier to get it than having to reimplement it, and people could share idea here.

[jerabaul29]

No worries at all :) .

If you are not willing to have them in your package, then I agree that a "Recipes" section of the Readme would be great :) .

[JulienPalard]

I documented the deprecations in the README, I'll slowly add recipes as needed, feel free to open PRs with some recipes if you have some nice ones on your end.

Beware I'm releasing pipe==2.0 soon, hope it won't break something.

[dit-zy]

Hi! Not looking to resurrect the topic, just wanted to share my perspective (although at the end I realized an idea):

I was surprised when reading the README and seeing that closing pipes were taken out, so I came here to understand why.

...I found they were misleading to some users (not easily understanding why they could not "continue to use pipes after a closing pipe" to caricature...

This explained a lot. So even though I feel differently, I respect the choice. But I'd still like to provide my perspective.

I found this library when looking for a way to do function chaining in python. Honestly, I'm surprised that a language as focused on developer experience as python is, doesn't support function chaining more easily, and especially for functional programming styles. That's what's so exciting about this library, because the chaining style is significantly more readable than the nesting style that's default in python:

list(filter(lambda x: ..., map(lambda x: ..., map(lambda x: ..., some_list))))

And that's why I think that closing/terminating pipes are still really valuable to have in the library, because it seems core to the value proposition of the library, to me. I think that some core terminators like as_list, as_set, and as_dict would be super commonly used, exactly because it's more readable and a natural thing to want to do with pipes. So it feels like a big break from one of the values of the library, to still have to use nesting style for terminators.

P.S.
I wasn't planning on making any suggestions, but I realized an option while writing this. You could put terminators in a separate package, like pipe.terminators and maybe even have a separate operator to use them, like maybe >> which would take a Pipe on the left and a Terminator on the right. That way they could be built into the library but not mislead anyone.

But regardless, thank you for providing this library! It's really wonderful!

edit: I decided to try implementing the idea I suggested and submit a PR, but I discovered why operators like >> won't work, as | already has the lowest precedence of operators that have a reverse form like __ror__(), and a reverse form is required with how this library works. But I'll try the idea without a separate operator and if you don't like it that's alright.

edit2: found that I could actually use the > operator, so implemented that into #86.