Interpolate do syntax closures into single _

[c42f]

This allows a single _ to be treated as the slot into which the closure created by do syntax is inserted.

It's somewhat overloaded in that @_ map(_, xs) normally means the _ is the identity, while @_ map(_, xs) do ... would mean the same as normal do syntax without the _, but conceptually we're marking a "slot of map" to receive a closure so it could be acceptably consistent.

Also, it seems likely to be extremely convenient.

A more realistic example would be a function which takes multiple functions as arguments (eg mapreduce(f, _, xs) do ...) allowing the second argument to be given with the do block.

CC @tkf @MasonProtter and anyone else interested — thoughts? Does this seem acceptably consistent?

Closes #4 by replacing it

[MasonProtter]

Personally, I’m tempted to say this should just be a separate macro from @_. Its not horribly inconsistent, but it does make it so that you need to understand more context in order to see what @_ is doing. But it’s also not that bad either.

[tkf]

I think it'd be nice to have this if this is compatible with everything else. Otherwise, I think I agree with @MasonProtter that separating it out from @_ may be a wise choice.

In particular, I wonder how it interacts with |>. I'm guessing that this works?

@_ function ((i, d),)
    x = get(d, :a, nothing)
    x === nothing && return nothing
    y = get(x, :b, nothing)
    y === nothing && return nothing
    z = tryparse(Float64, y)
    z === nothing && return nothing
    return (i, z)
end |> mapreduce(__, _, pairs(data); init = nothing) do a, b
    a === nothing && return b
    b === nothing && return a
    a[2] < b[2] ? b : a
end

(I'm expecting function ((i, d),) to be fed to the first argument of mapreduce and do a, b to the second argument.)

[c42f]

I'm guessing that this works?

Yes that should work (but I had a bug, oops. Fixed now).

It's interesting that you can place the closures on both sides 😆 Certainly confusing to look at IMO, but no reason to disallow it.

julia> @_ function (x)
           @show x
           x^2
       end |> mapreduce(__, _, [1,2,3]) do x,y
           x+y
       end
x = 1
x = 2
x = 3
14

[c42f]

Overall I think supporting do is certainly frustrating in Underscores because it's non-obvious how to do it in a visually straightforward but entirely consistent way. This is the best I've found so far though it borders on being a bit featury.

this should just be a separate macro from @_

My only objection to that is that macros don't really compose naturally when they act on the same type of syntax for the same reasons. So in some ways it's tempting to build all the closely related behaviors together into one macro.

[tkf]

Don't worry I won't do this in important code 😆. It's just that it was the first example I could think of to use __ and _ while only using Base function.

I guess more practical example is SplitApplyCombine.groupreduce:

@_ ... |> groupreduce(x -> x.key, _, __) do a, b
    a.value < b.value ? b : a
end

BTW, it looks like this does not work?

@_ collect(Filter(_), -3:3) do x
    x > 0
end
# => collect(Filter(x -> x > 0), -3:3)

I guess allowing this introduces more inconsistencies, though. I wondered a bit if it makes sense to add do block to the list of "scope delimiters" [*1] |>,<|,∘. This way, I guess

@_ collect(Filter(__), -3:3) do x
    x > 0
end

would work. But then the groupreduce example becomes

@_ ... |> itr -> groupreduce(x -> x.key, __, itr) do a, b
    a.value < b.value ? b : a
end

which is not nice...

That said, extending the crazy thought, I wonder if it makes sense to use the number of _ to denote how much you escape to the outer scopes:

• _: all but one call up to a delimiter
• __: all calls up to a delimiter
• ___: pass one delimiter, and then all but one call up to an outer delimiter
• ____: pass one delimiter, and then all calls up to an outer delimiter

We then have

@_ ... |> groupreduce(_.key, __, ___) do a, b
    a.value < b.value ? b : a
end

Well, too bad that it's hard to count the number of _s 😆

[*1] That's how I interpret them.

[c42f]

Well, too bad that it's hard to count the number of _s laughing

Haha! Yes...

I did consider that the closure created by do could just bind to three underscores ___

BTW, it looks like this does not work?

@_ collect(Filter(_), -3:3) do x
    x > 0
end
# => collect(Filter(x -> x > 0), -3:3)

I guess allowing this introduces more inconsistencies, though.

Yes. Though the fact that you might want this to work kind of suggests to me that this PR isn't really mergeable. It's just too confusing and overloaded, in a similar way that #4 was too confusing.

I feel like the best option may be just to go with an ugly but very clear identifier like _do_ or something.

I think the groupreduce example is perfect btw — SplitApplyCombine's grouping operations are one key use cases I had in mind for Underscores.jl. I feel that SplitApplyCombine has a very conceptually nice set of primitives, they only suffer from poor syntax usability in requiring multiple function arguments (CC @andyferris :) )

[andyferris]

Yeah agreed. do isn’t the best when you have lots of function inputs. Keywords are gimped on type constructors (widened to DataType etc). So it’s a bit tricky to make the interface to groupreduce readable.

[tkf]

Though the fact that you might want this to work kind of suggests to me that this PR isn't really mergeable.

TBH, I think it's OK to not do everything at AST. Though this is probably mainly because I can tweak Transducers.jl API to add |>-friendly forms. For example, __ is not a hard requirement for me as long as I'm using Transducers.jl (because transducers are curried by default). Likewise, collect(Filter(_), -3:3) example can now be written as

@_ -3:3 |> Filter(_ > 0) |> collect

# or even
-3:3 |>
    Filter() do x
        x > 0
    end |>
    collect

do isn’t the best when you have lots of function inputs.

I'm actually experimenting API for challenging this :). With wheninit etc., I can write

julia> averaging =  # transducer version of `OnlineStats.Mean`
           function add_average((sum, count), x)
               (sum + x, count + 1)
           end |>
           wheninit() do
               (Init(+), 0)  # attach "loop header"
           end |>
           whencomplete() do (sum, count)
               sum / count  # attach "loop footer"
           end |>
           whencombine() do (sum1, count1), (sum2, count2)
               (sum1 + sum2), (count1 + count2)  # for threaded reduce
           end;

I think this would be more readable than kwargs-based API especially when you have more complex function bodies

julia> averaging = AdHocRF(
           oninit = () -> (Init(+), 0),
           complete = (sum, count) -> sum / count,
           combine = ((sum1, count1), (sum2, count2)) -> ((sum1 + sum2), (count1 + count2)),
       ) do (sum, count), x
           (sum + x, count + 1)
       end;

For groupreduce, maybe we can make something

julia> @_ 1:100 |>
          Map((k = gcd(_, 42), v = _)) |>
          grouped(_.k, __) .|>
          Map(_.v) .|>
          maximum

work. Then, since each expression between |> has only one function, we can use do block when the body is long enough.