ruby-like blocks

[JeffBezanson]

This has come up a couple of times. I'd like to accumulate syntax suggestions here.

[jckarter]

What do you think of Smalltalk-style alternating keyword: {block} keyword: {block} forms?

[JeffBezanson]

Good that it generalizes to N blocks, but in our case the problem will be that we're out of ascii characters :)

[avibryant]

I think it's useful to take a couple of motivating examples. One thing Ruby's syntax optimizes for is the case where you have one or more non-block args, and one block arg. For example, a method that opens a path, gives your block the file, and then cleans up when the block is done:

File.open("foo", "w") {|file| file.puts("hello world") }

In Julia this would currently have to look something like:

file_open("foo", "w", file -> write(file, "hello world"))

You could imagine some similar convention to Ruby's where if the last arg is a function, it can be broken out into curly braces outside the parens:

file_open("foo", "w"){file -> write(file, "hello world")}

The other thing Ruby's blocks optimize for is long pipelines; for example,

array.map{|x| x * 2}.filter{|x| x > 10}.map{|x| x*x}

In Julia, this would look like:

map(x -> x*x, filter(x -> x > 10, map(x -> x * 2, array)))

It's less clear how to bring the spirit of what Ruby offers over, given the fundamental difference between the single-dispatch message-passing OO style it uses and the generic function style that Julia uses. One possibility would be a pipe operator which takes the result of one function call and splices it in as the last argument of the next, so that you'd get something like this:

array | map(x -> x * 2) | filter(x -> x > 10) | map(x -> x*x)

But I'm not at all sure that you want to go there...

[StefanKarpinski]

I've thought about a "pipe operator" before, but more shell-like. The idea would be to take the producer/consumer business that we already support with coroutines but make it more efficient by allowing the specification of a buffer of some size for produced objects. That would reduce the frequency of context switches, making things run faster. And it would you do to things like this:

producer | filter | consumer

Not sure about the details, but it seems a shame to not be able to do shell-style coding in a high-level language. Why not?

That, of course, doesn't at all address the Ruby-style block issue. One observations is that we can't really use curly braces because we're already using them for type parameters. ASCII is limited but making core language features non-ASCII strikes me as a very bad idea.

[StefanKarpinski]

Ok, so I spent a long time thinking about this (while hanging out an estancia a couple hours outside of Buenos Aires), and settled on this following scheme, which Jeff seems on board with. Barring major objections, I think this is what we should go with. It's very simple and derives quite a lot from Ruby, which, of course, pioneered this blocks-attached-to-function-calls-with-syntax business. This is so anticlimactic:

foo(a,b) do x,y
  # ...
end

is just syntax for this:

foo((x,y)->begin
  # ...
end, a,b)

So yeah, it's a lot like Ruby's block-on-a-method syntax. Which is a good thing. It will be obvious to anyone who's done any Ruby programming what's going on. It's pretty simple too: it just adds a new keyword — which is actually not insignificant, but I think we can spare the identifier do, which would be a really confusing variable name anyway.

The trivial difference is that it doesn't use Ruby's weirdo |x,y| syntax, which frankly has always just struck as strange and unnecessary. I guess the | delimiters are handy in the one-line braces form: {|x,y| ... }. However, we're not going to have one of those. If the function is small enough to have inline, then you just use the normal calling form and an inline function: foo((x,y)->x+y, a,b). The block syntax is only for large functions that are too long or unwieldy to put on a single line.

The major difference from Ruby is that this argument is not a special named argument and it doesn't go at the end. Instead, it's just the first argument to the function being called. This is because most higher-order functions take the function as their first argument. Thus, you can write a complicated map operation like this:

w = map(v) do x
  # something complicated with x
end

At first I was thinking of having the function block passed as the last parameter for obvious reasons — partly driven by the use case of open where I imagined that the block that you want to execute with a function handle would go at the end of the function call. With the function going at the front, we'd want the signature of open with a block to be this:

open(f::Function, file::String, mode::String)
open(f::Function, file::String) # defaults to mode="r"

This would allow the block calling form like so:

open("file") do r
  # do something with the r filehandle
end

At first I thought that the non-block form of open like this would be awkward, but it actually works. For example:

x = open(readall, "file")

This opens the file file for reading, applies the function readall, thereby reading the contents of the file, then closes the file handle, finally returning the read contents, returned by readall. That's actually really handy. The other alternatives would be writing this:

r = open("file")
x = readall(r)
close(r)

which is really verbose for something that simple, or to just let the file be closed when gc reclaims the file handle object:

x = readall(open("file"))

Comparing this with putting the function at the end: x = open("file", readall), I think having the function at the end is less clear. So even in the open use case, having the function argument come first seems better. We can just keep this in mind when writing further standard library functions: if you want the do-block syntax to be applicable, then have the function argument first.

The other difference from Ruby, of course, is that there is no yield keyword in the calling function, nor are there any implicit block arguments or anything like that. This proposal changes nothing about the called function behavior or syntax. Imo, Ruby's use of the yield keyword confuses its behavior with coroutines — which is not in any way what Ruby has. Ruby's yield is just syntax for calling an anonymous function argument. Of course, Julia has actual coroutines, which this proposal has nothing to do with.

[avibryant]

It's interesting to look at how my contrived "pipeline" example looks with this syntax:

map(filter(map(array) do x
x * 2
) do x
x > 10
) do x
x * x
end

I can't imagine anyone ever using that. You'd just introduce some local variables instead, I guess.

However, for the file-open case, it definitely does work well.

[StefanKarpinski]

That's not quite the syntax — there's a bunch of ends missing. Also, I certainly wasn't thinking about having these to be chained. The chaining thing really only works in Ruby for the short block syntax, which we're not going to have, and because Ruby operates on containers by doing v.map{...}.filter — i.e. it relies on the method call syntax — which we also don't have. I'm way less interested in the operation chaining than making sure that passing multiline blocks of code to higher-order functions is not awful. The chained version should, imo, just be written in plain old functional notation in Julia.

[StefanKarpinski]

Another possible addition to this would be support for else blocks as proposed here. The idea would be that something like

match(r"b(a)(r)", str) do x,y
  # do something with the match contents
else
  # handle not matching
end

would be syntax for this:

match((x,y)->begin
  # do something with the match contents
end, ()->begin
  # handle not matching
end, r"b(a)(r)", str)

The open function could also support this form too and else block could be used to handle failure to open the file instead of wrapping the whole thing in a try/catch construct. In general, this sort of syntactic construct would allow a lot of syntax-like functionality to be implemented just using higher-order functions, which is the goal of this issue.

[avibryant]

I like that a lot. I'm a Smalltalker, and Smalltalk has a very natural syntax for methods that take multiple blocks/functions that it uses for things like if/else, try/catch, etc. I'd say that in 90% of the cases with 2 blocks, one can be thought of as an "else" block. (It would be lovely to see Julia become so consistent around this that if() was just a method, so that it would be if(bool) do ... else ... end; nearly always inlined, obviously)

[StefanKarpinski]

This non-breaking feature would address the regex redesign issue #88 as well, so I'm going to close that.

[o-jasper]

Won't we want stuf like:

match(r"b(a)(r)", str) do x,y
  # do something with the match contents
else match(r"someregex", str) do x,y
  # handle not matching
end end

Shortened too? for instance avoiding repeating str, getting rid of the double end, and then we're basically at a switch like thing. So we'll want a case like in CL to work somewhat like that too? Maybe have a 'clause macro' clauses value ...clauses... end and have different functions for different clauses.

 #First argument is what `clause_of` was told, rest is what comes after the keyword.
 defclause_fun regex (string::String, regular_expression) 
    return match(Regex(regular_expression,false), string) != nothing
 end

And that clause will be executed if that function returns true. else is: defclause_fun else (ignored::Any) return true end

clauses_for some_string
regex ".+case_1.+"
   #body if in there
else
  #otherwise
end

Other stuff can then be added, i guess. I called it defclause_fun there because we might want to have macro version or something. For instance if we have regexes with variables 'in the reverse way' we can format strings. "$(a::"variable a will eat the match.+") and $(b::Integer) b eats integer".(basically the functionality i was proposing here)

The clauses can of course also do stuff like case of CL; defclause_fun case (value::Number, compare) return value == compare end. We'll probably want that more optimized than just comparing in sequence. We also have defclause elseif (value::Any, cond) return cond end

We might also want to allow users to define what the body before the first keyword does, with that it could go as far as 'emulate' if completely.(Stuff like if, for, while seem currently 'special' wrt parsing, right?) I am just throwing this idea out there, it certainly needs more thought. For instance, we can't have all possible clause names be special like if,elseif and such. One idea is to can disallow 'body-level' variables completely, and require them explicitly returned or returned as (variable), but the problem with that is that if, while and such are allowed off 'body level' too like 1 + if something 0 else 1 end, i guess that can be disallowed too. While those will basically never affect me, i don't really like that solution..

[miau]

+1 for the syntax array | map(x -> x * 2) | filter(x -> x > 10) | map(x -> x*x). Here the | operates like a pipeline operator in F#, it might go well with Julia's syntax.

[satoshi-murakumo]

I propose this syntax.
array | map(_, x -> x * 2) | filter(_, x -> x > 10) | map(_, x*x)
This syntax is a case where an argument is only one. _ is implicit variable.
In many cases piped argument is one. So I think that it is good to prepare special syntax.
The syntax for two or more arguments is as follows.
(array1, array2) | (xs, ys) -> filter(xs, x -> x > 10), filter(ys, y -> y < 10) | (xs, ys) -> vcat(xs, ys)

[StefanKarpinski]

A Python take on code blocks: http://mtomassoli.wordpress.com/2012/04/20/code-blocks-in-python/.

[miau]

I wrote a macro that permit notation like array | map(x -> x * 2) | filter(x -> x > 10) | map(x -> x*x). Julia is so powerful that I can change the behavior for myself, I'd appreciate if Julia would be bundled with a feature like it.

julia> macro pipe(ex)
         expand_pipe = expr -> begin
           if typeof(expr) == Expr && expr.head == :call && length(expr.args) == 3 &&
              expr.args[1] == :| && typeof(expr.args[3]) == Expr
             push(expr.args[3].args, expand_pipe(expr.args[2]))
             expr.args[3]
           else
             expr
           end
         end
         expand_pipe(copy(ex))
       end

julia> @pipe [1:10] | map(x -> x * 2) | filter(x -> x > 10) | map(x -> x*x)
5-element Int32 Array:
 144
 196
 256
 324
 400

[pao]

@miau, that looks a lot like desugared LINQ/desugared Scala for comprehension/bind chain on the List monad. I'm not sure if this is really the same feature as code blocks anymore, though I do like the idea. I'd be happy to see a macro implementing LINQ-style query expressions (or for comprehensions, or do notation, I'm not picky) in extras/.

EDIT: On this particular topic, see also #550.

[StefanKarpinski]

Huh. That's very cool. I do think this is a very different feature from code blocks, which serve to make coding with closures feel more like syntactic constructs. But this sort of piped programming also has its place.

[o-jasper]

Basically piping is function composition.. Ok, not entirely; filter can discard elements for instance, so there it would have to be function composition with some conventions, like that the returned result is a list.(it can return more than one) Also they can also be seen as programs with a stdin and stdout(though presumably here we work with objects, not bytes) i guess the latter is more accurate, because when implemented as calling functions, if one decides to wait, the whole thing stops, whereas in piping other 'programs' keep running. Yet another way to see it is that it is a series for functions that have access to the next function, they are run if the previous outputs to stdout and they run the next guy if they themselves output to stdout.

Anyway guessing one thing we want is to have these things also work nicely outside piping. Like filter, map works nicely, because one more argument means it just operates on that sequence. match could do the same, return boolean if it also has a string, and return a filtering block otherwise. Macros would need a similar convention so they can either generate code for blocks or 'run immediately' aswel, dont really see a problem there.

The clauses/else aughto fit in somewhere. I guess the condition of a clause could be considered true if the pipe blocks are done and something came out the end. But for the case of macros you'll want to get the variables as used in the filters out aswel.(hope this comment isnt too stupid)

[StefanKarpinski]

Re-opening for else-blocks.

[JeffBezanson]

I guess I should allow chaining, as in

f(x) do
  ...
else g(x) do
  ...
end

?

[StefanKarpinski]

Why allow chaining? There's no obvious use case for that.

[JeffBezanson]

If this is used for matching-type thingies, one often wants to try matching a series of patterns in succession.

[StefanKarpinski]

Ok, that sounds plausible. Can you give an example so I can get a more concrete sense?

[StefanKarpinski]

I'm closing this since we can just open a new issue if we ever want else blocks.