scoping issues, part 1

[thomie]

Hello Julia developers,

congratulations with your great new language. I was singing 'O Julia' out loud all day yesterday.

But now my bug report. Notwithstanding the let, global and local keywords, it seems there is not always a clear distinction in Julia between variable declaration and assignment. This breaks encapsulation and can lead to hard to find bugs.

function some_scope()
    ...
    ... many lines of code
    ...
    f = () -> x = 42  # Not clear if this declares a local variable or assigns to one from the enclosing scope.
end

Matlab's scoping rules we don't even need to discuss [1], Coffeescript is a disaster [2], please don't copy Ruby on this one [3], Perl and Javascript are ok as long as you don't forget your my's and var's (or use strict), Python fixed it in version 3 [4], and one could say Scheme got it right the first time in 1970 [5].

What is the reasoning behind Julia's intricate scoping rules?

[1] http://www.mathworks.nl/help/techdoc/matlab_prog/f4-39683.html#f4-73993
[2] jashkenas/coffeescript#712
[3] http://www.rubyist.net/~matz/slides/rc2003/mgp00010.html
[4] http://www.python.org/dev/peps/pep-3104/
[5] http://news.ycombinator.com/item?id=3379962

[JeffBezanson]

A variable introduced in a scope is inherited by all inner scopes, unless one of the inner scopes overrides that.

So in f = () -> local x=42 the x is always local to f no matter what. Without the local, x is local to f if there is no other x around. But if an x is introduced in some_scope, then f will inherit that x. This seems reasonable to me.

A variable can be "introduced" by either declaration or assignment.

In Python, the problem was that there were certain reassigning operations that were not possible to achieve. We don't have that problem, since the default is to fully inherit variables.

We did things this way so assignment can implicitly create variables in an unsurprising way, without the distinction between define and set!.

Would you suggest something like outer x = ... to rebind a variable from an outer scope?

[jckarter]

Assignment shouldn't introduce new variables. You should require local or let to bind a new variable.

[StefanKarpinski]

I feel strongly that avoiding the verbosity of writing var or my or whatever in front of every new local variable trumps the additional clarity provided by requiring such declarations. Python almost gets this right, but not quite.

The main thing here is that you never have to scan beyond an obvious and usually quite limited, and always contiguous region of code to figure out where a variable is local to. Using functions as the typical example of a new scope at the top level, any variable assigned inside the function must be local unless it is explicitly declared to be global; anything that's never assigned must be global or a parameter (which is also obvious). In other words the behavior that is what you almost always want— that assignments are to new local variables — is the default, easiest thing to accomplish.

[JeffBezanson]

From a pure programming perspective, you're probably right, but matlab/octave/etc. users expect to be able to simply write x = 0 anywhere and have it work. I know matlab isn't a standard of good design, but to me this isn't something they got horribly wrong.

[StefanKarpinski]

Also, in practice, I never find that the current scheme is confusing — it just does what I want. I agree, however, that reading the rules in the manual it sounds a little complicated. But that's often the way to get something that just works intuitively — the rules need to be a little complicated.

[jckarter]

If you don't like a keyword, how about using a = b for binding and a := b for reassignment? That has some precedent in traditional notation.

[StefanKarpinski]

I gotta say, I'm pretty happy with how this works right now. Give it a chance to grow on you. If we get lots of people who hate the way this works, we should obviously reconsider, but otherwise we'll leave it as is.

[ivanmantova]

I tend to agree with @StefanKarpinski. I quite like the way it works now. It's straightforward, clean and simple, and it avoids dangerous unexpected behavior (like changing the value of a global inadvertently).
Given that you don't have to declare variables, it sounds pretty reasonable to me that you introduce new variables at assignment.

[thomie]

Thank you for your explanation. I understand you want x=0; x = x + 1 to just work, and at the same time prevent accidental assignments to global variables. Although @jcarter's suggestions of var x = 0; x = x + 1 or x = 0; x := 1 would have you scan less lines of code to find out what a function does, Julia scoping rules are a trade off between 'the right thing to do' and convenience.

But some day a Julia user might learn that it's bad practice to use global variables. So she decides to change this code:

x = 0
function f()
    x = 10
end
f()
println(x)

into this code:

function namespace()
    x = 0
    function f()
        x = 10
    end
    f()
    println(x)
end
namespace()

The first prints 0, the second prints 10. I don't like this at all. Those 2 things should really do the same thing. We're going to end up with a 'julialint strict' some day, which besides enforcing local, prevents you from writing code like this:

let x = 0
    let x = 10
    end
    println(x)
end

let x = 0
    let
        x = 10
    end
    println(x)
end

The first prints 0, the second prints 10. I'm sure there will be more examples like this.

The current scoping rules get out of your way for a long time. Until they don't. This may happen once per week, once per year, but it will happen. To you. And it will be nasty.

Julia gets a lot of things right, please make the scoping rules be one of them.

[JeffBezanson]

You definitely have a point here.

Of course we don't want to "fix" this by making the first case print 10.
If I take the requirements to be (1) addressing the issue you bring up, (2) allow plain "x=0" to introduce a local, then it seems the fix is to make assignment always introduce a new local, and require something like "outer x = 10" to re-assign a variable from an outer scope.

This might be acceptable, since I find it quite rare to modify one function's variables from an inner function.
This is only annoying if you want to "delay" part of a function by wrapping it:

function f()
  x = 0
  ...
  x = 10
  ...
end

to

function f()
  x = 0
  ...
  function delayed()
    x = 10
    ...
  end
end

in that case you would have to add "outer" declarations. Also a bit of a concern for macros, but there it could be solved by a hygiene system in the future.

[StefanKarpinski]

I agree that there's something wrong here. The smoking gun for me is not being able to move code from a local scope to a global scope and have it work the same. Another smell is how the repl needs to use trickery to emulate a local scope that's pseudo-global.

The question is how to fix it without sacrificing the good parts of what we have now.

[StefanKarpinski]

Here's an idea for how to fix this. The idea is to have "hard scope" and "soft scope". Functions introduce "hard scope" whereas various blocks introduce "soft scope". The difference would that in a new hard scope, assignment without a "global" or (hypothetical) "outer" declaration always creates a new variable local to that scope, regardless of whether a variable with that name exists in the outer scope. In a new soft scope, assignment updates a variable in the outer scope if such a variable exists or creates a new inner-scoped variable if no such variable exists. Some examples of the proposed behavior (not how things work now):

x = 0
while x < 10
  println(x)
  x += 1
end

Works: prints 0 through 9.

x = 0
function step()
  println(x)
  x += 1
end
while x < 10
  step()
end

Fails: complains that x inside of step() is undefined.

[JeffBezanson]

That is actually a fix for issue #424, not this one.

[StefanKarpinski]

Yeah, that's true. I thought this conversation had converged with that one.

[StefanKarpinski]

Nevertheless, what do you think?

[JeffBezanson]

The hard-scope, soft-scope distinction actually already existed, but the problem was that I sometimes expanded expressions in a file before prior expressions had been evaluated. I will do a commit soon that fixes this to make loading from a file identical to entering stuff in the repl.

[StefanKarpinski]

!!!

[StefanKarpinski]

This #423 (comment) is the main potential issue here but it seems like we're at an optimal behavior now. Ergo, closing.