improve documentation of soft vs. hard scope

[StefanKarpinski]

See julia-dev discussion here: https://groups.google.com/forum/#!topic/julia-dev/-qIVQGq-f_U

[mauro3]

There was some more discussion on julia-users https://groups.google.com/d/msg/julia-users/bQxvNw21hC4/s3wHzRCe9UQJ. There a good reference to an old comment of Stefan was made: #423 (comment)

In that julia-users thread, I posted these soft/hard rules:

In a soft scope (while loops, for loops, try blocks, catch blocks, let blocks)
s1) normal assignment `x = 5`
  1) if a binding exists in the global scope, assign to that
  2) if no binding exists, make a new *local* binding
s2) local assignment `local x=5`
  1) make a new local binding
s3) global assignment `global x=5`
  1) make a new global binding

In a hard scope (functions, type)
h1) normal assignment `x = 5` 
  1) make a new *local* binding
h2) local assignment `local x=5`
  2) make a new local binding (i.e. equivalent to h1.1)
h3) global assignment `global x=5`
  3) make a new global binding

If no assignment happens, just reading, then hard and soft are equivalent.

However, these rules break down for nested functions, which seem to have a soft scope?! Here an old example: #423 (comment):

function namespace()
    x = 0
    function f()
        x = 10
    end
    f()
    println(x)
end
namespace() # prints 10, suggesting the inner function has soft-scope!?

I think this is a bug, right? Or should it feature in the rules: only outer functions have hard scope.

[Sisyphuss]

Hello mauro3, your rules make sense to me. I have done an notebook illustration based on your rules. You can find it at https://drive.google.com/open?id=0B30Cah_MN0i0TlZfRlBXLWgxbE0&authuser=0

I tested it in IJulia notebook. Behaviors in script remain untested.

[Sisyphuss]

Furthermore, for nested function, only the outermost function should be considered to be a hard scope. The inner functions are soft scope.

[mauro3]

Here is the notebook rendered: http://nbviewer.ipython.org/gist/mauro3/9b92ac4adfb1d845f3fc. Looks like the rules work for your examples, thanks!

About the semi-soft scope of inner functions, I filed #10559 for discussion.

[StefanKarpinski]

This really ought to be documented. @JeffBezanson, you implemented it and probably understand the subtleties the best of anyone and thus are in the best position to document it. Are you willing to do that?

[mauro3]

I think I now see through it but I can't take care of it until July. See @elextr and @JeffBezanson comments here: #10559 (comment)

[asoffa]

It would be great if someone could add the rationale behind why begin and if blocks behave differently from other types of blocks. The general rule seems to be that a block (defined as the lines of code between a keyword requiring an end and the end) defines a new, inner level of scope, but that begin and if blocks are exceptions. (Additionally, functions defined without the function keyword also introduce a new level of scope.) Indeed, this is stated in the docs:

"Notably missing from this list are begin blocks and if blocks, which do not introduce new scope blocks."

But why? What is so special about begin and if (but not, e.g., try) blocks that they ought to break the rules? I personally would prefer a consistent scheme and just add a local declaration before the begin or if block when I want a variable therein to "leak," but perhaps I am missing something here.

Making matters even more confusing (though this is probably just a bug) is that there is an exception to the exception when a begin or if block occurs at the global level in that variables declared as local do stay within the scope of the begin or if block:

begin
    x = 1
    const y = 2
    local z = 3
end
println(x)  # works because of the exception for begin and if blocks
println(y)  # works because of the exception for begin and if blocks
println(z)  # fails because of the exception to the exception for begin and if blocks

So, what is the rationale/justification behind this complexity?

And thank you for the fantastic language you have coming along!

[vtjnash]

begin exists as a way of grouping statements without introducing a new scope (it's scoped-counterpart is named let)

[asoffa]

I am aware of let, and looking over the docs again, it looks like the rationale for begin blocks is that a begin block provides an alternative, equivalent way of writing a sequence of statements wrapped in parentheses (which also doesn't introduce a new level of scope). This allows code of the form

my_anon_func = x -> begin
    #...
end

to be written without adding two new levels of scope (one for x -> ... by the "new function" scope rule and the other for begin) in cases where the alternative

my_anon_func = function (x)
    #...
end

isn't desired for (e.g.) style uniformity in cases where multiple anonymous functions that are mostly one-liners are used. Is there any other use case for begin? (And if not, couldn't a let here be automatically corrected so that it is equivalent to a begin in this case under the hood?) I am not seeing any other use case for begin in the docs other than assigning the final expression of lines of code to a variable when I want the rest of the block in the same scope as the variable assignment, but why this would be useful and worth a new type of block construct (not to mention one that breaks the usual scope rule for blocks) is not explained.

And the exception for if blocks was likely made so that new variables could be defined without a local declaration before the if block when their initialization depends on a particular Boolean condition, though one could just as easily write

x, y = if condition
    1, 2
else
    3, 4
end

(adding new lines after commas if needed for longer variable names and/or initialization expressions) in place of

# `local x, y` not currently needed here
if condition
    x = 1
    y = 2
else
    x = 3
    y = 4
end

Was the decision to make if blocks unscoped made just for the use case above?

[mbauman]

Is there any other use case for begin?

I highly recommend digging into the source if you're interested in these sorts of questions. In this case, you'll find that almost every usage of begin within the standard library is as an argument to a macro. There are many examples outside base, too, like Match.jl and JuMP, which enable custom block-like constructs.

As far as discussions of if's scope, that's harder to find with search. There was some previous discussion at #6872.

[asoffa]

Ah yes, I didn't consider the utility of an unscoped begin block as a macro argument - that makes a lot of sense. And the discussion you linked me to provides at least a partial answer for the case of if blocks. Thank you for filling me in!

As for my question about the code snippet above (shown below for reference), it is not a bug. local implies 'not global,' even if it appears in global scope. Consider the following code (called from global scope):

local z = 3
println(z)  # fails (ERROR: UndefVarError: z not defined)

x = 1
local x = 2
println(x)  # prints 1, not 2

Original code in question:

begin
    x = 1
    const y = 2
    local z = 3
end
println(x)  # prints 1
println(y)  # prints 2
println(z)  # fails (ERROR: UndefVarError: z not defined)

[mauro3]

It would be better to have using local in global scope throw an error than do something odd.
Concerning ifs: Matlab and Python don't introduce a new scope with them either. If you end up improving the docs, maybe best to do it against PR #12146; I have high hopes that it will get merged eventually...

[StefanKarpinski]

I believe this thing is that begin introduces a soft scope but not a hard scope. Therefore assignment inside a begin assigns to a variable in the surrounding hard scope, but you can force an assignment in the soft scope by using local. But I agree that this all still needs clarification.

[asoffa]

To illustrate what I think @StefanKarpinski is referencing (the following at global scope):

local x = 1  # currently works, but this `x` can never be used
local y      # fails (ERROR: syntax: misplaced "local" declaration)

begin
    local z  # ok here because `z` is assigned by the end of the block
    z = 1    # but `z` can't be used outside of the block
end

[asoffa]

@mauro3 - But ifs in Python follow the usual rule for scope in Python - that only definition-type blocks (def and class), not blocks in general, introduce new scope - rather than introduce an exception to the rule. (And then the usual rule can be explicitly overridden with the keywords global or nonlocal.) Somewhat similarly, Matlab has workspace and function scope.

On the other hand, Julia (seemingly) has more complex scope rules that vary on a case-by-case basis, which I understand is a trade-off between consistency and pragmatism. It would be nice if Julia's scope rules could be summarized more concisely as is the case with some languages (e.g. Python and Matlab). As it stands, types, functions, loops, let blocks, try-catch-finally blocks, and optionally macros are what introduce new scope. More of a mouthful than def and class! I suppose "blocks other than begin and if blocks" isn't the worst, though that doesn't have quite the same conceptual unity (though does probably address the most common use cases).

One way to make things look more uniform in Julia would be to make the closing keyword e.g. done instead of end for any block that doesn't introduce new scope (i.e. begin and if blocks), possibly even allowing let to close with end or done so that begin is no longer needed (and allowing if blocks also to close with either end or done for scoped vs. unscoped). I was not involved in making these sorts of decisions, so I'm just throwing this out there as a speculative idea.

[mauro3]

@StefanKarpinski, no, begin blocks do not make a soft scope:

julia> begin
       z=1
       end
1

julia> z
1

julia> let 
       zz=2
       end
2

julia> zz
ERROR: UndefVarError: zz not defined

The examples of @asoffa must be bugs, unreported as far as I can tell. Can you file an issue for this?

[mbauman]

It's #10472.

[mauro3]

I thought this issue seemed familiar, but failed to search in the closed issues, thanks!

[asoffa]

Here's another issue (either a bug or missing documentation): functions and e.g. let blocks behave differently when used at the global level:

x = 1

function f()
    x = 2
end
f()
println(x)  # prints 1

let
    x = 3
end
println(x)  # prints 3  <--- desired?

The behavior of f above is consistent with what is described in the current "Scope of Variables" documentation and in #11801. But if the behavior of the let block above is desired, then function and non-function scoped blocks introduce scope in different ways when used at the global level. Are functions meant to be special in this sense?

[mauro3]

This is soft vs hard scope. Have a read of the overhauled scope section: #12146 (here a semi-rendered version.

[asoffa]

@mauro3 - Thank you for pointing me to the upcoming scope documentation. This is much more complete, and the table at the beginning is really helpful for seeing which language construct introduces which type of scope (or lack thereof in the case of begin and if).

Julia's scope rules now seem more complex than ever, requiring an understanding of the the distinction between global vs. hard local vs. soft local vs. no new scope for each language construct along with how each scope-related declaration (global, local, const) overrides the usual rules. Perhaps the following "road map" (or something similar) could be added for conveying what type of scope each type of block introduces (which the scope table then elaborates explicitly):

module-type blocks ---> global scope
definition-type blocks ---> hard local scope
conditional control flow and begin blocks ---> no new scope
other control flow and let blocks ---> soft local scope.

This would provide a higher-level starting point that would make learning the scope table easier.

[mauro3]

Fixed with #12146, please close.