make &x sugar for RefValue(x)

[stevengj]

As suggested in #27563. Note that &x was already parsed as a special expression node Expr(:&, :x), which previously was used for a deprecated ccall syntax. So all this PR does is to change the lowering to RefValue(x) (eliminating the deprecation).

This is especially useful for "scalarizing" arguments in dot calls (it also works with @.). For example:

julia> x = [3,4,5]; 

julia> string.(x)
3-element Array{String,1}:
 "3"
 "4"
 "5"

julia> string.(&x)
"[3, 4, 5]"

julia> @. string(&x)
"[3, 4, 5]"

I updated the Base, stdlib, and test code to use the & syntax and it looks pretty natural to me.

To do (assuming the consensus is in favor):

• Tests. (Already exercised in Base.)
• Documentation.

[andyferris]

Out of curiousity, why is this lowering logic instead of using a standard unary operator?

Also, have we considered how this relates to all the @ stuff to get references to fields/array elements that Keno prototyped? E.g being able to get a reference to a field of a mutable struct via &a.b or something like that (I haven’t really thought about precise syntax here) would be super awesome.

[andyferris]

Following up on the discussion in #27563, my gut instinct is to be a little defensive against having & sometimes mean “a method of &” and sometimes not.

(I guess I feel we could afford to be a bit more creative and not pile meanings into the same symbol.... on that note I’d honestly prefer to move Boolean and bitwise operations to English and double down on references in a serious way - I’m thinking we could eventually in the distant future even support some Pony or Rust type of stuff, but that’s surely an aside!).

[stevengj]

Out of curiousity, why is this lowering logic instead of using a standard unary operator?

Technically, the parser is treating it as a "syntactic" unary operator. I guess your question is, why is it parsed as a special AST node rather than as a function call?

It all boils down to wanting to distinguish between the programmer writing &x and (&)(x). If you parse &x the same as (&)(x), then that information is lost, even for macro writers. Something like the @. macro would have to distinguish unary calls to functions spelled "&", which would be problematic — what if there was a different meaning assigned to & in the local scope?

Basically, if we want &x to mean Ref(x), I don't think we want to do it through a call to the & function.

A separate question is whether we want &x to act like Ref(x). This discussion should probably take place mostly in #27563. I guess the main argument is that acting like Ref(x) is very useful for opting out of broadcasting, and &x has at least some precedent in C/C++, whereas the unary bitwise (&)(x) is pretty non-useful and takes up valuable ASCII real estate.

[StefanKarpinski]

Should the precedence of & be higher than that of []?

[staticfloat]

I would vote no, but I think that's mostly the C/C++ roots in me than any principled argument.

[andyferris]

I was speculating in #27563 (comment) that maybe &a[i] could (in the future) be convenient syntax a reference to the ith element of a::Array.

[StefanKarpinski]

It looks pretty good to me. I still really wish this gave us a concise syntax for operations like sum(A, i) dropping the reduced dimension—or conversely, a way to ask to keep a dimension in reductions so that the default can be to drop them. It doesn't do that since scalar i and Ref(i) are both zero-dimensional, so they should behave similarly with respect to that. And somewhat related: are we sure we want to spend such precious syntax on this? If it solved more problems, I'd be more sold on that.

[andyferris]

A separate question is whether we want &x to act like Ref(x)

I realized that I've said a lot, but not this: I really like the idea of using & for references. Also treating these as special expressions rather than calls to & might enable some of the stuff I was speculating (wildly) about in #27563 (comment). +1

[mbauman]

Back when we introduced APL indexing, I was convinced we'd need a syntax to preserve leading scalar dimensions in indexing — and thought the & syntax could be used to create a 1-length vector. That's largely not been the case… but for broadcasting purposes there's not a huge difference between a zero-dimensional thing and a 1-element vector. There are two ways they're different: 1) the all-zero-dimensional case will return an unwrapped scalar, whereas the inclusion of any non-zero-dimensional thing will return an array of some sort. 2) the BroadcastStyle container promotion system treats zero-dimensional things as things to be ignored when deciding what kind of container to return. So while we could make this syntax do double-duty with dimension-preserving-indexing by returning a 1-element vector thing instead of a Ref, it'd make its use in broadcasting much less compelling.

Now reductions dropping/keeping dimensions is definitely a more common issue, but unlike indexing, the shape of the dims argument isn't meaningful: it's just an iterable list of dimensions. I'm not sure I've thought about this before, but now that dims is a keyword argument it dawns on me that we could have a parallel squeeze keyword.

[StefanKarpinski]

it dawns on me that we could have a parallel squeeze keyword.

Yeah, not a bad idea. In any case, carry on. The arguments for using & for more kinds of reference constructions is appealing and makes having it be syntax more justifiable:

1. &x for a reference to a single value
2. &a[i] for a reference to a slot in an array
3. &x.f for a reference to a field in a struct

Having syntax here is useful because these all create different kinds of things: RefValue, RefArray, RefStruct (hypothetical) but are conceptually similar.

[stevengj]

Right now, &a[i] and &a.f parse as &(a[i]) and &(a.f). a&[i] parses as the bitwise & of a and [i]. a&.f is a syntax error.

If &a[i] lowers to Ref(a, i), then I guess you would need to explicitly type Ref(a[i]) if you want that?

[StefanKarpinski]

Right now, &a[i] and &a.f parse as &(a[i]) and &(a.f).

Right, I'm proposing that we change that.

[stevengj]

Just to be clear, are you proposing a change in the parsing, or just in the lowering? i.e. we could keep the parsing of &a[i] the same and just lower it differently, but then it would lower the same as &(a[i]).

Or we could parse &a[i] to e.g. Expr(:&, :a, :i), which lowers to Ref(a, i), whereas &(a[i]) continues to lower to RefValue(a[i]).

And I'm not sure what to do about &a.f … what does e.g. RefStruct(a, :f) even mean if a is an immutable struct?

[andyferris]

Just to be clear, are you proposing a change in the parsing, or just in the lowering?

I would guess that both &a[i] and &a.b parse in the form Expr(:&, ...)? (Wouldn't that be friendly for the @. macro for exactly the same reasons as &a?)

And I'm not sure what to do about &a.f … what does e.g. RefStruct(a, :f) even mean if a is an immutable struct?

I find this to be a very interesting question. We even say that Julia bindings have reference semantics so I sometimes get my head in a spin as to what exactly Ref(a) is meant to be when a is immutable! :)

I'm guessing we want & to be a reference we write to, but this is actually more powerful than we need in many instances (like broadcasting, @., passing constant values to fortran/C, etc). Would an immutable reference make sense?

The other thing to consider is that the dot syntax refers to properties not fields now, which is a whole another kettle of fish. You might have a property that you can read and write to, but which you can't get e.g. a pointer to. Is a Ref{T} something that you can convert to a Ptr{T}? Or a one-element container? Or something else?

[stevengj]

Wouldn't that be friendly for the @. macro for exactly the same reasons as &a?

In this PR, the lowering is completely independent of the @. macro, which I think is the way it should be. You shouldn't need the @. macro to use this syntax.

[JeffBezanson]

In these proposals, & can be seen as a macro (we could even implement it that way if we really wanted to). So it can continue to parse as a 1-argument expression with head :&. One suggestion seems to be parsing &a[i] and &(a[i]) differently, but that doesn't seem ideal to me. It would be best if we can be certain that RefValue(a[i]) and RefArray(a, i) behave the same --- just two different implementations of referencing the value a[i]. Then adding fancier referencing might be non-breaking.

[JeffBezanson]

I should clarify that they can't behave the same with respect to mutation, but they can be the same in all other ways.

[stevengj]

It would be best if we can be certain that RefValue(a[i]) and RefArray(a, i) behave the same.

It seems to me that these are different concepts. RefValue(a[i]) should be the same as let x = a[i]; RefValue(x); end, i.e. a reference to a "copy" of the value, not a reference that lets you mutate an entry of the array (except to the extent that x itself is mutable).

they can't behave the same with respect to mutation

Exactly.

The question is, which concept should &a[i] refer to, RefValue (the current version of this PR) or RefArray? And should it be the same as &(a[i])? And what, if anything, should we do differently for &a.f?

[JeffBezanson]

But those would be the same thing as far as broadcasting is concerned, right?

[vtjnash]

It should be the same on the RHS, but may be different on the LHS?

[stevengj]

The proposed &x syntax wouldn't be limited to broadcasting (there were a whole bunch of places in Base that called Ref explicitly and could be changed to &, see the second commit in this PR), so you have to assume that the resulting Ref object may be used on the LHS of an assignment, and hence we have to decide what the mutation semantics are supposed to be.

[StefanKarpinski]

Parsing &(...) as Expr(:&, ...) everywhere seems fine. I think the ref lowering should not be specific to @. howeve. Yes, it's useful for that and that's the motivation here but my understanding was that we're talking about making &x a syntax for RefValue(x)—at least that's what this PR implements.

The proposed behavior of ra = &a[i] and rf = &x.f would be that ra[] = y would have the same effect as a[i] = y and rf[] = z the same as x.f = z. I'm not sure if we want that but we already have RefArray so it seems like RefStruct would be only natural as an analogue. As r-values in broadcasting, &a[i] would behave just like &(a[i]) and likewise &x.f as &(x.f) but as l-values they would potentially behave differently, and outside of broadcasting they would allow mutation of arrays and structs, respectively through a reference object.

I guess what I'm getting at here is that I don't find writing &x that much better than writing Ref(x), at least not on its own. So as it stands the only thing this PR fixes is letting & be a nice syntax for "scalarizing" things in broadcasting—and I'm not sure that's enough benefit for such a slick syntax. As soon as & becomes part of a uniform syntax for taking safe references to things, through which they can be mutated, then the whole things starts to seem worth having syntax for.

[stevengj]

I don't find writing &x that much better than writing Ref(x)

If x is an array, you currently have to type Base.RefValue(x) or something similarly convoluted to scalarize it, and in and @. expression you have to add $ — Ref(x) doesn't work. So even as merely a synonym for RefValue the &x syntax has some real utility.

I'm not opposed to lowering &x[i] to Ref(x, i), except:

1. RefArray only works for arrays. At a syntactic level, what do we do with &x[i] if x is some other type, e.g. a dictionary, that happens to support getindex?

2. Do we want a short syntax for RefValue(x[i]) as well, e.g. &(x[i]) (which is currently parsed the same as &x[i]? Similarly for &x.f vs. &(x.f)?

3. How do we implement RefStruct for immutables?

[StefanKarpinski]

So even as merely a synonym for RefValue the &x syntax has some real utility.

Yes, I get that, but I feel like it's a weak motivation for using some very prime syntax.

[andyferris]

I agree with what Stefan is saying. This kind of syntax seems valuable to solve some of our more general reference problems.

I have always been slightly uneasy about (ab)using Ref to scalarize broadcast. I feel there is a difference between "make me a container with this 1 element" (what broadcast fundamentally needs) and what a reference is for (it points to some data, you might be able to mutate it or observe other people mutating it).

If all this is just ugliness from RefValue and dollar signs in @. macros or whatever, and we want to make broadcast easier, another way out might be to define (immutable) struct Box{T}; x::T; end; getindex(b::Box) = b.x and use this instead of Ref, RefValue, etc in broadcast-land. (I realize that "Box" already means something in Julia but that's not normally visable from user-land, and "box" is just an example). I've been using StaticArrays.Scalar this way for years and it's worked great.

[chethega]

While I really like the syntax, it is somewhat problematic with respect to its C connotations: A typical use could look like

pt = &C_NULL
ccall(:posix_memalign, Cint, (Ptr{Ptr{Nothing}}, Csize_t, Csize_t), pt, 16, 64)
#use pt[]

This has the same symbol and typing behavior as the C addressof, with entirely different mutation semantics: In a different scenario, we could have ref = &var, which pulls a copy of immutable var. This is so close and yet so far away from C that I would consider its optical closeness to C adressof as an anti-feature.

Can't we take a different unary ascii for this and use & somewhere else?

[mbauman]

I don't see how that's so problematic — in your example, pt is a mutable 0-dimensional container. And since it's a container, it broadcasts like one. Even when you're wholly in C, the common mental model is that pointers are how you refer to the "box" wherein values are stored. Regardless of what the ccall-reference and broadcast-boxing syntaxes are, they're both creating the same thing — a Ref.

We also just don't have many options. It's gotta be significantly shorter than (x,) to be worth the syntax, so we really are limited to one-character operators.

That said, since this is "just" a shorthand we could consider unicode. □ (\square) looks to be available (not entirely available as it's a valid leading identifier character) and representative.

[chethega]

The confusing issue with C is that, with this, one would use &var in both C and julia to get a pointer / pointer-analogue to a memory address that initially contains the value of var.

In C, the resulting pointer &var points at var itself, i.e. (&var)[0] += 1 ; increments var. In julia, the resulting RefValue points at a newly allocad address that is initialized with the value of var, and (&var)[] += 1 does not affect var at all. The C & acts on bindings/symbols (lvalues), while julia & would act on values (rvalues). The difference is so large that both have almost nothing to do with each other, except for a similar signature and the same ascii symbol.

My example was bad, since C_NULL is an rvalue and not an lvalue. But I could see people getting confused by the semantics when staring at code with &var where var is both lvalue and rvalue. Ref(var) makes it clear that this is a function call and var is interpreted as rvalue, while &var could be mistaken for a syntactic construction (instead of an ordinary unary operator) that treats var as an lvalue.

On the other hand, the special parsing is probably necessary: I could see code using foo(args...) = (&)(args...), which should not return a Ref when called with a single argument.

[JeffBezanson]

For syntax as nice as this, we should spend some more time thinking about how references can/should work more generally. For example &a[i] could do something special.

[StefanKarpinski]

On triage it came up that having &a[i] as syntax for RefArray(a, i) would fit with this and is a more broadly useful syntax. Similarly for &x.f and various other kinds of Ref constructs.

[Keno]

We discussed this on triage. Points that were discussed

1. &a[i] need not mean &(a[i]), but could be a view syntax
2. Do we want to make this syntax align with WIP: Make mutating immutables easier #21912
   Tabled for this week to let people think about it more for next time.