allow slurping in lhs of assignment

[simeonschaub]

This is currently branched off #37268. I will rebase once that is merged, but for now, only the last commit is relevant. This implements @StefanKarpinski's comment in #2626 (comment). I am not sure, slurp_rest is the best name and API for this, perhaps it would make more sense to extend iterate_and_index to return a third function for slurping, since slurp_rest currently relies on what iterate does, which could lead to gotchas if users overload iterate_and_index.
fixes #2626

[JeffBezanson]

Cool! My current thinking is that if we do this, it should use Base.rest (or tail?), which can be overloaded appropriately --- arrays can return arrays (or views?), if you had a linked list (:rofl:) it could return the list tail, tuples can return tuples, and infinite or lazy iterators can return Rest iterators. I don't think this is necessarily inconsistent with varargs, since arguments are always implicitly a tuple

[simeonschaub]

Ok, that can certainly be discussed, I was just under the impression that Stefan's comment concluded the discussion in #2626. I personally would still favor just always returning tuples, because I think it's going to be difficult to find a rule for which container types should slurp to which type for the rest and it's simpler to remember that it's just always a tuple. For example, what should a..., = 1 do? a, b... = 1? I see that it's not necessarily inconsistent with varargs, but to me, it still feels a bit weird for such a language feature to change meaning based on the input type instead of returning a tuple.

[JeffBezanson]

It's not changing meaning, it's using a different implementation of that meaning. Multiple data structures support the concept of rest, and each one implements it by returning a smaller version of the same structure. Using only tuples prevents e.g. infinite iterators from working, when they could work. Lots of syntax features like this lower to overloadable functions.

[simeonschaub]

I mostly want this just for tuples, so what to do for other iterators is not a hill I'm willing to die on. What do you reckon would be the best way forward here? Should this just be triaged, or would it make sense to reopen the discussion in #2626, so that people that participated there can chime in?

[JeffBezanson]

Ok, let's have this call Base.rest(itr, state), and it can be overloaded to be eager for e.g. arrays and tuples.

[simeonschaub]

So I'm thinking as the default return types of Base.rest:

• Tuple -> Tuple
• AbstractArray -> Vector
• everything else -> Iterators.rest

Does that make sense, or is there anything else that would make sense to special-case? Perhaps UnitRange?

[simeonschaub]

Ah, this doesn't work with just passing the iteration state to Base.rest, because with #37268, iterate_and_index(x)[1], won't tell us anything about the current iteration state for things like tuples or vectors, since it just returns the object itself and indexes with the index generated by lowering. We either need to change how iterate_and_index works for those or need to pass the index generated by lowering to Base.rest as well.

[JeffBezanson]

So the interesting thing is that this feature is slightly incompatible with #37268, since that change means we don't always have an iteration state available. To get elements, you don't need the iteration state, but for ... you do. Hence the need for _rest. But it probably doesn't matter much since you only have to implement _rest if you implemented indexed_iterate before (right?), and nobody was supposed to do that :)

[JeffBezanson]

I think we are not going to do the indexed_iterate lowering change for now, so this can be rebased on master and we can merge it.

[simeonschaub]

So we decided to make Base.rest part of the public API, but not export it, right? Should probably have a docstring and some separate tests then. Also still needs news. The actual implementation should pretty much be done from my side now.

[JeffBezanson]

How about collect(T, Iterators.rest(a, state))? Then for the 1-argument case Vector{T}(a). Better to avoid repeated push! since it's a bit slow.

[simeonschaub]

Then for the 1-argument case Vector{T}(a).

That won't work for multi-dimensional arrays, unfortunately:

julia> Vector{Int}([1 2; 3 4])
ERROR: MethodError: no method matching Array{Int64,1}(::Array{Int64,2})
Closest candidates are:
  Array{Int64,1}(::AbstractArray{S,N}) where {T, N, S} at array.jl:562
  Array{Int64,1}() where T at boot.jl:425
  Array{Int64,1}(::UndefInitializer, ::Int64) where T at boot.jl:406
  ...
Stacktrace:
 [1] top-level scope at REPL[25]:1

collect also doesn't always return a Vector and I think it would be weird if Base.rest(::A) and Base.rest(::A, st) returned different types.

[simeonschaub]

Also, note that:

julia> Base.IteratorSize(Iterators.rest([1 2; 3 4], 1))
Base.SizeUnknown()

so collect will end up just using push! anyways.

[JeffBezanson]

Looking good.

[StefanKarpinski]

OMG, I did not think this was happening!