Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iterators.map #34352

Merged
merged 13 commits into from
Jun 18, 2020
3 changes: 2 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Build system changes

New library functions
---------------------

* `Iterators.map` is added. It provides another syntax `Iterators.map(f, iterators...)`
for wiring `(f(args...) for args in zip(iterators...))` ([#34352]).
tkf marked this conversation as resolved.
Show resolved Hide resolved

New library features
--------------------
Expand Down
42 changes: 31 additions & 11 deletions base/iterators.jl
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ using .Base:
@inline, Pair, AbstractDict, IndexLinear, IndexCartesian, IndexStyle, AbstractVector, Vector,
tail, tuple_type_head, tuple_type_tail, tuple_type_cons, SizeUnknown, HasLength, HasShape,
IsInfinite, EltypeUnknown, HasEltype, OneTo, @propagate_inbounds, Generator, AbstractRange,
LinearIndices, (:), |, +, -, !==, !, <=, <, missing, map, any, @boundscheck, @inbounds
LinearIndices, (:), |, +, -, !==, !, <=, <, missing, any, @boundscheck, @inbounds

import .Base:
first, last,
Expand All @@ -24,6 +24,26 @@ import .Base:

export enumerate, zip, rest, countfrom, take, drop, takewhile, dropwhile, cycle, repeated, product, flatten, partition

"""
Iterators.map(f, iterators...)

Create a lazy mapping. This is another syntax for writing
`(f(args...) for args in zip(iterators...))`.

!!! compat "Julia 1.5"
This function requires at least Julia 1.5.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe cross-reference with the docstring of Base.Generator

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think Base.Generator is included in the documentation. This is partially why I want to add Iterators.map.

# Examples
```jldoctest
julia> collect(Iterators.map(x -> x^2, 1:3))
3-element Array{Int64,1}:
1
4
9
```
"""
map(f, args...) = Base.Generator(f, args...)

tail_if_any(::Tuple{}) = ()
tail_if_any(x::Tuple) = tail(x)

Expand Down Expand Up @@ -314,25 +334,25 @@ function _zip_min_length(is)
end
end
_zip_min_length(is::Tuple{}) = nothing
size(z::Zip) = _promote_shape(map(size, z.is)...)
axes(z::Zip) = _promote_shape(map(axes, z.is)...)
size(z::Zip) = _promote_shape(Base.map(size, z.is)...)
axes(z::Zip) = _promote_shape(Base.map(axes, z.is)...)
_promote_shape(a, b...) = promote_shape(a, _promote_shape(b...))
_promote_shape(a) = a
eltype(::Type{Zip{Is}}) where {Is<:Tuple} = _zip_eltype(Is)
_zip_eltype(::Type{Is}) where {Is<:Tuple} =
tuple_type_cons(eltype(tuple_type_head(Is)), _zip_eltype(tuple_type_tail(Is)))
_zip_eltype(::Type{Tuple{}}) = Tuple{}
@inline isdone(z::Zip) = _zip_any_isdone(z.is, map(_ -> (), z.is))
@inline isdone(z::Zip, ss) = _zip_any_isdone(z.is, map(tuple, ss))
@inline isdone(z::Zip) = _zip_any_isdone(z.is, Base.map(_ -> (), z.is))
@inline isdone(z::Zip, ss) = _zip_any_isdone(z.is, Base.map(tuple, ss))
@inline function _zip_any_isdone(is, ss)
d1 = isdone(is[1], ss[1]...)
d1 === true && return true
return d1 | _zip_any_isdone(tail(is), tail(ss))
end
@inline _zip_any_isdone(::Tuple{}, ::Tuple{}) = false

@propagate_inbounds iterate(z::Zip) = _zip_iterate_all(z.is, map(_ -> (), z.is))
@propagate_inbounds iterate(z::Zip, ss) = _zip_iterate_all(z.is, map(tuple, ss))
@propagate_inbounds iterate(z::Zip) = _zip_iterate_all(z.is, Base.map(_ -> (), z.is))
@propagate_inbounds iterate(z::Zip, ss) = _zip_iterate_all(z.is, Base.map(tuple, ss))

# This first queries isdone from every iterator. If any gives true, it immediately returns
# nothing. It then iterates all those where isdone returned missing, afterwards all those
Expand Down Expand Up @@ -388,7 +408,7 @@ _zip_iterator_eltype(::Type{Is}) where {Is<:Tuple} =
_zip_iterator_eltype(tuple_type_tail(Is)))
_zip_iterator_eltype(::Type{Tuple{}}) = HasEltype()

reverse(z::Zip) = Zip(map(reverse, z.is))
reverse(z::Zip) = Zip(Base.map(reverse, z.is))

# filter

Expand Down Expand Up @@ -971,7 +991,7 @@ end
isdone(P) === true && return nothing
next = _piterate(P.iterators...)
next === nothing && return nothing
return (map(x -> x[1], next), next)
return (Base.map(x -> x[1], next), next)
end

@inline _piterate1(::Tuple{}, ::Tuple{}) = nothing
Expand All @@ -992,10 +1012,10 @@ end
isdone(P, states) === true && return nothing
next = _piterate1(P.iterators, states)
next === nothing && return nothing
return (map(x -> x[1], next), next)
return (Base.map(x -> x[1], next), next)
end

reverse(p::ProductIterator) = ProductIterator(map(reverse, p.iterators))
reverse(p::ProductIterator) = ProductIterator(Base.map(reverse, p.iterators))

# flatten an iterator of iterators

Expand Down
1 change: 1 addition & 0 deletions doc/src/base/iterators.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Base.Iterators.repeated
Base.Iterators.product
Base.Iterators.flatten
Base.Iterators.partition
Base.Iterators.map
Base.Iterators.filter
Base.Iterators.accumulate
Base.Iterators.reverse
Expand Down
7 changes: 7 additions & 0 deletions test/iterators.jl
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,13 @@ end
@test length(zip(1:3,product(1:7,cycle(1:3)))) == 3
@test length(zip(1:3,product(1:7,cycle(1:3)),8)) == 1

# map
# ----
@testset "Iterators.map" begin
@test collect(Iterators.map(string, 1:3)::Base.Generator) == map(string, 1:3)
@test collect(Iterators.map(tuple, 1:3, 4:6)::Base.Generator) == map(tuple, 1:3, 4:6)
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it might be good to test that Iterators.map can be applied to infinite iterables such as Iterators.cycle(1:3) and has the expected behaviour for the first n iterates.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason why testing infinite iterators is interesting here? I'm testing these two invocations since Base.Generator has different methods for them. So, I think it's somewhat meaningful to make sure we can hit these methods. Other than that, I don't think we need to add tests specially for Iterators.map and they can be done in Base.Generator. I think it makes sense to rename direct uses of Base.Generator in Base/stdlib. But I think it'd be better to do this in a separate PR to make this PR more "atomic".


# rest
# ----
let s = "hello"
Expand Down