Add Iterators.map #34352

tkf · 2020-01-12T01:16:59Z

As @JeffBezanson mentioned in #34033 (comment), it makes sense to have Iterators.map. This PR adds it (with some tests).

StefanKarpinski · 2020-01-13T17:08:57Z

NEWS.md

@@ -19,7 +19,7 @@ Build system changes

 New library functions
 ---------------------
-
+* `Iterators.map` is added ([#34352]).


Might want to indicate what it does.

JeffBezanson · 2020-01-13T19:18:08Z

base/iterators.jl

@@ -22,7 +22,27 @@ import .Base:
    getindex, setindex!, get, iterate,
    popfirst!, isdone, peek

-export enumerate, zip, rest, countfrom, take, drop, takewhile, dropwhile, cycle, repeated, product, flatten, partition
+export enumerate, zip, rest, countfrom, take, drop, takewhile, dropwhile, cycle, repeated, product, flatten, partition, map


It should not be exported, since it will conflict with Base.map.

I'm a little sad that we didn't stick with the convention that the capitalized type name is the lazy construct while the lowercase function name is the eager construct. IIRC, we had agreed on that around 0.2 era but then didn't end up sticking to it.

tkf · 2020-01-14T00:32:32Z

I addressed the points in the review. Now Iterators.map is not exported and NEWS.md has a bit more description.

bramtayl · 2020-01-14T01:16:38Z

Can we have generators lower to Iterators.map instead of Base.Generator?

tkf · 2020-01-14T01:29:55Z

That sounds like a great idea. Same holds for Iterators.filter, Iterators.flatten, and Iterators.product. Can you open a new issue to track the idea?

bramtayl · 2020-01-14T01:57:38Z

Sure #34368

NEWS.md

Co-Authored-By: Stefan Karpinski <stefan@karpinski.org>

StefanKarpinski · 2020-01-15T18:52:25Z

Seems to have trailing whitespace.

tkf · 2020-01-15T20:21:52Z

buildbot/whitespace_linux32 (https://build.julialang.org/#/builders/150/builds/4226) seems to fail due to network failure (?):

fatal: git fetch-pack: expected ACK/NAK, got 'ERR upload-pack: not our ref 7ad6d9c178229d71962daa2913aee8b21865c175'
fatal: The remote end hung up unexpectedly

oxinabox · 2020-02-03T23:38:43Z

This has been in IterTools.jl forever as imap
I feel like adding every function from IterTools.jl into Base.Iterators is a kind of bad path.

Julia since the 0.7 release (and even before) has pushed towards functionality should live in packages, not that standard library.

In particular, Base.Iterators provides the minimal set of iterators to implement the rest of the standard libraries.

I also have this complaint about #34033 but I didn't see it before it was merged.

tkf · 2020-02-03T23:42:18Z

In particular, Base.Iterators provides the minimal set of iterators to implement the rest of the standard libraries.

#34033 doesn't seem to imply this.

oxinabox · 2020-02-03T23:45:25Z

As I said, I would have called #34033 out if I had have seen it before it was merged.

bramtayl · 2020-02-03T23:45:38Z

I support not adding unnecessary stuff to Base.

This issue isn't about adding a new iterator, just an interface to one that already exists (namely, Base.Generator).

tkf · 2020-02-04T00:23:27Z

Yes, that's a good point. It'd be nice to have a public and functional interface for constructing Base.Generator.

stev47

I support this change, since it essentially just adds an alias that is very intuitive.

stev47 · 2020-05-03T16:44:57Z

test/iterators.jl

+@testset "Iterators.map" begin
+    @test collect(Iterators.map(string, 1:3)::Base.Generator) == map(string, 1:3)
+    @test collect(Iterators.map(tuple, 1:3, 4:6)::Base.Generator) == map(tuple, 1:3, 4:6)
+end


it might be good to test that Iterators.map can be applied to infinite iterables such as Iterators.cycle(1:3) and has the expected behaviour for the first n iterates.

What is the reason why testing infinite iterators is interesting here? I'm testing these two invocations since Base.Generator has different methods for them. So, I think it's somewhat meaningful to make sure we can hit these methods. Other than that, I don't think we need to add tests specially for Iterators.map and they can be done in Base.Generator. I think it makes sense to rename direct uses of Base.Generator in Base/stdlib. But I think it'd be better to do this in a separate PR to make this PR more "atomic".

stev47 · 2020-05-03T16:45:29Z

base/iterators.jl

+
+!!! compat "Julia 1.5"
+    This function requires at least Julia 1.5.
+


Maybe cross-reference with the docstring of Base.Generator

I don't think Base.Generator is included in the documentation. This is partially why I want to add Iterators.map.

stev47 · 2020-05-03T19:44:20Z

What is the reason why testing infinite iterators is interesting here?

It tests the laziness of `Iterators.map` since non-lazy `map` can't handle infinite iterators.

I don't think we need to add tests specially for `Iterators.map` and they can be done in `Base.Generator`. I think it makes sense to rename direct uses of `Base.Generator` in Base/stdlib. But I think it'd be better to do this in a separate PR to make this PR more "atomic".

I agree.

stev47 · 2020-05-03T19:44:54Z

I don't think `Base.Generator` is included in the documentation. This is partially why I want to add `Iterators.map`.

It has a docstring on Julia 1.4.1 at least.

tkf · 2020-05-03T21:14:49Z

It tests the laziness of Iterators.map since non-lazy map can't handle infinite iterators.

Existing tests already check that Iterators.map returns Base.Generated.

It has a docstring on Julia 1.4.1 at least.

Having a docstring does not mean it's a public API. See #35715

stev47 · 2020-05-03T22:05:42Z

Having a docstring does not mean it's a public API. See #35715

Thank you for clarifying, but even internal documentation can benefit from cross-referencing. Mentioning the alias `Iterators.map` in the docstring of `Base.Generator` does not seem like any harm to me.

tkf · 2020-05-03T22:29:20Z

There is a discussion on the extension of Iterators.map #34368. Documenting what it returns limits such extension. In general, I think it's better to document how a certain object behaves rather than what it actually is.

tkf · 2020-05-03T23:05:55Z

I resolved the merge conflict and CI is all green now. Can someone merge this?

(ping @StefanKarpinski @JeffBezanson)

tkf · 2020-06-07T04:05:12Z

(ping @StefanKarpinski @JeffBezanson)

andyferris · 2020-06-18T23:11:15Z

This is exciting!

I'd note that the NEWS entry seems to imply that (f(x) for x in iter) would lower to Iterators.map(f, iter), but that doesn't appear to be implemented here. (I realize there is the discusison at #34368).

andyferris · 2020-06-18T23:12:58Z

Hmm... or maybe I'm just reading that sentence wrong?

tkf · 2020-06-18T23:18:18Z

My intention was that both (f(x) for x in iter) and Iterators.map(f, iter) evaluate to the same thing. But yeah, I think lowering the comprehensions to Iterators.map etc. would be great.

tkf added 3 commits January 11, 2020 17:15

Add Iterators.map

d08b151

Mention Iterators.map in NEWS.md

3f0652f

Fix test/iterators.jl

ff8734a

StefanKarpinski reviewed Jan 13, 2020

View reviewed changes

JeffBezanson reviewed Jan 13, 2020

View reviewed changes

tkf added 2 commits January 13, 2020 14:43

Unexport Iterators.map

e8f14a3

Explain what Iterators.map does in NEWS.md

4f1e092

StefanKarpinski reviewed Jan 14, 2020

View reviewed changes

NEWS.md Outdated Show resolved Hide resolved

Update NEWS.md

cf60cf2

Co-Authored-By: Stefan Karpinski <stefan@karpinski.org>

tkf added 2 commits January 28, 2020 00:38

Merge branch 'master' into Iterators.map

90d8729

Merge branch 'master' into Iterators.map

e6f59da

tkf mentioned this pull request Feb 3, 2020

convenience tmap function JuliaFolds/Transducers.jl#200

Closed

tkf mentioned this pull request Mar 29, 2020

Deprecate/remove map(f) and foreach(f) #35293

Closed

stev47 reviewed May 3, 2020

View reviewed changes

Merge branch 'master' into Iterators.map

a96f1e5

tkf mentioned this pull request May 3, 2020

upgrade threads from experimental to stable-with-caveats #35686

Merged

tkf added 4 commits June 6, 2020 17:14

Remove NEWS item

b04cf45

Merge branch 'master' into Iterators.map

60382d4

Add NEWS back

9d900d0

Fix compat

7757336

tkf requested review from StefanKarpinski and JeffBezanson June 7, 2020 00:20

tkf mentioned this pull request Jun 7, 2020

RFC: Define API invariance for keys, values, getindex and pairs #36175

Open

StefanKarpinski added the triage This should be discussed on a triage call label Jun 11, 2020

JeffBezanson merged commit 1f8b442 into JuliaLang:master Jun 18, 2020

JeffBezanson removed the triage This should be discussed on a triage call label Jun 18, 2020

tkf deleted the Iterators.map branch June 18, 2020 19:01

tkf mentioned this pull request Jun 19, 2020

generators should lower to Iterators.map, not Base.Generator #34368

Open

simeonschaub pushed a commit to simeonschaub/julia that referenced this pull request Jun 28, 2020

Add Iterators.map (JuliaLang#34352)

71d586d

tkf mentioned this pull request Jul 20, 2020

Create Compat.Iterators; add takewhile, dropwhile and map JuliaLang/Compat.jl#715

Merged

simeonschaub pushed a commit to simeonschaub/julia that referenced this pull request Aug 11, 2020

Add Iterators.map (JuliaLang#34352)

46358cf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Iterators.map #34352

Add Iterators.map #34352

tkf commented Jan 12, 2020

StefanKarpinski Jan 13, 2020

JeffBezanson Jan 13, 2020

StefanKarpinski Jan 13, 2020

tkf commented Jan 14, 2020

bramtayl commented Jan 14, 2020 •

edited

Loading

tkf commented Jan 14, 2020

bramtayl commented Jan 14, 2020

StefanKarpinski commented Jan 15, 2020

tkf commented Jan 15, 2020

oxinabox commented Feb 3, 2020 •

edited

Loading

tkf commented Feb 3, 2020

oxinabox commented Feb 3, 2020

bramtayl commented Feb 3, 2020

tkf commented Feb 4, 2020

stev47 left a comment

stev47 May 3, 2020

tkf May 3, 2020

stev47 May 3, 2020

tkf May 3, 2020

stev47 commented May 3, 2020 via email

stev47 commented May 3, 2020 via email

tkf commented May 3, 2020

stev47 commented May 3, 2020 via email

tkf commented May 3, 2020

tkf commented May 3, 2020

tkf commented Jun 7, 2020

andyferris commented Jun 18, 2020

andyferris commented Jun 18, 2020

tkf commented Jun 18, 2020


		!!! compat "Julia 1.5"
		This function requires at least Julia 1.5.

Add Iterators.map #34352

Add Iterators.map #34352

Conversation

tkf commented Jan 12, 2020

StefanKarpinski Jan 13, 2020

Choose a reason for hiding this comment

JeffBezanson Jan 13, 2020

Choose a reason for hiding this comment

StefanKarpinski Jan 13, 2020

Choose a reason for hiding this comment

tkf commented Jan 14, 2020

bramtayl commented Jan 14, 2020 • edited Loading

tkf commented Jan 14, 2020

bramtayl commented Jan 14, 2020

StefanKarpinski commented Jan 15, 2020

tkf commented Jan 15, 2020

oxinabox commented Feb 3, 2020 • edited Loading

tkf commented Feb 3, 2020

oxinabox commented Feb 3, 2020

bramtayl commented Feb 3, 2020

tkf commented Feb 4, 2020

stev47 left a comment

Choose a reason for hiding this comment

stev47 May 3, 2020

Choose a reason for hiding this comment

tkf May 3, 2020

Choose a reason for hiding this comment

stev47 May 3, 2020

Choose a reason for hiding this comment

tkf May 3, 2020

Choose a reason for hiding this comment

stev47 commented May 3, 2020 via email

stev47 commented May 3, 2020 via email

tkf commented May 3, 2020

stev47 commented May 3, 2020 via email

tkf commented May 3, 2020

tkf commented May 3, 2020

tkf commented Jun 7, 2020

andyferris commented Jun 18, 2020

andyferris commented Jun 18, 2020

tkf commented Jun 18, 2020

bramtayl commented Jan 14, 2020 •

edited

Loading

oxinabox commented Feb 3, 2020 •

edited

Loading