Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds support for es6 iterators #898

Closed
wants to merge 2 commits into from
Closed

Adds support for es6 iterators #898

wants to merge 2 commits into from

Conversation

megawac
Copy link
Collaborator

@megawac megawac commented Aug 31, 2015

Still not sure if this is the right take to address this. The main issues is I expect there will be a perf hit as a result of creating a new array ([value, key]) at each step iteration step, instead of just passing keys directly (benchmarks to follow).

Fixes #839

Benchmarks

Comparing v1.3.0 with current on Node v2.3.3
--------------------------------------
each(10) v1.3.0 x 36,160 ops/sec ±3.60% (24 runs sampled), 0.0277ms per run
each(10) current x 35,297 ops/sec ±1.45% (28 runs sampled), 0.0283ms per run
v1.3.0 is faster
--------------------------------------
each(300) v1.3.0 x 6,017 ops/sec ±1.55% (26 runs sampled), 0.166ms per run
each(300) current x 5,883 ops/sec ±1.06% (28 runs sampled), 0.17ms per run
v1.3.0 is faster
--------------------------------------
each(10000) v1.3.0 x 215 ops/sec ±1.33% (28 runs sampled), 4.64ms per run
each(10000) current x 207 ops/sec ±3.90% (28 runs sampled), 4.84ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(10) v1.3.0 x 19,552 ops/sec ±1.27% (27 runs sampled), 0.0511ms per run
eachSeries(10) current x 18,455 ops/sec ±4.19% (26 runs sampled), 0.0542ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(300) v1.3.0 x 1,115 ops/sec ±2.07% (27 runs sampled), 0.897ms per run
eachSeries(300) current x 1,023 ops/sec ±2.77% (25 runs sampled), 0.978ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(10000) v1.3.0 x 35.27 ops/sec ±1.47% (29 runs sampled), 28.4ms per run
eachSeries(10000) current x 31.43 ops/sec ±3.30% (27 runs sampled), 31.8ms per run
v1.3.0 is faster
--------------------------------------
eachLimit(10) v1.3.0 x 25,702 ops/sec ±4.46% (28 runs sampled), 0.0389ms per run
eachLimit(10) current x 25,670 ops/sec ±2.65% (27 runs sampled), 0.039ms per run
Tie
--------------------------------------
eachLimit(300) v1.3.0 x 2,282 ops/sec ±2.68% (28 runs sampled), 0.438ms per run
eachLimit(300) current x 2,321 ops/sec ±3.81% (25 runs sampled), 0.431ms per run
Tie
--------------------------------------
map(10) v1.3.0 x 32,804 ops/sec ±3.23% (25 runs sampled), 0.0305ms per run
map(10) current x 32,166 ops/sec ±1.56% (27 runs sampled), 0.0311ms per run
Tie
--------------------------------------
map(300) v1.3.0 x 5,134 ops/sec ±1.12% (28 runs sampled), 0.195ms per run
map(300) current x 4,830 ops/sec ±2.39% (28 runs sampled), 0.207ms per run
v1.3.0 is faster
--------------------------------------
map(10000) v1.3.0 x 172 ops/sec ±2.41% (28 runs sampled), 5.81ms per run
map(10000) current x 165 ops/sec ±2.19% (28 runs sampled), 6.07ms per run
v1.3.0 is faster
--------------------------------------
mapSeries(10) v1.3.0 x 17,971 ops/sec ±1.44% (27 runs sampled), 0.0556ms per run
mapSeries(10) current x 17,425 ops/sec ±2.19% (27 runs sampled), 0.0574ms per run
v1.3.0 is faster
--------------------------------------
mapSeries(300) v1.3.0 x 969 ops/sec ±3.29% (25 runs sampled), 1.03ms per run
mapSeries(300) current x 962 ops/sec ±2.49% (27 runs sampled), 1.04ms per run
Tie
--------------------------------------
mapSeries(10000) v1.3.0 x 30.75 ops/sec ±4.29% (26 runs sampled), 32.5ms per run
mapSeries(10000) current x 30.44 ops/sec ±5.64% (26 runs sampled), 32.8ms per run
Tie
--------------------------------------
mapLimit(10) v1.3.0 x 24,858 ops/sec ±1.12% (27 runs sampled), 0.0402ms per run
mapLimit(10) current x 25,084 ops/sec ±1.20% (28 runs sampled), 0.0399ms per run
Tie

@aearly
Copy link
Collaborator

aearly commented Aug 31, 2015

👍 The performance impact seems negligible.

@megawac
Copy link
Collaborator Author

megawac commented Aug 31, 2015

Probably should be tested in more environments before a decision is made
here
On Aug 31, 2015 7:16 PM, "Alexander Early" notifications@github.com wrote:

[image: 👍] The performance impact seems negligible.


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 5, 2015

@megawac Maybe create an object {key, value} for a smaller penalty performance?
https://jsperf.com/new-array-vs-new-object-in-iterators

  function _iterator(coll) {
        var i = -1;
        var len;
        var keys;

        if (_isArrayLike(coll)) {
            len = coll.length;
            return function next() {
                i++;
                return i < len ? {value: coll[i], key: i} : null;
            };
        }
        else if (iteratorSymbol && coll[iteratorSymbol]) {
            var iterator = coll[iteratorSymbol]();
            return function next() {
                var item = iterator.next();
                if (item.done) return null;
                return _isArray(item.value) ? {value: item.value[0], key: item.value[1]} : {value: item.value};
            };
        }
        else {
            keys = _keys(coll);
            len = keys.length;
            return function next() {
                i++;
                return i < len ? {value: coll[keys[i]], key: keys[i]} : null;
            };
        }
    }

p.s. But not {0, 1}. This object will confuse V8 JIT optimizator.

@megawac
Copy link
Collaborator Author

megawac commented Oct 5, 2015

Afaik, arrays have historically been well optimized in v8
On Oct 5, 2015 9:37 AM, "Ilya" notifications@github.com wrote:

@megawac https://github.com/megawac Maybe create an object {key, value}
for a smaller penalty performance?
https://jsperf.com/new-array-vs-new-object-in-iterators

p.s. But not {0, 1}. This object will confuse V8 JIT optimizator.


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 5, 2015

For iteration, but not for creation.

@megawac
Copy link
Collaborator Author

megawac commented Oct 5, 2015

Would you be interested in forking this branch and running the benchmarks?
It'd be nice to squeeze every bit of juice out of this
On Oct 5, 2015 10:21 AM, "Ilya" notifications@github.com wrote:

For iteration, but not for creation.


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 5, 2015

I can do it little bit later (during this week).


Why "return _isArray(item.value) ? item : [item]" is bad idea:

var mySet = new Set();
mySet.add("a");
mySet.add({});
mySet.add([1, 2]);

var  setIter = _iterator(mySet);

setIter(); // => ["a"];
setIter(); // => [{}];
setIter(); // => [1, 2], but we awaiting [[1, 2]];

@megawac
Copy link
Collaborator Author

megawac commented Oct 5, 2015

How would you suggest plucking keys from a map iterator in that case?

var m = new Map() m.set({a:1}, {b: 5}) mSymbol.iterator.next()
// Object {value: Array[2], done: false}

On Mon, Oct 5, 2015 at 11:01 AM, Ilya notifications@github.com wrote:

I can do it, but little bit later (during this week).

Why "return _isArray(item.value) ? item : [item]" is bad idea:

var mySet = new Set();
mySet.add("a");
mySet.add({});
mySet.add([1, 2]);
var setIter = _iterator(mySet);

setIter(); // => ["a"];
setIter(); // => [{}];
setIter(); // => [1, 2], but we awaiting [[1, 2]];


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 5, 2015

Best solution would be Array.from but not all environments supports this.

var m = new Map();
m.set({a:1}, {b:5});
m.set([1,2]);
_iterator(m);

....
else if (iteratorSymbol && coll[iteratorSymbol]) {
    var iterator = Array.from(coll)); // Array.from(coll) == > [ [{a:1}, {b:5}], [[1,2], undefined] ]
    len = iterator.length;
    return function next() {
       i++;
       return i < len ? [iterator[i][0], iterator[i][1]] : null;
     };
}
....

@megawac
Copy link
Collaborator Author

megawac commented Oct 5, 2015

Wouldn't that run into the same issue with sets of arrays and arrays of
arrays?

On Mon, Oct 5, 2015 at 12:25 PM, Ilya notifications@github.com wrote:

Best solution would be Array.from but not all environments supported this.

var m = new Map();
m.set({a:1}, {b:5});
m.set([1,2]);
....else if (iteratorSymbol && coll[iteratorSymbol]) {
var iterator = Array.from(coll)); // Array.from(coll) == > [ [{a:1}, {b:5}], [[1,2], undefined] ]
len = iterator.length;
return function next() {
i++;
return i < len ? [iterator[i][0], iterator[i][1]] : null;
};
}


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 5, 2015

Other cases of iterators return only value without key? Check for instanceof Map before calls Array.from? :)

@megawac
Copy link
Collaborator Author

megawac commented Oct 5, 2015

Probably makes most sense to ignore EntryIterator types (such as the map
iterator) as typically the user is required to manually destructure the
arguments for these case (i.e. for (let [x, y] of map){}). Therefore we
can suggest our users use similar strategies when passing entry iterators
(async.fn(iterator, function([val, key], next) {}))

Thoughts @aearly @ex1st

On Mon, Oct 5, 2015 at 12:59 PM, Ilya notifications@github.com wrote:

Other case of iterators return only value without key? Check instanceof
Map before calls Array.from? :)


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 6, 2015

You mean a users must manually convert such objects to array/collection?

@aearly
Copy link
Collaborator

aearly commented Oct 6, 2015

Is there a simple way to detect the difference between the Map style iterator ([key, value]) and the Array style iterator ([value])?

@megawac
Copy link
Collaborator Author

megawac commented Oct 6, 2015

You mean a users must manually convert such objects to array/collection?

Leave value, key iterators out of scope for the implementation and suggest how to handle it in documentation (i.e. async.forEach( ([value, key], next) => {}))

Is there a simple way to detect the difference between the Map style iterator ([key, value]) and the Array style iterator ([value])?

I don't believe so

@ex1st
Copy link

ex1st commented Oct 6, 2015

@megawac, sounds good. But still need some keys for forEachOf. Do you agree?

else if (iteratorSymbol && coll[iteratorSymbol]) {
    var iterator = coll[iteratorSymbol]();
    return function next() {
      i++;
      var item = iterator.next();
      if (item.done) return null;
      return [item.value, i];
     };
}
async.forEachOf( ([value, key], index, next) => {}))

@aearly, it's unreal. User can create custom iterator for his collections.

@megawac
Copy link
Collaborator Author

megawac commented Oct 6, 2015

I would suggest always giving undefined keys for async.*Of functions when
iterating an iterators/generators

On Tue, Oct 6, 2015 at 3:09 PM, Ilya notifications@github.com wrote:

@megawac https://github.com/megawac, sounds good. But still need some
keys for forEachOf. Do you agree?

else if (iteratorSymbol && coll[iteratorSymbol]) {
var iterator = colliteratorSymbol;
return function next() {
i++;
var item = iterator.next();
if (item.done) return null;
return [item.value, i];
};
}

async.forEachOf( ([value, key], index, next) => {}))

@aearly https://github.com/aearly, it's unreal.


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 6, 2015

Ok, i agree :)

@ex1st
Copy link

ex1st commented Oct 7, 2015

@megawac, async.map will return a empty array if keys will be 'undefined'

@megawac
Copy link
Collaborator Author

megawac commented Oct 7, 2015

Not if its implemented via _iterator
On Oct 7, 2015 3:51 PM, "Ilya" notifications@github.com wrote:

@megawac https://github.com/megawac, async.map will return a empty
array if keys will be 'undefined'


Reply to this email directly or view it on GitHub
#898 (comment).

@ex1st
Copy link

ex1st commented Oct 7, 2015

You suggest pass 'undefined' only in async.*Of, but _iterator must return keys always?

@aearly aearly added the feature label Oct 26, 2015
@aearly
Copy link
Collaborator

aearly commented Oct 26, 2015

One thing I discovered: creating simple Objects is slightly faster than creating Arrays: http://jsperf.com/object-vs-array-vs-class . Most JS engines convert small, non-dynamic Objects to structs internally. (I remember discussing this with @megawac a while back).

This might make the cost of converting an interator that returns just values, to an iterator that returns {key, value} pairs.

@megawac
Copy link
Collaborator Author

megawac commented Oct 26, 2015

/cc @jdalton do you have any insight on ^

On Sun, Oct 25, 2015 at 9:05 PM, Alexander Early notifications@github.com
wrote:

One thing I discovered: creating simple Objects is slightly faster than
creating Arrays: http://jsperf.com/object-vs-array-vs-class . Most JS
engines convert small, non-dynamic Objects to structs internally. (I
remember discussing this with @megawac https://github.com/megawac a
while back).

This might make the cost of converting an interator that returns just
values, to an iterator that returns {key, value} pairs.


Reply to this email directly or view it on GitHub
#898 (comment).

@jdalton
Copy link
Contributor

jdalton commented Oct 26, 2015

From the benchmarks it looks like perf is ok. As for the arrays or objects I think arrays map better to a map which can be initialized as an array of key-value pairs. I'm not sure what you're wanting to provide though as I see if you're iterating an array you return [value, index] and if you're iterating an iterator you return [value].

@aearly
Copy link
Collaborator

aearly commented Mar 9, 2016

Now that we're using lodash internally, would we get iterator support for free if lodash supported them?

@jdalton
Copy link
Contributor

jdalton commented Mar 9, 2016

Now that we're using lodash internally, would we get iterator support for free if lodash supported them?

Lodash has limited iterator support. It currently only supports them in _.toArray and the chaining/sequence wrapper is itself and iterator and iterable.

@aearly
Copy link
Collaborator

aearly commented Mar 21, 2016

Now that we've majorly refactored Async, this should be revisited in another PR.

@aearly aearly closed this Mar 21, 2016
@megawac megawac deleted the es6-iterators branch April 12, 2016 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants