Iterator.drop cannot handle (very) large iterators

[charpov]

I sometimes use very large iterators (more than Int.MaxValue elements) to check that students don't expend in memory when they shouldn't. However, some iterator methods, like drop, don't work correctly on those:

class TenBillionIterator extends AbstractIterator[Long]:
   private var n        = 0L
   def hasNext: Boolean = n < 10_000_000_000L
   def next: Long       = { val r = n; n += 1L; r }

val i: Iterator[Long] =
   TenBillionIterator()
      .drop(1_000_000_000)
      .drop(1_000_000_000)
      .drop(1_000_000_000)

i.next() // 2147483647, should be 3000000000

Interestingly, ++ (concat) does work:

val j = TenBillionIterator() ++ TenBillionIterator()
var n = 13_000_000_000L
while n > 0 do
   n -= 1L
   j.next()

j.next() // 3000000000, correctly

I understand it's a somewhat corner case, but the documentation is silent about such a limitation and the behavior caught me off-guard (especially because ++ worked and things only fell apart when I started using drop). I don't know if this is a documentation issue or if some code needs fixing (maybe just a matter of making dropping a Long in class SliceIterator).

(Note that Iterator.range(0L, 10_000_000_000L) also doesn't work, but it's a different issue.)

I'm using Scala 3.5.2, but this is a Scala 2 collection problem.

[som-snytt]

The linked solution just evaluates the underlying iterator, rather than cap the drop count.

An iterator wanting different behavior would implement appropriate overrides (such as sliceIterator).

The current workaround would be to call hasNext, which has the same effect.

[charpov]

Would this work?

val sum = dropping + lo
if (sum < 0) {
  dropping = Int.MaxValue
  skip()
  sum + Int.MaxValue
}
else sum

I find it easier to read, but I don't trust myself with 2s-complement arithmetic...

[som-snytt]

Simpler is

if (sum < 0) {
  skip()
  lo
}

which has the advantage of forcing less, while still being an easy read.

To force least, then skip with dropping at lo - bump, and end with dropping at MaxValue.

[charpov]

I like the simplest code, even if it forces more than necessary (and I meant to have this discussion in the PL, not here).

[som-snytt]

Since it doesn't matter analytically (and if I can convince myself; there is a case for minimal evaluation), I will take your advice and update the PR. It's good to practice simplicity: use it or lose it.

[SethTisue]

@scala/collections