Table iteration leaves hanging goroutine causes out of memory

[jeff-emanuel]

I'm iterating over a large (userdata) data structure. For each outer iteration, I iterate over a small table. Eventually I get a Go out-of-memory panic and the stack dump includes (at least) tens of thousands of goroutines all with this trace:

goroutine 23869 [select]:
github.com/milochristiansen/lua.newTableIter.func1(0xc04225c120, 0xc29e378ea0, 0xc29e378f00)
c:/dev/go/src/github.com/milochristiansen/lua/table.go:443 +0x3eb
created by github.com/milochristiansen/lua.newTableIter
c:/dev/go/src/github.com/milochristiansen/lua/table.go:428 +0xdf

When I run the attached code with an argument of 1 million, the process grows to 450 Mb before it ends. An argument of 4 million results in a process size of 1.6 Gb, and with 16 million it grows to 6.8 Gb.
There is clearly linear memory growth with the number of iterations. My real case has about 4 million outer iterations, but that process already has a large memory footprint and can't spare another 1.6 Gb.

luatableiteration.zip

[milochristiansen]

Those goroutines will close when the iterator object is collected. Something must be holding the iterator objects in memory.

I'll see if I can track the cause down.

[milochristiansen]

After a bit of experimenting and going through the relevant code I can't find any way the iterators could still be reachable after their stack entry is overwritten. Since the stack is not growing, and manually calling the GC still does not result in the finalizer being called, I am obviously missing something. When I first put that iterator into place I tested to make sure the iterator goroutine was not preventing finalization, so I am pretty sure that isn't it... I need to re-test it anyway.

Until I find the cause there is one possible work-around: Use next. This is an alternate iterator that acts like the one in the Lua spec, but is not reentrant for a single table (trying to iterate over a table again will reset the iterator for that table).

[jeff-emanuel]

Thanks, I'll try next

[milochristiansen]

Actually, don't bother. I just fixed it. I was adding a function to manually kill iterators and changed the way that a kill is signaled. Suddenly it started working... It was getting hung up on something related to the kill channel somehow. Still not sure why it wasn't working despite it having been tested and working before...

I'll push the fix shortly.

[jeff-emanuel]

I get a lot of printout: it works now...

…

On Fri, Oct 26, 2018 at 2:29 PM Milo Christiansen ***@***.***> wrote: Closed #17 <#17> via 1d8bec7 <1d8bec7> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AhhKJ0DBqeenAhBqBsUzxP6esIEmMhUHks5uo3CagaJpZM4X8mhI> .

[milochristiansen]

Yeah, I just pushed a fix for the debugging text I left in like an idiot :( Update, and it should go away.

To be clear, each time that prints another iterator is dying and being collected :)

[jeff-emanuel]

Thanks again!