lots of goroutines hanging in handleOutgoingMessage

[whyrusleeping]

I have a node (A) that has added a file and another (B) that is catting it. Node A's RAM usage spikes massively until it runs out of memory, sending a panic before the out of memory happens shows a massive number of goroutines waiting to send their messages in handleOutgoingMessage. Not entirely sure why this is happening yet, but its probably a bug.

[btc]

[whyrusleeping]

i like all of those points. the errChan is a likely suspect In my opinion.

[whyrusleeping]

Although, upon review, i dont see that happening anywhere.

[btc]

i dont see that happening anywhere

don't see what? 😕

[whyrusleeping]

The errChan being sent on, Im not seeing that case happen in my tests

[btc]

Ah I see

https://gist.github.com/maybebtc/c6034877dadbee2b97de

3255 instances of handleOutgoingMessage and 3000 instances of sync.(*Mutex).Lock(...). The lock is held to Get from LevelDB.

This doesn't begin to explain why the muxer's outgoing pipe is blocked up so severely.

[whyrusleeping]

I think one thing we might be able to do to help is not make handleOutgoingMessage its own goroutine. Or at least limit the number allowed in flight at a time.

[btc]

one thing we might be able to do to help is not make handleOutgoingMessage its own goroutine

agreed. 👍 to rate-limiting producers. 904e9d5

[jbenet]

2. yeah should continue. we should probably get rid of max message size though. or make it a very large limit (100MB?) just to catch runoff problems. What's the realistic largest message we'd send through here? a large block?

My guess would be 5).

[whyrusleeping]

Fixed in 1026244