memory fail when adding >200GB

[jbenet]

I'm adding all of downloads.openwrt.org to ipfs. and it fails at around ~15/16GB. I'm doing it with the daemon off (for better perf). I think it's memory consumption-- maybe leaking stuff (or rather, holding on to it while adding and not letting gc do its work.)

[jbenet]

yeah, it's memory. it climbs steadily, still. @whyrusleeping best way to make a profile?

[whyrusleeping]

run with IPFS_PROF set to true.

[whyrusleeping]

I can take a look when i'm done with all this moving stuff.

[whyrusleeping]

tried debugging this last night. added a 300GB directory of smaller ~4MB files. memory did slowly climb, but veeeeery slowly, ~600MB over an hour and a half. The main issue i noticed was how absurdly slow it was, i'm going to try again later after temporarily disabling sync on my datastore

[jbenet]

@whyrusleeping did you try this o/ ?

[whyrusleeping]

I beleive that I did, and am running into the open file descriptor bug now.

[ghost]

This is still an issue, but closing in favor of #1924