Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory leak during ipfs add without daemon running #2846

Closed
notslang opened this issue Jun 13, 2016 · 1 comment
Closed

memory leak during ipfs add without daemon running #2846

notslang opened this issue Jun 13, 2016 · 1 comment

Comments

@notslang
Copy link
Contributor

I'm trying to hash the contents of https://github.com/cdnjs/cdnjs so I can detect and rewrite all in-browser requests for cdnjs.cloudflare.com to go to IPFS and be loaded from my local cache or a nearby computer. It's about 29GB of highly duplicated content and contains 2.5 million files. The command I'm using is ipfs add -r -H cdnjs/ajax/libs/ and the ipfs version is 0.4.2.

I tried running this with the IPFS daemon, but it failed with 10:29:48.139 ERROR commands/h: unexpected EOF client.go:247 after running all night. This morning, I killed the daemon and ran the same command again. It seems to be more reliable and faster without the daemon, but after a few hours it used up all the RAM on my system, slowed down, and I had to kill it:

1465808486

As a workaround, I suppose I could hash it in chunks and then add them together with mfs, but 29GB doesn't seem like an unreasonable amount of data to add at once.

@Kubuxu
Copy link
Member

Kubuxu commented Jun 13, 2016

go-ipfs 0.4.2 has pretty severe leak, it was fixed on current master but there is another one, we are tracking it in #2823 , also looks like you are duplicating the effort as @magik6k took over my job on that and is also adding cdnjs: ipfs-inactive/archives#35

I will close it now as the issues is tracked elsewhere already.

@Kubuxu Kubuxu closed this as completed Jun 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants