Support large file uploads to IPFS

[iurimatias]

Outline

Currently attempting to upload a large file to IPFS through EmbarkJS api (see uploading file in documentation) will get the browser stuck. The goal of this task is to fix the code so it can support large file uploads to decentralized systems, namely IPFS.

Acceptance Criteria

• Uploading a file larger than 1GB using EmbarkJS.Storage.uploadFile as specified in the documentation should not get the browser stuck and the file should be able to be downloaded afterwards.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 180.0 DAI (180.0 USD @ $1.0/DAI) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $26,787.14 more funded OSS Work available on the Gitcoin Issue Explorer

[kziemianek]

Hello embark! this is my first contribution to this repository.
I will try to reproduce the problem, then i'll apply suggested solution.

[kziemianek]

Seems like we need to wait for ipfs-inactive/js-ipfs-http-client#851
The problem is sumarized here : https://github.com/ipfs/js-ipfs-api/issues/842

[vs77bb]

@iurimatias Do you agree that we have a blocker here? Thanks for digging around, @kziemianek!

[gitcoinbot]

@kziemianek Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[iurimatias]

@kziemianek @vs77bb IMO fixing it it in js-ipfs-api if necessary can be considered part of the bounty.

[kziemianek]

@iurimatias it's getting solved in linked PR. What can be done here is ipfs-api version bump and verification after release of the feature in ipfs-api. If it's ok for You I can keep an eye on it.

[kziemianek]

@iurimatias seems like ipfs-api version bump will be required to get this working, embark is currently at 17.x.x while ipfs-api most recent is 24.x. Would it be a problem ?

I tried it locally but demo is not working after migration to v24.0.1 so it might not be trivial.

Ever considered updating ipfs-api version to most recent ?

[gitcoinbot]

@kziemianek Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[vs77bb]

@kziemianek Are you waiting on feedback from @iurimatias here to continue? Snoozing Gitcoin Bot for 5 days 👍

[kziemianek]

@vs77bb yes .

[gitcoinbot]

@kziemianek Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[kziemianek]

I'm waiting for fix in ipfs-api, please snooze for 100 days because it may take long time.

[lidel]

Hey @kziemianek you may want to try ipfs-api v25.0.0, it shipped with a fix (ipfs-inactive/js-ipfs-http-client#868) that should make 1GB+ uploads possible.

[valerysntx]

Hey @kziemianek you may want to try ipfs-api v25.0.0, it shipped with a fix (ipfs/js-ipfs-api#868) that should make 1GB+ uploads possible.

So, this should be ready to play the example with that v25.0.0 fix?
https://github.com/ipfs/js-ipfs-api/tree/master/examples/upload-file-via-browser

[lidel]

Thank you for bringing this to my attention!
The example was really old, I created PR to update it: ipfs-inactive/js-ipfs-http-client#884

In short, when working with bigger files you don't want to create a Buffer from file contents (like here),
instead you create a stream and pass it to ipfs-api (see example from my PR).

[gitcoinbot]

@karankohli13 Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@karankohli13 Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[frankchen07]

@kziemianek - Frank from Gitcoin here - still waiting on feedback?

[Neurone]

Suggestion: while waiting for streaming through HTTP is working as expected, why don't just use ipfs.addFromFs instead? In the most cases (I suggest to users in ALL cases ^_^) IPFS node is running on the client machine, there's no need to disturb the browser to add file to it. Also, ipfs.addFromFs works for all browsers, not only for those supporting HTTP streaming.

[gitcoinbot]

Issue Status: 1. Open 2. Cancelled

---

The funding of 180.0 DAI (180.0 USD @ $1.0/DAI) attached to this issue has been cancelled by the bounty submitter

• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $44,469.85 more funded OSS Work available on the Gitcoin Issue Explorer

[0x-r4bbit]

@iurimatias does it make sense to close this one (and revisit later) due to inactivity?