ipfs.files.get() hangs on certain directories

[witten]

• Version: 0.27.5 and 0.27.4
• Platform: Linux flux 4.2.0-23-generic #28~14.04.1-Ubuntu SMP Thu Dec 31 13:40:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux (Firefox 57.0.1)
• Subsystem: unknown

Type: Bug

Severity: High

Description:

When trying to call ipfs.files.get() with the multihash of certain directories stored on IPFS, the call responds as expected with an array of files: one for the directory itself (with undefined content), and one for each file such as an index.html (with actual content). An example of such a working multihash is: /ipfs/QmTeW79w7QQ6Npa3b1d5tANreCDxF2iDaAPsDvW6KtLmfB/docs/examples/

However, on other multihashes, calling ipfs.files.get() never responds and never errors. For instance, this behavior occurs with the following multihash: /ipfs/QmTeW79w7QQ6Npa3b1d5tANreCDxF2iDaAPsDvW6KtLmfB/docs/ .. Note that this is a valid path with actual content as per the ipfs.io gateway: https://gateway.ipfs.io/ipfs/QmTeW79w7QQ6Npa3b1d5tANreCDxF2iDaAPsDvW6KtLmfB/docs/

Expected behavior: Instead of never responding, I would expect ipfs.files.get() to return an array of files contained in the directory, or at least respond with an error callback.

Steps to reproduce the error:

Save this example repro index.html file, and then open it in your browser: https://gist.githubusercontent.com/witten/bfb32cf627f1fb13acdb9ff70e05f410/raw/9e1adb1a161efaa290df74b205447548b6e5ab8f/index.html

Open your browser console and observe the log trying to get a file, but then nothing else: No errors, no array of files logged, etc.

Then, edit the file and change the multihash passed to node.files.get() from .../docs/ to .../docs/examples/. Reload the page, and note that by making that multihash change, this other directory loads just fine as evidenced by console logging.

[mitra42]

This seems like a different manifestation of the bug reported in issue #1156 i.e. the call never returns, or throws an error just waits forever.

[witten]

Could very well be. One difference from #1156 is that this is not a new file. Another potential difference is that loading the file from the gateway in another browser tab does not cause the get() call to return. Reloading the get() call also doesn't work.

[witten]

For what it's worth, I've also seen this hanging behavior occur on IPFS files and not just directories. For instance, if I try a get() on the image file mentioned in #1156, the call hangs: /ipfs/QmZ5TBouUQrZrfUFxARh2RNUPEPgoRRk4QMesvkTaJSXNL

[autonome]

@witten are you still seeing this happen?

[autonome]

(on contemporary js-ipfs?)

[witten]

I no longer have the reproduction code (link above 404s.. thanks GitHub).

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterward (see #4336).

This issue is most likely resolved in Helia, please try it out!

I would highly recommend checking out https://pl-diagnose.on.fleek.co/, or the older https://check.ipfs.network/, in order to investigate whether your content is accessible.

For configuring timeouts on almost all Helia/Libp2p methods, you can use AbortSignals.

[mitra42]

AbortSignals were new to me, but timeouts as an approach to this problem with .get were rejected for our app a long time ago because the app has no idea what is going on inside the .get call, it doesn't know if things (sub-steps within the get) are progressing or have in effect failed (never likely to succeed) so it doesn't know how long to wait before falling back to HTTP or some other backup method. I have, over the 6 years since this was reported, spoken to several other devs who also gave up on IPFS at least in part because of this specific problem.

Think about it - how many milliseconds should you wait for the response to a .get since you don't even know the size of the file being retrieved?

I think this is even true for whatever helia uses to replace ipfs.catReadableStream (and I'm just presuming you have the equivalent of catReadableStream since I can't find it in the API docs). How long should an app wait before presuming an attempt to open a stream to an IPFS resource has failed? Only the internal code inside catReadableStream knows when some sub-step (such as looking the resource up in the DHT) has failed.

I also note that torrent has a very neat solution to this, which is to support http backup within the torrent file, which has the side effect of reseeding a file that has a known http address back into the decentralized network. The http fetch starts in parallel to the decentralize get.