gateway: URI-escaping of paths

[ugjka]

For example, if file url's contain @ or # they don't work

https://ipfs.io/ipfs/QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs/@#$_%5e_%20chillmix.mp3

will return
Path Resolve error: no link named "@" under QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs

Or

https://ipfs.io/ipfs/QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs/Cat%20D###%20Goo.mp3

will return
Path Resolve error: no link named "Cat D" under QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs

Directory
https://ipfs.io/ipfs/QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs/

Tested on chrome and firefox

[ghost]

You'll have to escape # as %23 because it has a meaning in URLs. The actual request ends up as https://ipfs.io/ipfs/QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs/@, so everything looks as expected here.

[ghost]

Oh I think I understand, the directory listing should be escaping these, but doesn't. Is that what you mean?

[ugjka]

Yes the directory listing in the browser doesn't escape them!

[ghost]

Yep that's a bug, thank you

[ghost]

Feel free to give it a try, the respective code is in core/corehttp/gateway_handler.go -- otherwise we'll get to it soon

[ugjka]

Interestingly this link with @ _ and ~ in it will work without escaping for some reason
https://ipfs.io/ipfs/QmZbfX4p7LUtuWgLtp1XYt6tkPnwfhqBTzqPDCanucYQrs/ugis@cherry_~.mp3

Something funky going on

[ghost]

No that looks totally fine to me -- the hash (#) within a path has the meaning of "fragment identifier", while @ doesn't have a meaning within a path.

https://en.wikipedia.org/wiki/Fragment_identifier

[ugjka]

Ah, that makes sense!

[ugjka]

Ha, found another way how to confuse the web gateway
Create files with file-names that mimics character escapes

added QmZrX1xUiB9SB5eoMYSCDZ9CEH6Zvi4RBjBGSht5wAPuLP test/%23%20%23
added QmPrn13UrUuNVwWLtrYA4FEmN7Tbf6WkL9eQaAF2diHJxT test/test%20test
added QmYDE4uYhsRpT1L7Q8zJbRVGfuDjbUMN5TeED8jhw739ki test

results:
Path Resolve error: no link named "# #" under QmYDE4uYhsRpT1L7Q8zJbRVGfuDjbUMN5TeED8jhw739ki

Path Resolve error: no link named "test test" under QmYDE4uYhsRpT1L7Q8zJbRVGfuDjbUMN5TeED8jhw739ki

[jbenet]

we'll want to distill the solutions to these problems into a small doc that states clearly when to escape, and why.

[ghost]

Ha, found another way how to confuse the web gateway

Yeah I suppose the lack of escaping works both ways :)

we'll want to distill the solutions to these problems into a small doc that states clearly when to escape, and why

Agreed 👍 also, when not to escape

[ugjka]

https://en.wikipedia.org/wiki/Percent-encoding

I have found so far 3 characters that break things and should be escaped

# --> %23
? --> %3F
% --> %25

[adrian-bl]

I created PR #2092 which should fix the issue.
It does by adding a custom template function, as (AFAIK) there is no way to change the auto-escaping of html/template to not ignore some chars.

[ghost]

has been addressed long ago in ipfs/dir-index-html#9