-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support http range header #10
feat: support http range header #10
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self review, some comments
On mobile but wanted to give a shout that ive got a refactor going that cleans this up a bunch |
Done a first pass, will look again after the refactor lands. |
…port FYI: some tests were failing after `npm run reset` and `npm run test` was run. I had to fix them manually.
FYI: some tests were failing when merging main into byte-range PR due to new ipns things. changes at |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, pending remaining discussion points
Co-authored-by: Alex Potsides <alex@achingbrain.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Going over code one last time
* - Content-Range: <unit> <byteStart>-<byteEnd>/<byteSize> | ||
* - Content-Range: <unit> <byteStart>-<byteEnd>/‍* | ||
*/ | ||
// - Content-Range: <unit> */<byteSize> // this is purposefully not in jsdoc block |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@achingbrain is this better than all as single line comments?
} catch (err) { | ||
this.log.error('error parsing resource %s', resource, err) | ||
|
||
return badRequestResponse('Invalid resource') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Less throwing, more bad fetch responses
commenting elsewhere since the thread (#10 (comment)) was collapsed: we could solve both concerns by setting internalFileSize so we know which value we set vs consumers.. but that could get confusing. i'll leave as is for now. i've got _isRangeRequest updated locally, and have isValidRangeRequest updated as well. So now we have: inaccessable properties outside the ByteRangeContext class:
|
## @helia/verified-fetch [1.2.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.1.3...@helia/verified-fetch-1.2.0) (2024-03-15) ### Features * support http range header ([#10](#10)) ([9f5078a](9f5078a)) ### Trivial Changes * fix build ([#22](#22)) ([01261fe](01261fe))
🎉 This PR is included in version 1.2.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.7.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
* feat: implement new ipns record&answer properties * fix: parseUrlString response includes defined ttl, set ttl if ipnsCached * test: fix firefox failure * feat: support http range header (#10) * chore: limit body parameters to the types used * chore: add response-header helper and tests * feat: add range header parsing support * feat: verified-fetch supports range-requests * test: fix dns test asserting test failure since we are catching it now * fix: return 500 error when streaming unixfs content throws * fix: cleanup code and unexecuting tests hiding errors * chore: some cleanup and code coverage * tmp: most things working * fix: stream slicing and test correctness * chore: fixed some ByteRangeContext tests * test: add back header helpers * fix: unixfs tests are passing * fix: range-requests on raw content * feat: tests are passing moved transform stream over to https://github.com/SgtPooki/streams * chore: log string casing * chore: use 502 response instead of 500 * chore: use libp2p/interface for types in src * chore: failing to create range resp logs error * chore: Apply suggestions from code review * chore: fix broken tests from github PR patches (my own) * chore: re-enable stream tests for ByteRangeContext * chore: clean up getBody a bit * chore: ByteRangeContext getBody cleanup * chore: apply suggestions from code review Co-authored-by: Alex Potsides <alex@achingbrain.net> * fix: getSlicedBody uses correct types * chore: remove extra stat call * chore: fix jsdoc with '*/' * chore: fileSize is public property, but should not be used * test: fix blob comparisons that broke or were never worjing properly * chore: Update byte-range-context.ts Co-authored-by: Alex Potsides <alex@achingbrain.net> * chore: jsdoc cleanup * Revert "chore: fileSize is public property, but should not be used" This reverts commit 46dc133. * chore: jsdoc comments explaining .fileSize use * chore: isRangeRequest is public * chore: getters/setters update * chore: remove unnecessary _contentRangeHeaderValue * chore: ByteRangeContext uses setFileSize and getFileSize * chore: remove .stat changes that are no longer needed --------- Co-authored-by: Alex Potsides <alex@achingbrain.net> * chore(release): 1.2.0 [skip ci] ## @helia/verified-fetch [1.2.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.1.3...@helia/verified-fetch-1.2.0) (2024-03-15) ### Features * support http range header ([#10](#10)) ([9f5078a](9f5078a)) ### Trivial Changes * fix build ([#22](#22)) ([01261fe](01261fe)) * chore(release): 1.7.0 [skip ci] ## @helia/verified-fetch-interop [1.7.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-interop-1.6.0...@helia/verified-fetch-interop-1.7.0) (2024-03-15) ### Dependencies * **@helia/verified-fetch:** upgraded to 1.2.0 * chore: apply pr comments * fix: some ipns ttl precision cleanup --------- Co-authored-by: Alex Potsides <alex@achingbrain.net> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
* fix: set cache-control header conditionally Fixes #17 * chore: bump deps * feat: implement new ipns record&answer properties (#23) * feat: implement new ipns record&answer properties * fix: parseUrlString response includes defined ttl, set ttl if ipnsCached * test: fix firefox failure * feat: support http range header (#10) * chore: limit body parameters to the types used * chore: add response-header helper and tests * feat: add range header parsing support * feat: verified-fetch supports range-requests * test: fix dns test asserting test failure since we are catching it now * fix: return 500 error when streaming unixfs content throws * fix: cleanup code and unexecuting tests hiding errors * chore: some cleanup and code coverage * tmp: most things working * fix: stream slicing and test correctness * chore: fixed some ByteRangeContext tests * test: add back header helpers * fix: unixfs tests are passing * fix: range-requests on raw content * feat: tests are passing moved transform stream over to https://github.com/SgtPooki/streams * chore: log string casing * chore: use 502 response instead of 500 * chore: use libp2p/interface for types in src * chore: failing to create range resp logs error * chore: Apply suggestions from code review * chore: fix broken tests from github PR patches (my own) * chore: re-enable stream tests for ByteRangeContext * chore: clean up getBody a bit * chore: ByteRangeContext getBody cleanup * chore: apply suggestions from code review Co-authored-by: Alex Potsides <alex@achingbrain.net> * fix: getSlicedBody uses correct types * chore: remove extra stat call * chore: fix jsdoc with '*/' * chore: fileSize is public property, but should not be used * test: fix blob comparisons that broke or were never worjing properly * chore: Update byte-range-context.ts Co-authored-by: Alex Potsides <alex@achingbrain.net> * chore: jsdoc cleanup * Revert "chore: fileSize is public property, but should not be used" This reverts commit 46dc133. * chore: jsdoc comments explaining .fileSize use * chore: isRangeRequest is public * chore: getters/setters update * chore: remove unnecessary _contentRangeHeaderValue * chore: ByteRangeContext uses setFileSize and getFileSize * chore: remove .stat changes that are no longer needed --------- Co-authored-by: Alex Potsides <alex@achingbrain.net> * chore(release): 1.2.0 [skip ci] ## @helia/verified-fetch [1.2.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.1.3...@helia/verified-fetch-1.2.0) (2024-03-15) ### Features * support http range header ([#10](#10)) ([9f5078a](9f5078a)) ### Trivial Changes * fix build ([#22](#22)) ([01261fe](01261fe)) * chore(release): 1.7.0 [skip ci] ## @helia/verified-fetch-interop [1.7.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-interop-1.6.0...@helia/verified-fetch-interop-1.7.0) (2024-03-15) ### Dependencies * **@helia/verified-fetch:** upgraded to 1.2.0 * chore: apply pr comments * fix: some ipns ttl precision cleanup --------- Co-authored-by: Alex Potsides <alex@achingbrain.net> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> * chore: add matchUrlGroups typeguard * chore: remove unnecessary headerValue != null check * test: remove unnecessary redefinition of verifiedFetch --------- Co-authored-by: Daniel N <2color@users.noreply.github.com> Co-authored-by: Russell Dempsey <1173416+SgtPooki@users.noreply.github.com> Co-authored-by: Alex Potsides <alex@achingbrain.net> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
## @helia/verified-fetch [1.3.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.2.1...@helia/verified-fetch-1.3.0) (2024-03-21) ### Features * abort signals are respected ([#26](#26)) ([30148fe](30148fe)) ### Bug Fixes * set cache-control header correctly ([#19](#19)) ([114f3a4](114f3a4)), closes [#17](#17) [#23](#23) [#10](#10)
🎉 This PR is included in version 1.0.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
## 1.0.0 (2024-10-14) ### ⚠ BREAKING CHANGES * upgrade to helia v5 (#107) ### Features * abort signals are respected ([#26](#26)) ([30148fe](30148fe)) * allow use as a singleton ([#413](https://github.com/ipfs/helia-verified-fetch/issues/413)) ([7cbeed0](7cbeed0)) * create @helia/verified-fetch ([#392](https://github.com/ipfs/helia-verified-fetch/issues/392)) ([06d36fe](06d36fe)) * customize ipns dnsResolvers ([#445](https://github.com/ipfs/helia-verified-fetch/issues/445)) ([8f60822](8f60822)) * download tars from @helia/verified-fetch ([#442](https://github.com/ipfs/helia-verified-fetch/issues/442)) ([70ddd00](70ddd00)) * require content-type parser to set content-type ([#423](https://github.com/ipfs/helia-verified-fetch/issues/423)) ([b39d07c](b39d07c)) * support `Accept` header in `@helia/verified-fetch` ([#438](https://github.com/ipfs/helia-verified-fetch/issues/438)) ([54c4383](54c4383)) * support downloading car files from @helia/verified-fetch ([#441](https://github.com/ipfs/helia-verified-fetch/issues/441)) ([703980c](703980c)) * support http range header ([#10](#10)) ([9f5078a](9f5078a)) * support IPFS/IPNS paths, Gateways, etc ([#4](#4)) ([e7f1816](e7f1816)) * support redirects for UnixFS directories ([#5](#5)) ([4601d46](4601d46)) * support requesting raw IPNS records in @helia/verified-fetch ([#443](https://github.com/ipfs/helia-verified-fetch/issues/443)) ([e92086a](e92086a)) * upgrade to helia v5 ([#107](#107)) ([91a6473](91a6473)) * use blockstore sessions ([#50](#50)) ([541dd64](541dd64)) * use the waterworks trustless gateway by default ([#94](#94)) ([2db8d1e](2db8d1e)) ### Bug Fixes * @helia/verified-fetch init args are optional ([#412](https://github.com/ipfs/helia-verified-fetch/issues/412)) ([e6d41a0](e6d41a0)) * aborted signals throw libp2p AbortError ([#30](#30)) ([4575791](4575791)) * add interop tests and update project config ([fdc83b8](fdc83b8)) * allow contentTypeParser with Helia instance ([#427](https://github.com/ipfs/helia-verified-fetch/issues/427)) ([b5c95e6](b5c95e6)) * append query path to path resolved from IPNS name ([#3](#3)) ([fd86e6a](fd86e6a)) * byte range request end should never equal file size ([#24](#24)) ([aafc567](aafc567)) * content-type response header hints how to process response ([#426](https://github.com/ipfs/helia-verified-fetch/issues/426)) ([8b78e79](8b78e79)) * decodeURIComponent on path parts ([#40](#40)) ([f628cf6](f628cf6)) * do not coerce `undefined` to `null` for JSON serialization ([#2](#2)) ([d36ce29](d36ce29)) * gateway conformance improvements ([#85](#85)) ([7281078](7281078)) * gateway conformance tests ([#81](#81)) ([d0a3b6b](d0a3b6b)) * identity CIDs use contentTypeParser ([#49](#49)) ([3014498](3014498)) * implicit accept header can be overridden by format query ([#36](#36)) ([75c0b75](75c0b75)) * missing paths returns 404 instead of 502 ([#59](#59)) ([291a054](291a054)), closes [#53](#53) * pass url and body to badRequestResponse ([#44](#44)) ([cc228e6](cc228e6)) * prevent interop timeouts with fast fixture loading ([#73](#73)) ([a43d994](a43d994)) * reduce dagPb and dagCbor handler complexity ([#45](#45)) ([3b41752](3b41752)) * remove redundant abort controller ([#41](#41)) ([04b220d](04b220d)) * remove stubs from verified-fetch tests ([#431](https://github.com/ipfs/helia-verified-fetch/issues/431)) ([cbab8b0](cbab8b0)) * set cache-control header correctly ([#19](#19)) ([114f3a4](114f3a4)), closes [#17](#17) [#23](#23) [#10](#10) * support https?://<dnsLink>.ipns.<gateway> urls ([#16](#16)) ([0ece19a](0ece19a)) * ttl and caching for ipns urls ([#34](#34)) ([44ac5a1](44ac5a1)) * unixfs dir redirect ([#33](#33)) ([32ca87f](32ca87f)) * update @helia/ipns and dns config ([#18](#18)) ([9f88c54](9f88c54)) * use CID object for verified-fetch progress events ([#425](https://github.com/ipfs/helia-verified-fetch/issues/425)) ([b010184](b010184)) * use ipfs-unixfs-exporter directly ([#42](#42)) ([4532bf1](4532bf1)) * verified-fetch etag header ([#434](https://github.com/ipfs/helia-verified-fetch/issues/434)) ([f9b1ffe](f9b1ffe)) * verified-fetch header cache-control typo ([#433](https://github.com/ipfs/helia-verified-fetch/issues/433)) ([ea39b48](ea39b48)) * walking dag-cbor paths ([#39](#39)) ([99668ce](99668ce)) * X-Ipfs-Path is set correctly ([#46](#46)) ([5bb6685](5bb6685)) ### Trivial Changes * add tests for directory redirects for gateways ([#15](#15)) ([269609d](269609d)) * fix build ([#22](#22)) ([01261fe](01261fe)) * initial import ([4b4fb8c](4b4fb8c)) * **release:** 1.0.0 [skip ci] ([ef5b6a5](ef5b6a5)), closes [#68](#68) * **release:** 1.0.0 [skip ci] ([a584a1a](a584a1a)) * **release:** 1.0.0 [skip ci] ([5207a64](5207a64)) * **release:** 1.0.1 [skip ci] ([5499724](5499724)), closes [#81](#81) * **release:** 1.0.1 [skip ci] ([8d2fdda](8d2fdda)), closes [#2](#2) * **release:** 1.0.2 [skip ci] ([104af2d](104af2d)), closes [#3](#3) * **release:** 1.1.0 [skip ci] ([e300af2](e300af2)) * **release:** 1.1.0 [skip ci] ([f4fcb4d](f4fcb4d)), closes [#4](#4) [#5](#5) [#6](#6) * **release:** 1.1.0 [skip ci] ([b9b959b](b9b959b)) * **release:** 1.1.1 [skip ci] ([8ff0b8b](8ff0b8b)), closes [#83](#83) * **release:** 1.1.1 [skip ci] ([6f2c6bd](6f2c6bd)), closes [#15](#15) [#14](#14) * **release:** 1.1.2 [skip ci] ([6161aba](6161aba)), closes [#85](#85) * **release:** 1.1.2 [skip ci] ([4e1abc0](4e1abc0)), closes [#16](#16) * **release:** 1.1.3 [skip ci] ([14c2302](14c2302)), closes [#18](#18) * **release:** 1.10.0 [skip ci] ([4589c26](4589c26)) * **release:** 1.10.1 [skip ci] ([140bab3](140bab3)), closes [#33](#33) * **release:** 1.11.0 [skip ci] ([39bb14e](39bb14e)) * **release:** 1.12.0 [skip ci] ([50682bf](50682bf)) * **release:** 1.13.0 [skip ci] ([e00d41c](e00d41c)) * **release:** 1.14.0 [skip ci] ([2521838](2521838)) * **release:** 1.15.0 [skip ci] ([a38a0c6](a38a0c6)) * **release:** 1.15.1 [skip ci] ([f59e862](f59e862)), closes [#47](#47) * **release:** 1.16.0 [skip ci] ([0a18c44](0a18c44)) * **release:** 1.17.0 [skip ci] ([f5299cf](f5299cf)) * **release:** 1.18.0 [skip ci] ([7771af2](7771af2)) * **release:** 1.19.0 [skip ci] ([089635d](089635d)) * **release:** 1.2.0 [skip ci] ([befd270](befd270)) * **release:** 1.2.0 [skip ci] ([e836abf](e836abf)), closes [#10](#10) [#22](#22) * **release:** 1.2.0 [skip ci] ([77d5e9e](77d5e9e)) * **release:** 1.2.1 [skip ci] ([718644b](718644b)), closes [#24](#24) * **release:** 1.20.0 [skip ci] ([abaaeab](abaaeab)) * **release:** 1.21.0 [skip ci] ([aeb5a89](aeb5a89)) * **release:** 1.21.1 [skip ci] ([f7ac2e7](f7ac2e7)), closes [#73](#73) * **release:** 1.22.0 [skip ci] ([6d4b26c](6d4b26c)), closes [#50](#50) * **release:** 1.23.0 [skip ci] ([754e219](754e219)) * **release:** 1.24.0 [skip ci] ([c1a795f](c1a795f)) * **release:** 1.24.1 [skip ci] ([9f8fb86](9f8fb86)), closes [#85](#85) [#54](#54) * **release:** 1.25.0 [skip ci] ([3d8bb78](3d8bb78)) * **release:** 1.3.0 [skip ci] ([43c41e4](43c41e4)), closes [#26](#26) [#17](#17) [#23](#23) [#10](#10) * **release:** 1.3.0 [skip ci] ([bf8af38](bf8af38)) * **release:** 1.3.1 [skip ci] ([0bf5847](0bf5847)), closes [#30](#30) * **release:** 1.3.10 [skip ci] ([6768fbd](6768fbd)), closes [#51](#51) * **release:** 1.3.11 [skip ci] ([8606b15](8606b15)) * **release:** 1.3.12 [skip ci] ([969a2e0](969a2e0)), closes [#42](#42) * **release:** 1.3.13 [skip ci] ([a180547](a180547)), closes [#53](#53) [#46](#46) * **release:** 1.3.14 [skip ci] ([31f238a](31f238a)), closes [#63](#63) * **release:** 1.3.2 [skip ci] ([2e141ca](2e141ca)), closes [#33](#33) * **release:** 1.3.3 [skip ci] ([19574cf](19574cf)), closes [#34](#34) * **release:** 1.3.4 [skip ci] ([7b06a56](7b06a56)), closes [#40](#40) * **release:** 1.3.5 [skip ci] ([554bb3e](554bb3e)), closes [#41](#41) * **release:** 1.3.6 [skip ci] ([489a1d6](489a1d6)), closes [#36](#36) * **release:** 1.3.7 [skip ci] ([d0114c1](d0114c1)), closes [#39](#39) * **release:** 1.3.8 [skip ci] ([0a37856](0a37856)), closes [#44](#44) [#47](#47) * **release:** 1.3.9 [skip ci] ([768e3ff](768e3ff)), closes [#49](#49) * **release:** 1.4.0 [skip ci] ([dab598a](dab598a)), closes [#50](#50) * **release:** 1.4.0 [skip ci] ([e008b20](e008b20)) * **release:** 1.4.1 [skip ci] ([e3c0a68](e3c0a68)), closes [#68](#68) [#72](#72) [#72](#72) * **release:** 1.4.2 [skip ci] ([2633d31](2633d31)), closes [#45](#45) * **release:** 1.4.3 [skip ci] ([924a6df](924a6df)), closes [#85](#85) * **release:** 1.5.0 [skip ci] ([4885fde](4885fde)), closes [#94](#94) * **release:** 1.5.0 [skip ci] ([71d613d](71d613d)) * **release:** 1.5.1 [skip ci] ([4e6775c](4e6775c)), closes [#12](#12) * **release:** 1.6.0 [skip ci] ([03d0409](03d0409)) * **release:** 1.7.0 [skip ci] ([8bf9c9f](8bf9c9f)) * **release:** 1.8.0 [skip ci] ([a240056](a240056)) * **release:** 1.9.0 [skip ci] ([8715740](8715740)) * start running CI again ([#64](#64)) ([9399f95](9399f95)) * stub blocks interface in tests ([#418](https://github.com/ipfs/helia-verified-fetch/issues/418)) ([a06f64b](a06f64b)) * unskip IPNS test ([#6](#6)) ([76485a4](76485a4)) * Update .github/pull_request_template.md [skip ci] ([5b825e6](5b825e6)) * Update .github/workflows/stale.yml [skip ci] ([3e40ad0](3e40ad0)) * update project config ([#419](https://github.com/ipfs/helia-verified-fetch/issues/419)) ([cb9dc90](cb9dc90)) ### Documentation * add link to blog post and ready-to-run example ([#63](#63)) ([696ed57](696ed57)) * add link to the blog post and example ([#87](#87)) ([f30f979](f30f979)) * change cidv0 to cidv1 in the readme ([#105](#105)) ([060e726](060e726)) * fix readme link ([#51](#51)) ([8a41c57](8a41c57)) * generate readme ([#70](#70)) ([a75567a](a75567a)), closes [#68](#68) [#72](#72) [#72](#72) * move the package docs to root of repo ([#1](#1)) ([8fb4a76](8fb4a76)) * update api docs link in readme ([#14](#14)) ([d615633](d615633)) * update documented default value ([161a470](161a470)), closes [/github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts#L6-L7](https://github.com/ipfs//github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts/issues/L6-L7) ### Dependencies * bump aegir from 42.2.11 to 44.1.4 ([#108](#108)) ([e36fbff](e36fbff)) * bump kubo from 0.26.0 to 0.27.0 ([#12](#12)) ([92cad49](92cad49)) * bump kubo from 0.27.0 to 0.28.0 ([#54](#54)) ([3579844](3579844)) * updating all deps ([#47](#47)) ([6d0ffd8](6d0ffd8)) ### Tests * add gateway conformance tests ([#67](#67)) ([30958fb](30958fb)), closes [#68](#68) * use successRate for gateway conformance tests ([#83](#83)) ([5f71a33](5f71a33))
🎉 This PR is included in version 1.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Title
Add Range header support.
Description
Fixes #9
Notes & open questions
handling ReadableStreams
tests, but these are covered byrange requests
tests.. going to try to fix quickly and if I can't i'll remove them.Change checklist