Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New very fast unixfs implementation. #347

Closed
wants to merge 41 commits into from
Closed

New very fast unixfs implementation. #347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
2ff6608
Initial Commit
Jorropo Oct 6, 2022
3bbf591
support multiblock files
Jorropo Oct 6, 2022
8ed010b
I should add a CI (go mod tidy)
Jorropo Oct 6, 2022
40d2196
fix: correctly decode inline CIDs
Jorropo Oct 6, 2022
9b572d0
chore: cleanup low and high in the default region
Jorropo Oct 6, 2022
8bb4200
refactor: use a simpler stack for the state
Jorropo Jan 5, 2023
2242d0b
fix: correctly handle rangeKnown
Jorropo Jan 5, 2023
86975d6
cleanup: make the maxElementSize clearer
Jorropo Jan 5, 2023
a20ddaa
feat: automatically close the connection if Read fails
Jorropo Jan 5, 2023
4707f58
cleanup: clean identity expression
Jorropo Jan 5, 2023
0169b32
fix: identity expression
Jorropo Mar 25, 2023
f5f8e33
unixfs: bootstrap new implementation
Jorropo Jun 11, 2023
0fa2b0b
unixfs: add pb file support
Jorropo Jun 12, 2023
a8c8644
unixfs: Add support for byte slices backed CIDs for zero allocations
Jorropo Jun 12, 2023
7ece6c8
unixfs: fix CI and style
Jorropo Jun 17, 2023
f7dd803
unixfs: fix handling of unknown fields
Jorropo Jun 17, 2023
6d04a4b
unixfs: hoist unixfs related checks in format blind code
Jorropo Jun 26, 2023
df5bf24
chore: bump go to 1.21
Jorropo Oct 5, 2023
f0e238a
unixfs: bring feather implementation in
Jorropo Oct 8, 2023
f4324f8
unixfs/feather: implement IPIP402 and IPIP412
Jorropo Oct 8, 2023
c782262
unixfs/feather: stop checking for the root CID of received cars
Jorropo Oct 8, 2023
2a7b531
cmd/feather: fix capitalised errors
Jorropo Oct 9, 2023
f7b2c40
feather: fix IPIP-412 request
Jorropo Nov 10, 2023
ecedd73
unixfs: fix UnmarshalText
Jorropo Nov 10, 2023
f21df27
unixfs: change Node to always use string like CIDs
Jorropo Nov 10, 2023
d7e2f95
unixfs/feather: make use of boxo/unixfs easy api
Jorropo Nov 10, 2023
d6bb1dd
unixfs/feather: normalize CIDs v0 to v1
Jorropo Nov 10, 2023
2268a17
unixfs/feather: remove useless early gc
Jorropo Nov 11, 2023
00cd233
unixfs/feather: cleanup if branch
Jorropo Nov 11, 2023
1b0f794
unixfs/feather: change semantics of Read to not close on error
Jorropo Nov 11, 2023
90b0001
unixfs/feather: harden against Read being called after an error happend
Jorropo Nov 11, 2023
969998d
unixfs/feather: add a basic test
Jorropo Nov 16, 2023
1526c37
unixfs/feather: normalise CIDs from the car
Jorropo Nov 16, 2023
eec31dc
unixfs/feather: stop checking the CAR header version
Jorropo Nov 16, 2023
e9bb00c
unixfs/feather: remove incorrect TODO
Jorropo Nov 16, 2023
ebb6727
unixfs/feather: hoist readBlockFromStream to it's own function
Jorropo Nov 16, 2023
6cdc275
unixfs/feather: add WithRetries option
Jorropo Nov 18, 2023
ca14179
unixfs, unixfs/feather: cleanup imports
Jorropo Nov 18, 2023
c3929ff
unixfs/feather: move retry logic to it's own next function
Jorropo Nov 18, 2023
b568a58
unixfs/feather: cleanup CID normalization
Jorropo Nov 18, 2023
2599f37
TODO
Jorropo Nov 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
cmd/boxo-migrate/boxo-migrate
vendor
57 changes: 57 additions & 0 deletions cmd/feather/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package main

import (
"fmt"
"io"
"os"

"github.com/ipfs/boxo/unixfs/feather"
"github.com/ipfs/go-cid"
)

func main() {
err := mainRet()
if err != nil {
os.Stderr.WriteString(err.Error())
os.Stderr.WriteString("\n")
os.Exit(1)
}
os.Exit(0)
}

func parseArgs() (cid.Cid, error) {
if len(os.Args) != 2 {
return cid.Cid{}, fmt.Errorf("expected one argument")
}

return cid.Decode(os.Args[1])
}

func mainRet() error {
c, err := parseArgs()
if err != nil {
return fmt.Errorf(`%w
Usage:
%s <CID>

Example:
%s bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi`, err, os.Args[0], os.Args[0])
}

f, err := feather.NewClient(feather.WithStaticGateway("http://localhost:8080/"))
if err != nil {
return fmt.Errorf("creating feather client: %w", err)
}

r, err := f.DownloadFile(c)
if err != nil {
return fmt.Errorf("starting file download: %w", err)
}
defer r.Close()

_, err = io.Copy(os.Stdout, r)
if err != nil {
return fmt.Errorf("downloading file: %w", err)
}
return nil
}
2 changes: 1 addition & 1 deletion examples/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.20
require (
github.com/ipfs/boxo v0.8.0
github.com/ipfs/go-block-format v0.1.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662
github.com/ipfs/go-datastore v0.6.0
github.com/ipld/go-car/v2 v2.10.2-0.20230622090957-499d0c909d33
github.com/ipld/go-ipld-prime v0.21.0
Expand Down
4 changes: 2 additions & 2 deletions examples/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,8 @@ github.com/ipfs/go-blockservice v0.5.0 h1:B2mwhhhVQl2ntW2EIpaWPwSCxSuqr5fFA93Ms4
github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.6/go.mod h1:6Ux9z5e+HpkQdckYoX1PG/6xqKspzlEIR5SDmgqgC/I=
github.com/ipfs/go-cid v0.4.1 h1:A/T3qGvxi4kpKWWcPC/PgbvDA2bjVLO7n4UeVwnbs/s=
github.com/ipfs/go-cid v0.4.1/go.mod h1:uQHwDeX4c6CtyrFwdqyhpNcxVewur1M7l7fNU7LKwZk=
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662 h1:jWQ5yOEmR1Fvv6Rj8mEye4QTeGQoKKECMieju9z5kgA=
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662/go.mod h1:4rtyA9XdBeZBapaRNJuTY9H+/6bG4URx/cVwjAzK6fw=
github.com/ipfs/go-datastore v0.6.0 h1:JKyz+Gvz1QEZw0LsX1IBn+JFCJQH4SJVFtM4uWU0Myk=
github.com/ipfs/go-datastore v0.6.0/go.mod h1:rt5M3nNbSO/8q1t4LNkLyUwRs8HupMeN/8O4Vn9YAT8=
github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk=
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ require (
github.com/ipfs/bbloom v0.0.4
github.com/ipfs/go-bitfield v1.1.0
github.com/ipfs/go-block-format v0.1.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662
github.com/ipfs/go-cidutil v0.1.0
github.com/ipfs/go-datastore v0.6.0
github.com/ipfs/go-detect-race v0.0.1
Expand Down Expand Up @@ -71,6 +71,7 @@ require (
go.opentelemetry.io/otel/trace v1.14.0
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.25.0
golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63
golang.org/x/oauth2 v0.8.0
golang.org/x/sync v0.3.0
golang.org/x/sys v0.11.0
Expand Down Expand Up @@ -157,7 +158,6 @@ require (
go.uber.org/dig v1.17.0 // indirect
go.uber.org/fx v1.20.0 // indirect
golang.org/x/crypto v0.12.0 // indirect
golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/net v0.14.0 // indirect
golang.org/x/text v0.12.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,8 +252,8 @@ github.com/ipfs/go-blockservice v0.5.0 h1:B2mwhhhVQl2ntW2EIpaWPwSCxSuqr5fFA93Ms4
github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.6/go.mod h1:6Ux9z5e+HpkQdckYoX1PG/6xqKspzlEIR5SDmgqgC/I=
github.com/ipfs/go-cid v0.4.1 h1:A/T3qGvxi4kpKWWcPC/PgbvDA2bjVLO7n4UeVwnbs/s=
github.com/ipfs/go-cid v0.4.1/go.mod h1:uQHwDeX4c6CtyrFwdqyhpNcxVewur1M7l7fNU7LKwZk=
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662 h1:jWQ5yOEmR1Fvv6Rj8mEye4QTeGQoKKECMieju9z5kgA=
github.com/ipfs/go-cid v0.4.2-0.20230612091241-80d1e915f662/go.mod h1:4rtyA9XdBeZBapaRNJuTY9H+/6bG4URx/cVwjAzK6fw=
github.com/ipfs/go-cidutil v0.1.0 h1:RW5hO7Vcf16dplUU60Hs0AKDkQAVPVplr7lk97CFL+Q=
github.com/ipfs/go-cidutil v0.1.0/go.mod h1:e7OEVBMIv9JaOxt9zaGEmAoSlXW9jdFZ5lP/0PwcfpA=
github.com/ipfs/go-datastore v0.6.0 h1:JKyz+Gvz1QEZw0LsX1IBn+JFCJQH4SJVFtM4uWU0Myk=
Expand Down
Loading