Uber migration #18
Merged
Uber migration #18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit was moved from ipfs/go-fetcher@99235cc
…sion add a fetcher constructor for the case where we already have a session This commit was moved from ipfs/go-fetcher@204874a
This commit was moved from ipfs/go-path@6f59923
This reverts commit b7dfe90. This commit was moved from ipfs/go-path@bda72a4
* creation of car from file / directory Co-authored-by: Daniel Martí <mvdan@mvdan.cc> Co-authored-by: Rod Vagg <rod@vagg.org> This commit was moved from ipld/go-car@9bd7416
This commit was moved from ipld/go-car@4a8d623
This commit was moved from ipld/go-car@5a583de
This commit was moved from ipfs/go-ipfs-exchange-interface@09692a6
This commit was moved from ipfs/go-ipfs-routing@be1ea98
This commit was moved from ipld/go-car@85f0751
* support extraction of unixfs content stored in car files This commit was moved from ipld/go-car@6d94b7b
This commit was moved from ipfs/go-ipfs-exchange-offline@6ef3e0a
This commit was moved from ipfs/go-blockservice@285f9bc
This commit was moved from ipfs/go-merkledag@bfaa766
This commit was moved from ipfs/go-fetcher@d2ffddc
This commit was moved from ipfs/go-path@19b77b2
This commit was moved from ipfs/go-ipfs-pinner@72f5e02
This commit was moved from ipfs/go-ipfs-provider@4aff05e
* Renamed UpgradeableDirectory to DynamicDirectory to indicate that we can switch between Basic and HAMT Directories * Transition between HAMT directory and Basic Directory based on the global sharding threshold * Unexported BasicDirectory.SwitchToSharding as an unnecessary exposure point until requested * NewDirectoryFromNode always returns a DynamicDirectory instead of an UpgradeableDirectory or HAMTDirectory * Added Swap and Take functions to HAMT Shards * Fix for the size estimation logic where we were not tracking that replacing an entry with a differently sized CID could trigger switching between being a Basic or HAMT directory * Use custom parallel DAG traversal to the EnumLinksAsync for HAMTs that is closer to DFS than BFS * Added lots of comments to the HAMT code * Exported LogTwo function to make it more accessible within the package Co-authored-by: Lucas Molas <schomatis@gmail.com> Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com> This commit was moved from ipfs/go-unixfs@bd53b6a
This commit was moved from ipfs/go-ipfs-blockstore@60f8b66
This commit was moved from ipfs/go-unixfs@abdf700
…ies (ipfs#88) * feat: update go-unixfs and use built in automatic sharding and unsharding * chore: update deps Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com> Co-authored-by: Gus Eggert <gus@gus.dev> This commit was moved from ipfs/go-mfs@e61420f
This commit was moved from ipfs/go-ipfs-provider@bb73e1b
This commit was moved from ipfs/go-ipfs-provider@56883e0
…-staticcheck Fix go vet and staticcheck This commit was moved from ipfs/go-ipfs-provider@49b7fe2
The blockstore reads the version of the given CAR payload to determine whether to generate an index for the given payload or not. When the payload represents a CARv1 and no index is specified, the backing reader is passed on for index generation. But the version is already read from the stream. Seek to the beginning of the backing reader before generating the index so that the index generation mechanism receives the complete CARv1. Fixes ipfs#265 This commit was moved from ipld/go-car@fa995b9
This is so that v0 and v1 have identical interfaces, which is required to avoid massive pain for consumers like estuary. Since we have already plumbed v0 all the way through, it's easiest to just remove the probably-unnecessary context from HashOnRead. This commit was moved from ipfs/go-ipfs-blockstore@fbe708e
This commit was moved from ipfs/go-filestore@9d7d5b4
This commit was moved from ipfs/go-namesys@aaf9aa8
Add a writing/creation API in the car v2 package. Allows creation of file and streaming car files in a selector order. This commit was moved from ipld/go-car@c35591a
This commit was moved from ipld/go-car@1762841
close ipld/go-car#242 Crude benchmark: ``` func BenchmarkAllKeysChan(b *testing.B) { path := filepath.Join(b.TempDir(), "bench-large-v2.car") generateRandomCarV2File(b, path, 10<<20) // 10 MiB defer os.Remove(path) bs, err := blockstore.OpenReadWrite(path, nil) if err != nil { b.Fatal(err) } b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { c, err := bs.AllKeysChan(context.Background()) if err != nil { b.Fatal(err) } for range c { } } } func BenchmarkAllKeysChanUseWholeCIDs(b *testing.B) { path := filepath.Join(b.TempDir(), "bench-large-v2.car") generateRandomCarV2File(b, path, 10<<20) // 10 MiB defer os.Remove(path) bs, err := blockstore.OpenReadWrite(path, nil, carv2.UseWholeCIDs(true)) if err != nil { b.Fatal(err) } b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { c, err := bs.AllKeysChan(context.Background()) if err != nil { b.Fatal(err) } for range c { } } } ``` Before: > BenchmarkAllKeysChan-12 885 1256865 ns/op 88911 B/op 1617 allocs/op > BenchmarkAllKeysChanUseWholeCIDs-12 1010 1253543 ns/op 57861 B/op 976 allocs/op After > BenchmarkAllKeysChan-12 8971 135906 ns/op 30864 B/op 642 allocs/op > BenchmarkAllKeysChanUseWholeCIDs-12 13904 86140 ns/op 144 B/op 2 allocs/op BenchmarkAllKeysChan --- 9.25X faster, allocate 2.9X less memory BenchmarkAllKeysChanUseWholeCID --- 14.5X faster, allocate 401X less memory. This commit was moved from ipld/go-car@a4629d3
…s#23) This commit was moved from ipfs/go-ipfs-pinner@9abb80f
This commit was moved from ipfs/go-path@6793198
This commit was moved from ipfs/go-ipfs-pinner@1174ddd
* Add NextInsecure() method Attempt to make CAR traversal a little bit faster by not forcing users to hash every single in a CAR. * Add TrustedCAR option to BlockReader (remove NextInsecure()) * Add test for TrustedCAR option * fix: apply suggestions from code review --------- Co-authored-by: Rod Vagg <rod@vagg.org> This commit was moved from ipld/go-car@e9a77cb
The goal being to be able to keep reading blocks and the index after safely finalizing the file on disk. Typically for indexing purpose. This commit was moved from ipld/go-car@b6ef2a4
…error Co-authored-by: Rod Vagg <rod@vagg.org> This commit was moved from ipld/go-car@34f4b63
This commit was moved from ipld/go-car@dbd9059
This commit was moved from ipld/go-car@15e6582
This commit was moved from ipld/go-car@ba7e4d7
This commit was moved from ipld/go-car@5647641
This commit was moved from ipld/go-car@5e5e40a
This commit was moved from ipld/go-car@413fe0d
This commit was moved from ipld/go-car@2d54909
Similar to blockservice.NewWriteThrough(). Currently the default blockstore trades read amplification for every write. While this is fine in cases where writes are very expensive and the datastore offers a quick Has() method, it is not always the case. Some datastore backends may be better off just getting all the writes no matter what. At least I would like to try it out. This commit was moved from ipfs/go-ipfs-blockstore@498084a
This commit was moved from ipfs/go-ipfs-blockstore@7ad322b
This commit was moved from ipfs/go-ipfs-blockstore@1323a47
* feat: use peer.String() to use a human-readable cache key * fix: use normalized key representation * fix: cache ipns key can be any format This commit was moved from ipfs/go-namesys@af35385
- updates docs link to .tech - makes it very clear what is missing This commit was moved from ipfs/go-namesys@e30a7b8
allows non-path "full" extraction of a unixfs CAR that doesn't have complete data Ref: ipfs/go-unixfsnode#44 This commit was moved from ipld/go-car@22c855b
…ng on disk
Before:
// BenchmarkHas-8 190216 6368 ns/op 744 B/op 16 allocs/op
After
// BenchmarkHas-8 1419169 845.6 ns/op 320 B/op 6 allocs/op
```
func BenchmarkHas(b *testing.B) {
ctx := context.TODO()
path := filepath.Join(b.TempDir(), "bench-large-v2.car")
generateRandomCarV2File(b, path, 200<<20) // 10 MiB
defer os.Remove(path)
subject, err := blockstore.OpenReadWrite(path, nil)
c, err := subject.AllKeysChan(ctx)
require.NoError(b, err)
var allCids []cid.Cid
for c2 := range c {
allCids = append(allCids, c2)
}
b.ReportAllocs()
b.ResetTimer()
var idx int
for i := 0; i < b.N; i++ {
_, _ = subject.Has(ctx, allCids[idx])
// require.NoError(b, err)
// require.True(b, has)
idx = (idx + 1) % len(allCids)
}
}
```
This commit was moved from ipld/go-car@d51b4a1
This commit was moved from ipld/go-car@c58b233
…eadWriteFile OpenReadWriteFile gives full control of the file to the caller, so it should not decide to close the file, even on error. This commit was moved from ipld/go-car@794f222
This commit was moved from ipld/go-car@2f59674
Include rename from: github.com/ipfs/go-libipfs => github.com/ipfs/boxo Migrated repos: - github.com/ipfs/interface-go-ipfs-core => ./coreiface - github.com/ipfs/go-pinning-service-http-client => ./pinning/remote/client - github.com/ipfs/go-path => ./path - github.com/ipfs/go-namesys => ./namesys - github.com/ipfs/go-mfs => ./mfs - github.com/ipfs/go-ipfs-provider => ./provider - github.com/ipfs/go-ipfs-pinner => ./pinning/pinner - github.com/ipfs/go-ipfs-keystore => ./keystore - github.com/ipfs/go-filestore => ./filestore - github.com/ipfs/go-ipns => ./ipns - github.com/ipfs/go-blockservice => ./blockservice - github.com/ipfs/go-ipfs-chunker => ./chunker - github.com/ipfs/go-fetcher => ./fetcher - github.com/ipfs/go-ipfs-blockstore => ./blockstore - github.com/ipfs/go-ipfs-posinfo => ./filestore/posinfo - github.com/ipfs/go-ipfs-util => ./util - github.com/ipfs/go-ipfs-ds-help => ./datastore/dshelp - github.com/ipfs/go-verifcid => ./verifcid - github.com/ipfs/go-ipfs-exchange-offline => ./exchange/offline - github.com/ipfs/go-ipfs-routing => ./routing - github.com/ipfs/go-ipfs-exchange-interface => ./exchange - github.com/ipfs/go-unixfs => ./ipld/unixfs - github.com/ipfs/go-merkledag => ./ipld/merkledag - github.com/ipld/go-car => ./ipld/car Fixes ipfs#215 Updates ipfs#202
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.