Align sharding threshold detection #149
Labels
kind/enhancement
A net-new feature or improvement to an existing feature
P2
Medium: Good to have, but can wait until someone steps up
released
status/ready
Ready to be worked
Comments
achingbrain
added
status/ready
achingbrain
added a commit
that referenced
this issue
Sep 12, 2021
js counterpart to ipfs/kubo#8114 Changes the `shardSplitThreshold` parameter to mean the size of the final DAGNode (including link names, sizes, etc) instead of the number of entries in the directory. Fixes: #149 BREAKING CHANGE: `shardSplitThreshold` now refers to node size, not number of entries
lidel
added a commit
to ipfs/js-ipfs
that referenced
this issue
Dec 7, 2021
Experimental.ShardingEnabled was replaced by autosharding.
To simulate the old behavior ("shard everything") one needs to set
Internal.UnixFSShardingSizeThreshold to 1B.
This is a temporary fix for sharding tests until js-ipfs implements
ipfs/js-ipfs-unixfs#149
lidel
added a commit
to ipfs/js-ipfs
that referenced
this issue
Dec 7, 2021
Experimental.ShardingEnabled was replaced by autosharding.
To simulate the old behavior ("shard everything") one needs to set
Internal.UnixFSShardingSizeThreshold to 1B.
This is a temporary fix for sharding tests until js-ipfs implements
ipfs/js-ipfs-unixfs#149
|
2022-11-22: still needs to be done; no time pressure. |
achingbrain
added a commit
that referenced
this issue
Feb 9, 2023
js counterpart to ipfs/kubo#8114 Changes the `shardSplitThreshold` parameter to mean the size of the final DAGNode (including link names, sizes, etc) instead of the number of entries in the directory. Fixes: #149 BREAKING CHANGE: the `shardSplitThreshold` option has changed to `shardSplitThresholdBytes` and reflects a DAGNode size where sharding might kick in
github-actions bot
pushed a commit
that referenced
this issue
Feb 9, 2023
## [ipfs-unixfs-v10.0.0](ipfs-unixfs-v9.0.1...ipfs-unixfs-v10.0.0) (2023-02-09) ### ⚠ BREAKING CHANGES * the `shardSplitThreshold` option has changed to `shardSplitThresholdBytes` and reflects a DAGNode size where sharding might kick in ### Features * auto-shard based on node size ([#171](#171)) ([6ef187f](6ef187f)), closes [#149](#149)
|
🎉 This issue has been resolved in version ipfs-unixfs-v10.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
github-actions bot
pushed a commit
that referenced
this issue
Feb 9, 2023
## [ipfs-unixfs-importer-v13.0.0](ipfs-unixfs-importer-v12.0.1...ipfs-unixfs-importer-v13.0.0) (2023-02-09) ### ⚠ BREAKING CHANGES * the `shardSplitThreshold` option has changed to `shardSplitThresholdBytes` and reflects a DAGNode size where sharding might kick in ### Features * auto-shard based on node size ([#171](#171)) ([6ef187f](6ef187f)), closes [#149](#149) ### Dependencies * update sibling dependencies ([1a37705](1a37705))
|
🎉 This issue has been resolved in version ipfs-unixfs-importer-v13.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
github-actions bot
pushed a commit
that referenced
this issue
Feb 9, 2023
## [ipfs-unixfs-exporter-v11.0.0](ipfs-unixfs-exporter-v10.0.1...ipfs-unixfs-exporter-v11.0.0) (2023-02-09) ### ⚠ BREAKING CHANGES * the `shardSplitThreshold` option has changed to `shardSplitThresholdBytes` and reflects a DAGNode size where sharding might kick in ### Features * auto-shard based on node size ([#171](#171)) ([6ef187f](6ef187f)), closes [#149](#149) ### Dependencies * update sibling dependencies ([54018a1](54018a1)) * update sibling dependencies ([1a37705](1a37705))
|
🎉 This issue has been resolved in version ipfs-unixfs-exporter-v11.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
go-IPFS has taken a different approach to js-IPFS in deciding when to start using a HAMT shard instead of a regular directory.
js-IPFS has an arbitrary 1k entry limit, overridable by the user.
go-IPFS calculates the likely block size of the directory node, sharding once it increases above a certain size.
We should align with the go-IPFS calculation, in order to not have divergent hashes for a given directory.
Ref: ipfs/kubo#7022 (comment)
The text was updated successfully, but these errors were encountered: