Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

ipfs files write corruption in large HAMTs #3921

Closed
aschmahmann opened this issue Oct 18, 2021 · 0 comments · Fixed by #3936
Closed

ipfs files write corruption in large HAMTs #3921

aschmahmann opened this issue Oct 18, 2021 · 0 comments · Fixed by #3936
Labels
P1 High: Likely tackled by core team if no one steps up

Comments

@aschmahmann
Copy link
Contributor

aschmahmann commented Oct 18, 2021 •
edited
Loading

  • Version:
    js-ipfs version: 0.11.1-5ddd0c555cc6577ee15667f4ef4e7f3a3b17d04c
    interface-ipfs-core version: ^0.151.1
    ipfs-http-client version: 1.0.0
    Repo version: 11
    System version: x64/linux
    Node.js version: v16.11.1
    Commit: 5ddd0c5

  • Platform:
    Linux AdinPC 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  • Subsystem:
    MFS

Severity:

High

Description:

  • Added a large HAMT to jsipfs
  • Copied the directory to MFS
  • Wrote a small file into the new MFS directory
  • Noticed that data disappeared

Steps to reproduce the error:

  1. Extract the zip
  2. jsipfs dag import basehamt.car (the output root CID is QmRrSZQc6bZBvdLzzeWy6ekLW4jGpNfTdiv1MeDstDmiM8)
  3. jsipfs files cp /ipfs/QmRrSZQc6bZBvdLzzeWy6ekLW4jGpNfTdiv1MeDstDmiM8 /hamttest-mfs
  4. cat supermodule_test | jsipfs files write --create /hamttest-mfs/supermodule_test
  5. jsipfs stat /hamttest-mfs (should be QmTES6nWfSKFiDHUfqh4gNpibq9STwrUTjXnUxMsrXcdHC)
  6. jsipfs ls QmRrSZQc6bZBvdLzzeWy6ekLW4jGpNfTdiv1MeDstDmiM8 | wc -l (output is 2006)
  7. jsipfs ls QmTES6nWfSKFiDHUfqh4gNpibq9STwrUTjXnUxMsrXcdHC | wc -l (output is 2005 despite us having just added a new file)
@aschmahmann aschmahmann added the need/triage Needs initial labeling and prioritization label Oct 18, 2021
@BigLep BigLep added P1 High: Likely tackled by core team if no one steps up and removed need/triage Needs initial labeling and prioritization labels Oct 19, 2021
@aschmahmann aschmahmann mentioned this issue Nov 10, 2021
Closed
achingbrain added a commit that referenced this issue Nov 11, 2021
@achingbrain
fix: do not lose files when writing files into subshards that contain…
1aba3f3 
… other subshards

When writing a file into a hamt shard we hash the filename to figure out where in the
shard to place the file.

If the hash means that we end up adding the file into an existing subshard that also
contains another subshard, we should populate the other subshard's children otherwise
they will not be there when we calculate the new CID for the subshard and they will
be lost.

Fixes #3921
@achingbrain achingbrain mentioned this issue Nov 11, 2021
Merged
achingbrain added a commit that referenced this issue Nov 12, 2021
@achingbrain
fix: do not lose files when writing files into subshards that contain…
8a3ed19 
… other subshards (#3936)

When writing a file into a hamt shard we hash the filename to figure out where in the
shard to place the file.

If the hash means that we end up adding the file into an existing subshard that also
contains another subshard, we should populate the other subshard's children otherwise
they will not be there when we calculate the new CID for the subshard and they will
be lost.

Fixes #3921
@aschmahmann aschmahmann mentioned this issue Nov 17, 2021
Closed
@tinytb tinytb moved this to Done in IP JS (PL EngRes) v2 Oct 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
P1 High: Likely tackled by core team if no one steps up
Projects
No open projects
IP JS (PL EngRes) v2
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: do not lose files when writing files into subshards that contain other subshards ipfs/js-ipfs
2 participants
@BigLep @aschmahmann