Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract Lotus CommP writer and use when calculating CommP in markets #4899

Closed
hannahhoward opened this issue Nov 18, 2020 · 0 comments · Fixed by filecoin-project/go-fil-markets#452 or #4929
Assignees
Labels
area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/enhancement Kind: Enhancement P0 P0: Critical Blocker

Comments

@hannahhoward
Copy link
Contributor

What

Currently when we calculate commP in go-fil-markets we pass an io.Reader of the whole piece to the FFI. The FFI seems to write a temp file the size of the whole piece to the OS temp directory during the process of calculating CommP. Instead of using the FFI directly, we can use Lotus's commP writer: https://github.com/filecoin-project/lotus/blob/master/lib/commp/writer.go which chunks up calls to the FFI when writing tempfiles so that it doesn't write a file the size of the whole piece to OS temp directory, which may not have a lot of space in a miner setup for large pieces.

How

  • extract lib/commp writer from lotus -- I think it would be nice to make a new shared repo that both go-fil-markets and lotus use -- also it may make sense to actually extract pieceIO from go-fil-markets as well and combine it. Though perhaps just extracting the commP writer as a start from lotus is a good place to be.

  • on top of Remove temp files go-fil-markets#449, rewrite GeneratePieceCommitment in PieceIO to use the commP writer. This will probably entail not re-using the GeneratePieceReader function in GeneratePieceCommitment any more, and instead calling PrepareCar directly and passing the extract commP writer directly to PreparedCar.Dump (then calling writer.Sum)

@hannahhoward hannahhoward added area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/enhancement Kind: Enhancement P0 P0: Critical Blocker labels Nov 18, 2020
@dineshshenoy dineshshenoy added this to the 💹Storage Deal Success milestone Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/enhancement Kind: Enhancement P0 P0: Critical Blocker
Projects
None yet
3 participants