Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove temporary piece file generation from go-fil-markets for ONLINE deals #4898

Closed
hannahhoward opened this issue Nov 18, 2020 · 3 comments · Fixed by #4929
Closed

Remove temporary piece file generation from go-fil-markets for ONLINE deals #4898

hannahhoward opened this issue Nov 18, 2020 · 3 comments · Fixed by #4929
Assignees
Labels
area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/feature Kind: Feature P0 P0: Critical Blocker

Comments

@hannahhoward
Copy link
Contributor

What

Currently when we verify CommP on the provider side, we read the data for the piece out of the blockstore and write the CAR file as a temporary file to disk for the piece, stored in the .lotusminer directory, and feed it later to the sector storage interface, deleting it only later. This can cause problems for miners who only have large storage allocated towards the sectors themselves.

It would be better to:

  1. Just write to an OS pipe like we do on the client side when generating commP.
  2. When we handoff to the sector storage interface, regenerate the CAR file, writing to an OS pipe again but this time feeding the reader to the sector storage interface.

Things to consider

  • The rarely used feature to write an index of all CIDs in a piece, and their offsets in the CAR file, using hooks in writing the car file on each new block (future ticket: remove this feature unless someone is going to use it)
  • We need to still enable the filestore flow for offline deals, at least for the moment
@hannahhoward
Copy link
Contributor Author

Implemented as draft in filecoin-project/go-fil-markets#449

Also requires ipld/go-car#37
^^^ though beware issues around having to continue to maintain a fork for filestore support :(

@hannahhoward hannahhoward added area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/feature Kind: Feature P0 P0: Critical Blocker labels Nov 18, 2020
@dineshshenoy
Copy link

@dineshshenoy dineshshenoy added this to the 💹Storage Deal Success milestone Nov 18, 2020
@dirkmc
Copy link
Contributor

dirkmc commented Nov 20, 2020

Fixed by filecoin-project/go-fil-markets#449

@dirkmc dirkmc closed this as completed Nov 20, 2020
@dirkmc dirkmc reopened this Nov 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/markets Area: Markets dif/medium Prior development experience with lotus is likely helpful kind/feature Kind: Feature P0 P0: Critical Blocker
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants