Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage-proofs): partial caching for SDR #1163

Merged
merged 12 commits into from
Jun 17, 2020
Merged

Conversation

dignifiedquire
Copy link
Contributor

@dignifiedquire dignifiedquire commented Jun 15, 2020

  • reduces memory usage during precommit_phase1 by 56GiB for 3i2GB sectors, when using max caching
  • generate and store parents cache on disk
  • mmap cache to use it
  • only mmap part of it, to minimize memory usage
  • cache is local, not global anymore
  • cache can be used by all sector sizes
  • enables cache usage by default, given its small overhead (112 KiB at the current configuration)
  • increases required fast disk space by 56GiB for 32GiB sectors, this file is stored in /var/tmp/filecoin-parents for now

Based on the ideas in #1135

Copy link
Collaborator

@cryptonemo cryptonemo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Is the CI error real?

@dignifiedquire
Copy link
Contributor Author

Looks good. Is the CI error real?

I believe it is a bug in the tests, but will investigate

porcuquine
porcuquine previously approved these changes Jun 17, 2020
Copy link
Collaborator

@porcuquine porcuquine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking request to make the cache size a setting.

Otherwise looks good, once CI is fixed. If fixing CI means changing the test, I'd like to see what's going on before we merge.

const NODE_GIB: u32 = (1024 * 1024 * 1024) / NODE_SIZE as u32;

// Number of nodes to be cached in memory
const DEFAULT_CACHE_SIZE: u32 = 2048;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should go in settings.rs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. If there are constraints on this value we know need to be honored, please document those too (in settings.rs). Does it need to evenly divide the number of nodes, for example? People should be allowed to shoot themselves in the foot, but we should warn them where the foot is, too.

@deltazxm
Copy link

no mem cache anymore?I think this will affect multiple sector sealing speeds.mem cache still necessary

@dignifiedquire
Copy link
Contributor Author

no mem cache anymore?

there is still a mem cache, it is simply done differently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cryptocomputelab CryptoComputeLab work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants