Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: sector index yugabyte implementation #11135

Merged
merged 21 commits into from
Aug 22, 2023

Conversation

shrenujbansal
Copy link
Contributor

@shrenujbansal shrenujbansal commented Aug 4, 2023

Related Issues

#11025

Proposed Changes

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@shrenujbansal shrenujbansal requested a review from a team as a code owner August 4, 2023 15:36
@shrenujbansal shrenujbansal changed the title WIP: sector index yugabyte impl feat: sector index yugabyte impl Aug 9, 2023
@shrenujbansal shrenujbansal changed the title feat: sector index yugabyte impl feat: sector index yugabyte implementation Aug 9, 2023
storage/paths/index.go Outdated Show resolved Hide resolved
@rjan90 rjan90 linked an issue Aug 9, 2023 that may be closed by this pull request
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, looks quite straightforward, the only thing is really addressing the todos, especially the transactional StorageAttach (that has to be correctly transactional, often related paths will be attached by multiple workers at the same time, e.g. when workers are bulk-restarted)

lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
cmd/lotus-miner/init.go Show resolved Hide resolved
lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
storage/paths/index.go Outdated Show resolved Hide resolved
return out, nil
}

func union(a, b []string) []string {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lo.Union() ?

storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Show resolved Hide resolved
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, this looks like a real thing now, just a few comments

cmd/lotus-miner/init.go Show resolved Hide resolved
node/config/types.go Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
Comment on lines 829 to 832
_, err := dbi.lock(ctx, sector, read, write)
if err != nil {
return err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably should retry / backoff / block when we don't get the lock immediately

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. Could you review the parameters for the retries and wait time and see if they make sense? @magik6k

storage/paths/db_index.go Show resolved Hide resolved
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is getting really close,

Few comments on the locking side of things, and it would be nice to avoid passing a query built through Sprintf in StorageBestAlloc. Other than that this looks shippable

lib/harmony/harmonydb/sql/20230712.sql Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Show resolved Hide resolved
@magik6k
Copy link
Contributor

magik6k commented Aug 17, 2023

(also CI seems sad, but not sure if this is related to this PR or feat/sturdypost)

Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly small nitpicky stuff, everything looks pretty solid now, next review should be shipping the code

lib/harmony/harmonydb/userfuncs.go Outdated Show resolved Hide resolved
node/builder_miner.go Outdated Show resolved Hide resolved
storage/paths/db_index.go Outdated Show resolved Hide resolved
@shrenujbansal shrenujbansal merged commit 5b6daa2 into feat/sturdypost Aug 22, 2023
@shrenujbansal shrenujbansal deleted the sbansal/sector-index branch August 22, 2023 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SectorIndex rework for HA
3 participants