Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve perf of msgindex backfill #10941

Merged
merged 1 commit into from
Jun 1, 2023
Merged

Conversation

fridrik01
Copy link
Contributor

@fridrik01 fridrik01 commented Jun 1, 2023

Context
When EnableMsgIndex is set we maintain a separate sqlite database in .lotus/sqlite/msgindex.db which maps message cid to the tipset and epoch its stored in. This allows us faster lookup and optimizes several RPC calls.

Currently the message index msgindex.db uses default journal mode which causes the database to be locked, especially when backfilling this database (using lotus-shed msgindex backfill ) over multiple epochs (like a FEVM arvhical node). When backfilling, the lotus node will soon lock the database and disable the msgindex logic until next lotus restart which also fails the backfilling itself.

This PR addresses these issues by:

  • switching this database to use the WAL journal mode. This will make writes at least an order of magnitude faster
  • updating the lotus-shed msgindex backfill code to not do writes if message already exists
  • removes the single long running transaction,
  • print some progress updates and improve error logging

Test plan

make lotus lotus-shed

Confirmed that backfilling no longer lockes the database and is also 20x+ faster than before:

time ./lotus-shed msgindex backfill --epochs=1000
2023-06-01T14:49:19.349Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       0/1000 processing epoch:2910817, nrRowsAffected:0
2023-06-01T14:49:20.119Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       100/1000 processing epoch:2910717, nrRowsAffected:0
2023-06-01T14:49:20.950Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       200/1000 processing epoch:2910617, nrRowsAffected:0
2023-06-01T14:49:21.703Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       300/1000 processing epoch:2910517, nrRowsAffected:0
2023-06-01T14:49:22.365Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       400/1000 processing epoch:2910417, nrRowsAffected:0
2023-06-01T14:49:23.343Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       500/1000 processing epoch:2910317, nrRowsAffected:7479
2023-06-01T14:49:24.451Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       600/1000 processing epoch:2910217, nrRowsAffected:20521
2023-06-01T14:49:25.559Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       700/1000 processing epoch:2910117, nrRowsAffected:33994
2023-06-01T14:49:26.681Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       800/1000 processing epoch:2910017, nrRowsAffected:47970
2023-06-01T14:49:27.919Z        INFO    lotus-shed      lotus-shed/msgindex.go:95       900/1000 processing epoch:2909917, nrRowsAffected:61662
2023-06-01T14:49:29.150Z        INFO    lotus-shed      lotus-shed/msgindex.go:128      Done backfilling, nrRowsAffected:76395

real    0m9,854s
user    0m5,389s
sys     0m1,132s

@fridrik01 fridrik01 marked this pull request as ready for review June 1, 2023 15:01
@fridrik01 fridrik01 requested a review from a team as a code owner June 1, 2023 15:01
@jennijuju jennijuju linked an issue Jun 1, 2023 that may be closed by this pull request
Copy link
Contributor

@arajasek arajasek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@arajasek arajasek merged commit cc66654 into master Jun 1, 2023
@arajasek arajasek deleted the fix-msgindex-backfill branch June 1, 2023 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants