Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swing-store import is slow for mainnet #8521

Closed
mhofman opened this issue Nov 9, 2023 · 0 comments · Fixed by #8522
Closed

Swing-store import is slow for mainnet #8521

mhofman opened this issue Nov 9, 2023 · 0 comments · Fixed by #8522
Assignees
Labels
bug Something isn't working swing-store

Comments

@mhofman
Copy link
Member

mhofman commented Nov 9, 2023

Describe the bug

Restoring from state-sync one mainnet (with a patch for #8325 applied), is very slow (more than 30 minutes), and uses more disk than expected (double). It turns out the culprit is the WAL mode which doesn't like an almost 40GB transaction.

To Reproduce

import-kernel-db with a mainnet state export.

Expected behavior

Data directory not using more than the target size and faster than current 30 minutes.

Screenshots

2023-11-09T17:16:08.076460335Z Restoring SwingSet state from snapshot at block height 12416000 with options {"exportDir":"/tmp/agd-swing-store-restore-12416000-702237452","artifactMode":"replay","exportDataMode":"all"}
2023-11-09T17:49:58.197655501Z 5:49PM INF restored swing-store export exportDir=/tmp/agd-swing-store-restore-12416000-702237452 height=12416000 module=x/swingset submodule=SwingStoreExportsHandler
$ ls -la
total 77078156
drwxr-xr-x 2 benchmark benchmark          89 Nov  9 17:16 .
drwx------ 9 benchmark benchmark         167 Nov  9 17:16 ..
-rw-r--r-- 1 benchmark benchmark 37754957824 Nov  9 17:49 swingstore.sqlite
-rw-r--r-- 1 benchmark benchmark    76513280 Nov  9 17:43 swingstore.sqlite-shm
-rw-r--r-- 1 benchmark benchmark 39391472472 Nov  9 17:42 swingstore.sqlite-wal
@mhofman mhofman added bug Something isn't working swing-store labels Nov 9, 2023
@mhofman mhofman self-assigned this Nov 9, 2023
@mergify mergify bot closed this as completed in #8522 Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working swing-store
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant