feat(swing-store): faster import of swing-store #8522
Conversation
| @@ -194,6 +194,21 @@ export function makeSwingStore(dirPath, forceReset, options = {}) { | |||
| // { verbose: console.log }, | |||
| ); | |||
|
|
|||
| function setUnsafeFastMode(enabled) { | |||
There was a problem hiding this comment.
There's probably a better name for this, but the functionality seems good to me.
packages/swing-store/src/internal.js
Outdated
| @@ -9,6 +9,7 @@ import { Fail, q } from '@agoric/assert'; | |||
| * transcriptStore: TranscriptStoreInternal, | |||
| * snapStore: SnapStoreInternal, | |||
| * bundleStore: BundleStoreInternal, | |||
| * setUnsafeFastMode: (enabled: boolean) => void, | |||
There was a problem hiding this comment.
This extension makes yarn lint complain about the first argument to assertComplete no longer satisfying type SwingStoreInternal at
agoric-sdk/packages/swing-store/src/exporter.js
Lines 111 to 112 in c368273
Maybe modification of SQLite modes should just remain internal to makeSwingStore?
There was a problem hiding this comment.
Looks good, I agree with @gibson042 about the API maybe being better but I think this is clear enough for now. Approved but see if you can add the two comments below.
| @@ -106,7 +109,10 @@ test('state-sync reload', async t => { | |||
| getArtifact: name => artifacts.get(name), | |||
| close: () => 0, | |||
| }; | |||
| const ss2 = await importSwingStore(datasetExporter); | |||
| const ssi = await importSwingStore(datasetExporter, importDbDir); | |||
There was a problem hiding this comment.
This means the SwingStore you get back from importSwingStore is no longer suitable for general use, because it's got all the commit-safety modes turned off, yeah?
Please update the docs in docs/data-export.md to mention this fact, around line 170-ish where the importSwingSstore is shown.
There was a problem hiding this comment.
Correct, it is not. I have now removed the kernelStorage facet from the return value of importSwingStore.
mhofman
force-pushed
the
mhofman/import-swing-store-fast
branch
from
c953c85 to
5401d58
Compare
Disable journaling and full synchronous pragma modes during import
mhofman
force-pushed
the
mhofman/import-swing-store-fast
branch
from
5401d58 to
6a833eb
Compare
feat(swing-store): faster import of swing-store
|
|
||
| So, to avoid pruning current-incarnation historical transcript spans when exporting from one swingstore to another, you must set (or avoid overriding) the following options along the way: | ||
|
|
||
| * the original swingstore must not be opened with `{ keepTranscripts: false }`, otherwise the old spans will be pruned immediately | ||
| * the export must use `makeSwingStoreExporter(dirpath, { artifactMode: 'replay'})`, otherwise the export will omit the old spans | ||
| * the import must use `importSwingStore(exporter, dirPath, { artifactMode: 'replay'})`, otherwise the import will ignore the old spans | ||
| * the `importSwingStore` call (and all subsequent `openSwingStore` calls) must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`). | ||
| * subsequent `openSwingStore` calls must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`). |
There was a problem hiding this comment.
retrospective nit: it might be a good idea to retain the admonition against having keepTranscripts: false in the importSwingStore options bag (maybe as a child-bullet of the previous line).. that line was mainly about the importance of including artifactMode: 'replay', but it's also important to not prune transcripts during the import itself
| @@ -11,14 +11,16 @@ import { assertComplete } from './assertComplete.js'; | |||
| */ | |||
|
|
|||
| /** | |||
| * Function used to create a new swingStore from an object implementing the | |||
| * Function used to populate a swingStore from an object implementing the | |||
There was a problem hiding this comment.
nit: It really is about creating a new swingStore. Changing it to say "populate a swingStore" kind of makes it sound like this could be used to fill a previously-created (perhaps empty, perhaps not) database, which suggests some sort of weird merge operation between the previous contents and the import dataset.
closes: #8521
Description
Disable journaling and full synchronous pragma modes during import, resulting in an import time for mainnet state of 9 minutes vs 33 minutes.
Adds a check that the journaling pragma has been successfully applied to prevent running in unsafe conditions.
To avoid confusions and mis-usage,
importSwingStoreno longer returns thekernelStoragefacet of the SwingStore, to make it clear it cannot be used for execution.Security Considerations
This change could technically result in a corrupted DB if interrupted, however the import process in cosmic-swingset writes the expected block height just before commiting. If that block height is not correct on start, the node will refuse to start thanks to hangover checks.
Scaling Considerations
Cuts disk usage in half during the swing-store import
Documentation Considerations
None
Testing Considerations
Manually tested on a patched follower, and modified the state-sync test to reflect the open close of connection.
Upgrade Considerations
None