Skip to content

Commit

Permalink
fix: start swing-store export without blocking checks (#9423)
Browse files Browse the repository at this point in the history
closes: #9418

## Description

This moves the completeness check from being eager when creating the
exporter to delayed when yielding the artifact names list.
It also updates the kernel export logic to yield to the event loop after
triggering the started event and before initiating the actual export, so
that any IPC messages have a chance to be sent.

### Security Considerations

None

### Scaling Considerations

The consumer will no longer get an early error if the DB is not suitable
for export. In practice that doesn't change anything since the
state-sync logic was already not in a position to handle such an early
error.

However the state-sync export relied on the "started" event to have
triggered by the time the next block commit was reached. The
completeness check here was happening before that point, and could take
a significant amount of time as described in #9418.

### Documentation Considerations

None

### Testing Considerations

Manually verified, and verifying with a patched upgrade-15 image on
devnet instagoric.
Updated tests to reflect that making the exporter does not immediately
fails if incomplete.

`export-kernel-db.js` is not sufficiently covered by test to automate
testing of the `started` message.

### Upgrade Considerations

None
  • Loading branch information
mergify[bot] committed May 31, 2024
2 parents df4954e + 3c94159 commit 0561702
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 17 deletions.
4 changes: 4 additions & 0 deletions packages/cosmic-swingset/src/export-kernel-db.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import { fileURLToPath } from 'url';
import { makePromiseKit } from '@endo/promise-kit';
import { Fail, q } from '@agoric/assert';
import { makeShutdown } from '@agoric/internal/src/node/shutdown.js';
import { waitUntilQuiescent } from '@agoric/internal/src/lib-nodejs/waitUntilQuiescent.js';
import { makeSwingStoreExporter } from '@agoric/swing-store';

import { isEntrypoint } from './helpers/is-entrypoint.js';
Expand Down Expand Up @@ -192,6 +193,9 @@ export const initiateSwingStoreExport = (
abortIfStopped();
startedKit.resolve();
log?.(`Starting DB export at block height ${savedBlockHeight}`);
// Let the `started` event be sent before proceeding with blocking processing:
// `getArtifactNames` may block the agent for a while.
await waitUntilQuiescent();

/** @type {StateSyncManifest} */
const manifest = {
Expand Down
27 changes: 15 additions & 12 deletions packages/swing-store/src/exporter.js
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,6 @@ export function makeSwingStoreExporter(dirPath, options = {}) {
const bundleStore = makeBundleStore(db, ensureTxn);
const transcriptStore = makeTranscriptStore(db, ensureTxn, () => {});

if (artifactMode !== 'debug') {
// throw early if this DB will not be able to create all the desired artifacts
const internal = { snapStore, bundleStore, transcriptStore };
assertComplete(internal, artifactMode);
}

const sqlKVGet = db.prepare(`
SELECT value
FROM kvStore
Expand Down Expand Up @@ -168,16 +162,25 @@ export function makeSwingStoreExporter(dirPath, options = {}) {
}
harden(getExportData);

/**
* @returns {AsyncIterableIterator<string>}
* @yields {string}
*/
async function* getArtifactNames() {
/** @yields {string} */
async function* artifactNames() {
yield* snapStore.getArtifactNames(artifactMode);
yield* transcriptStore.getArtifactNames(artifactMode);
yield* bundleStore.getArtifactNames();
}
harden(getArtifactNames);
harden(artifactNames);

/**
* @returns {AsyncIterableIterator<string>}
*/
function getArtifactNames() {
if (artifactMode !== 'debug') {
// throw if this DB will not be able to yield all the desired artifacts
const internal = { snapStore, bundleStore, transcriptStore };
assertComplete(internal, artifactMode);
}
return artifactNames();
}

/**
* @param {string} name
Expand Down
11 changes: 6 additions & 5 deletions packages/swing-store/test/exportImport.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -220,15 +220,16 @@ async function testExportImport(
await ssOut.hostStorage.commit();
}

// the exporter may be broken (the swingstore may be incomplete), but that is
// signaled during `getArtifactNames()`, not during construction (which must
// finish quickly, without scanning the whole DB)
const exporter = makeSwingStoreExporter(dbDir, { artifactMode: exportMode });

const incomplete = 'incomplete archival transcript: 3 vs 12';
function doExport() {
return makeSwingStoreExporter(dbDir, { artifactMode: exportMode });
}
if (failureMode === 'export') {
await t.throws(doExport, { message: incomplete });
await t.throws(() => exporter.getArtifactNames(), { message: incomplete });
return;
}
const exporter = doExport();

const exportData = [];
for await (const elem of exporter.getExportData()) {
Expand Down

0 comments on commit 0561702

Please sign in to comment.