zoe will stop tracking contract vat termination when vat-zoe is upgraded

[warner]

As mentioned in #8263 (comment) , while investigating outstanding mainnet promises, we concluded that Zoe will fail to re-subscribe to contract-vat done() promises if/when vat-zoe is upgraded.

Zoe uses vatAdminSvc to create a new vat for each contract instance, which returns a per-vat adminNode that serves as a control facet. Zoe can use this to terminate the vat or trigger an upgrade. In addition, calling E(adminNode).done() will return a Promise that doesn't fire until the vat is terminated. The done() promise fulfills if the vat exits "happy" (if it self-terminates with vatPowers.exitVat()), and rejects if it exits "sad" (vatPowers.exitVatWithFailure(), adminNode.terminateWithFailure(), or the vat exceeds its meter or has some fatal error).

Zoe currently uses an ephemeral E.when() inside startInstance to attach a call to exitAllSeats to this promise:

agoric-sdk/packages/zoe/src/zoeService/startInstance.js

Lines 283 to 289 in 2ffcef2

	E.when(
	E(adminNode).done(),
	completion => {
	instanceAdmin.exitAllSeats(completion);
	},
	reason => instanceAdmin.failAllSeats(reason),
	);

which means that if/when vat-zoe is restarted or upgraded, the callback will be lost. The zoe vat will still be subscribed to the promise (the kernel will deliver a dispatch.notify), but the new zoe incarnation will not have any callbacks registered to fire, so it will ignore the resolution.

As a result, when vat-zoe is upgraded, it will stop tracking that done() promise. So if any contract vats terminate, zoe will not clean up after them. Depending upon the exit conditions on each offer, that might make it difficult to close out a seat.

One possible fix is for Zoe to use globalThis.VatData.providePromiseWatcher() to establish a durable watcher on the done Promise. That would remove the need to re-fetch a done promise when zoe is upgraded (however we would still need the isUpgradedDisconnection() check to guard against vat-vat-admin being upgraded). However, that won't help with the existing done() promises currently tracked by the deployed zoe.

Another fix, which would help with the current deployment, would be for Zoe's buildRootObject to scan its durable storage for the adminNodes of all active contract vats, and call done() on each of them, to re-establish the callbacks.

If we wanted to combine the two techniques, zoe's per-contract-vat table should have a flag to track whether we have a durable promise watcher registered or not. At startup (buildRootObject), walk the table, and for every vat that lacks the flag, use providePromiseWatcher(), and set the flag. In startInstance, when starting a new contract vat, use providePromiseWatcher() and also set the flag.

[Chris-Hibbert]

Another fix, which would help with the current deployment, would be for Zoe's buildRootObject to scan its durable storage for the adminNodes of all active contract vats

I don't think Zoe has any strong maps of instances. Can we add the ability for an exo class to scan its members?

The closest thing is instanceToInstanceAdmin in instanceAdminStorage, but it's a weak map.