Expose chain-side balances for non-transactional reading

[warner]

What is the Problem Being Solved?

Currently, the only way to query a Purse balance is to send a getCurrentAmount message to the Purse, which requires an expensive chain transaction, and takes at least a full block time (5s), more if the chain is busy.

One way to improve this would be to publish all balances (or at least the subset that external followers are interested in) to the cosmos state vector, under some pre-arranged key. This would allow them to be queried by RPC. In particular, the RPC caller could get a copy of the Merkle proof from the block root to the balance, allowing the results to be safely published by a third party, and then verified by any client who fetches it. This would let us amortize the cost of the RPC queries.

Description of the Design

This should use the same basic approach as #4639, but might involve another level of indirection in which what gets published to the state vector is information that can be used to get the data from somewhere else rather than the data itself (see also #4331) and therefore minimize the volume of cosmos writes.

Security Considerations

The balances are public information, since they're running on a transparent chain. The handle/suffix used to identify the rendezvous point within the state vector is unique but not secret. The portion of the cosmos state vector in which balances are stored must be exclusively owned by the notification manager, otherwise other parties could write out an incorrect balance.

Test Plan

Unit tests within cosmic-swingset.

cc @erights @katelynsills

[erights]

How does a particular notifier come to be associated with a particular suffix? Where/what is the right to update state associated with that suffix? With this being driven by notifiers (or presumably subscriptions too?), none of this should be specific to purses or balances, right? It should apply to any notifier/subscription provided state, given that the state itself is simply-stringifiable data?

[katelynsills]

A nitpick with this part:

Currently, the only way to query a Purse balance is to send a getCurrentAmount message to the Purse, which requires an expensive chain transaction, and takes at least a full block time (5s), more if the chain is busy.

Purse notifiers already exist, and will give the purse balance. (In hindsight, we should change the loadgen code to use them.) So I think your plan to connect notifiers generally to the cosmos state vector is the right way to do it. It would be good for the ERTP/Zoe/contract code layer to be incentivized to use notifiers.

[erights]

When y'all say "notifier" do you mean "notifier/subscription"? Or does what you're saying not apply to subscriptions, only notifiers? Or just that you aren't as familiar with subscriptions?

[katelynsills]

I'm just not familiar with subscriptions :)

[warner]

When @dtribble pointed out that the client side of this should have the same API as a regular notifier, I realized my approach wouldn't work, or at least it's only half of the solution. In today's kernel meeting, @michaelfig and I walked through a possible approach to fix that.

The key issue is that we're adding an alternative communication pathway, in parallel to the existing chain-to-solo scheme (wait for a new block, read the outbound mailbox from the state vector, look for new messages, hand them to the mailbox device), but we want to wind up with an in-band Notifier-like object to which a vat in the wallet can subscribe. Something has to cross the levels, and work with things at both the swingset object level and the comms messaging/remote level.

We're looking to replace the chain-to-solo communication pathway with IBC, and part of that means putting the outbound messages in "Events" rather than saving them in the merkleized state hash. Events aren't hashed yet, but @michaelfig says that's in the works. Chain nodes aren't required to retain Events forever, but IBC messages can't be safely retrieved more than three weeks later anyways (unbonding period vs evidence submission deadline), so that might not be such an issue.

When a solo node is "receiving" messages from a chain, it will really be subscribing to receive specially-tagged outbox Events from the chain. The solo node will pick an RPC node, tell it about the event filter it wants to use, set up some sort of websocket-ish realtime notification mechanism, then ask for all the Events that match the filter (to catch up). Each event needs to include the full contents of the outbox, maybe, depending on how event retention works.

Now, things like balance announcements can also be emitted as events, and the same subscriber/follower thing can look/query for those events too. The tricky bit is how the receiving code should realize that there's a new event filter it needs to look for.

The design sketch we worked out looks like this:

• the comms vat is given access to a "chain event stream helper vat"
• the helper vat also gets to talk to vattp and/or the chain-following device (basically the mailbox, but probably more aware of chains than usual)
• we introduce a new reference type into the remote protocol, previously it was ro+-NN and rp+-NN, this would add maybe reXYZ for "remote event (stream)`
  • the XYZ is a chain-scoped unique key used to filter the event stream. One is allocated to each e.g. Notifier that's trying to use this non-message-based pathway
• when the comms vat receives a remote message that cites an reXYZ reference, it talks to the helper vat
  • "hi helper vat, please create an object for me that behaves like a notifier, using XYZ from Remote R"
  • "R" is either the remote name, or the "transmitter" object hosted in vat-vattp which comms uses when sending message to this remote
  • the helper vat talks to vat-vattp and says "please start following events with filter XYZ, and send them to me"
  • the helper vat creates a Notifier-like object, and routes the event updates to it's updating facet
• comms remembers the mapping from reXYZ to the helper vat's object reference, and translates any references it encounters in the future
• the downstream user-level vat receives the pseudo-Notifier object reference, and subscribes to it normally
• vat-vattp or the mailbox device or whatever adds XYZ to the filter predicate for the same chain follower as the receiver object was using

The idea is to avoid writing a lot of new code in comms (to implement a Notifier), but still have it be the main place that crosses levels.

[erights]

Between this and #3784 (comment) , it is worth pointing out the very different metering and load properties of notifiers vs subscriptions. Each new notifier client puts more load on the notifier source, as they get getUpdateSince messages from each client. Not a lot, but it is multiplied by the number of clients. The subscription pattern is purely through a shared promise stream that the clients only read or then. The source only resolves promises, magically multicasting to all clients with no per-client operations.

[warner]

@gibson042 will be driving this on the swingset side, with help from @JimLarson on the cosmic-swingset/golang parts.

qv the 08-dec-2021 kernel meeting, we're currently thinking:

• the chain-side vat continues to use a normal Notifier
  • but there's a new special device that knows how to publish changes to the cosmos Event/Results stream
    • TODO: figure out what the current state of Cosmos Event/Results are: we need something that's hashed into the state vector, so clients can rely upon it, and we need something with decent availability (so whoever the client is asking will keep the data around long enough)
  • we provide a chain-side library or service or vat or something to which the Purse-holding vat can submit the Notifier and get a wrapper, or we define a second kind of Purse that does this automatically
    • (from a least-authority point of view, we should probably wrap the notifier, not the purse)
  • these wrappers/new-notifier have an extra method like "tell me your notifier ID" (the index at which the Event/Results balances will be published)
  • we only do this for some Purses (e.g. the ones added to Wallets), not all of them, to avoid wasting bandwidth/effort on e.g. Zoe's short-lived escrow purses
• on the client side, we provide some new library or service or vat or something
  • the wallet code does library.getBetterNotifier.getNotifier()), or library.getNotifier(purse)
  • the return value has the normal Notifier API
  • internally the client-side library/thing knows how to talk to the chain-side library/thing to discover an Event/Results ID for the purse/notifier
  • the client-side library/thing has access to a client-side device which knows how to instruct its chain follower code to track that ID and get results/events

The net result is that client-side code could do the lots-of-messages Notifier protocol, but doesn't, because the fancier Notifier returned by the client-side library knows how to do it better. Developers should only need to know about two things:

• on the chain side, add librarything.registerNotifier(purse.getNotifier()) to allocate an index and prepare the publishing mechanism to follow the purse
• on the solo/wallet side, add notifier = librarything.getNotifier(purse.getNotifier()) to obtain the more-efficient Event/Result-based notifier client

This points to having a chain-side service, probably in its own vat, which provides both registerNotifier to chain-side vats, and some sort of getIDForNotifier to client-side requests. This chain-side vat should have access to a device that allows it to add data to the Event/Results. Then the service could have a simple WeakMap from original Notifier to ID. (note that vats can't discover that an object has been deleted/GCed, so we wouldn't have any way to reclaim the ID once the Notifier disappears, but I think that's fine).

The corresponding client-side service (again in its own vat) that provides getNotifier should have access to the chain-side service so it can call getIDForNotifier, and to a local device that allows it to ask the chain follower to watch for Event/Results for that ID.

@dtribble pointed out that the notifier's ID would be a great candidate for putting in auxdata (#2069), when we get that implemented. That would remove the need for an extra round-trip to call getNotifier, however it would require that the Purse create the augmented Notifier right off the bat, and return it directly from purse.getNotifier, instead of assuming some later lookup will happen.

[erights]

the chain-side vat continues to use a normal Notifier

Not a Notifier. A Subscription.

[michaelfig]

the chain-side vat continues to use a normal Notifier
Not a Notifier. A Subscription.

These objects are lossy, in that the chain node may not leave every state transition accessible to the client, only the latest one. So I think they're notifiers.

[warner]

@michaelfig @gibson042 and I talked about this (in separate conversations, alas) over the last few days, so here's an update:

On-Chain Wallet Agent can be Special Purpose

The primary use case for this is an on-chain wallet agent, which holds each user's Purses, subscribes to notifiers (ok ok subscriptions?) to hear about balances changing and new Payments arriving, etc. We want the user's off-chain wallet (which isn't a real vat, and isn't participating in object messages) to be able to learn about these events, and get data about them (like "of what brand is this new Payment?" and "what is the Payment's balance?") without performing expensive chain transactions.

A secondary use case, which we'll probably tackle later, is for non-wallet / non-user-specific data sources like the most recent trading price of an AMM.

The on-chain wallet will get the ability to publish data into the Cosmos IAVL tree within a certain sub-hierarchy of the keyspace, maybe swingset/wallet/*. It will also get the ability to add tendermint Events into the Results of a block, with a particular key or topic string that indicates swingset.wallet.something . These abilities will involve the Bridge device/vat, and some small amount of new code on the Golang side to react to certain kinds of bridge messages by making cosmos-sdk API calls to change the IAVL tree and/or add Events. (We already have the bridge device, and we already have some Go-side code to write to storage, so a lot of the job is already done, but I think it doesn't do Events yet).

We'll add an instance of marshal to the on-chain wallet, parameterized with a convertValToSlot() that translates Presences into a new woNNN slot type. This slot type is entirely a userspace thing: nothing in the kernel or liveslots will know about it, or ever see it. This convertValToSlot() will reject Promises entirely, and convertSlotToVal side will only accept woNNN slots that were allocated by an earlier serialization. (by analogy, if this were liveslots, it would allow exported Remotables but not imported Presences). There should be a separate table for each user.

When the on-chain wallet agent sees a user Purse, it will automatically subscribe to hear about balance changes. When these updates arrive, it will use the bridge device channel to update the IAVL tree at some user-specific key, and publish an Event with the same information.

The on-chain wallet also subscribes to something to hear about new Payments arriving for each user (-ish? I don't really know the details). When it learns about a new Payment, instead of merely publishing "you've received Payment wo123", it does a bunch of immediate queries to gather all the information that the off-chain wallet needs to know, like the Brand and the amount. After those queries have completed, then it publishes a complete record to the IAVL tree and Event. This lets the off-chain wallet receive one record and have everything it needs to update the UI, without spending more time doing additional queries. The off-chain wallet can do RPC queries but can't cause any object messages to be sent, so it depends upon the on-chain agent to do all the gathering first.

These woNNN identifiers (call them "wrefs", to parallel vat-side "vref" and kernel-side "kref" and comms/remote-side "rrefs") serve two purposes. First, they can be correlated between multiple published records or Events for a single user. One IAVL entry might be IAVL['swingset/wallet/$USERID/purses'] = '["wo4","wo5"]' (or the capdata equivalent, with body: and slots:), to point to a list of purses, while IAVL['swingset/wallet/$USERID/purses/wo4'] could hold the balance and brand, or maybe swingset/wallet/$USERID/wo4, or swingset/wallet/$USERID/objects/wo4. The off-chain wallet knows the convention, and can do a second RPC query if necessary to get the details.

The second purpose is that a suitably-empowered request can use them to get back to the Purse/Payment objects, like the on-chain wallet handler reacting to a signed "accept offer" message from the off-chain wallet (#3908, I think). The same marshal instance's unserialize could be called by a handler that has verified the right signature appeared on the request, and use it to identify the specific Payment being deposited, or the Purse from which an Offer's payments should be withdrawn. The off-chain client making these signed requests is not submitting object messages: they aren't speaking vattp, they aren't vats at all (they aren't persistent in the way a vat would need to be), and they aren't even allowed to decide the exact behavior they're invoking. The signed message can choose from a menu of pre-written actions, and the woNN values are the parameters of those actions. This way, the off-chain wallet can prepare a request, but it's the Ledger or other hardware signing token that can interpret the request for display to the user before he/she/they confirm the request and approve a signature.

So the on-chain wallet code is serializing and unserializing things, but those "things" are not raw object messages, so it can apply special-purpose code before and after.

This (Presence<->wref) table is similar in purpose to the vat/kernel vref<->kref c-list, or the comms vref<->wref per-remote c-list, in that it is mapping identities from one domain to another. But the wref table lives in userspace, and emits messages that are not interpreted by the kernel or comms (they travel into the IAVL tree and Events in blocks), where the off-chain client can read them with RPC queries. This data pathway is mostly one-way: there are no acks, no GC of dropped identifiers. Just updates about data changing. The off-chain wallet code knows the structure of the published data, so a lot of fields might be omitted or implicit (e.g. the balance of a Purse might be able to include only the token count, not the brand, if the off-chain code learns about the Brand ahead of time).

This on-chain wallet agent that's allocating wrefs might collaborate with the Board to find board IDs for some of the things it is translating. (This would be easier if the Board had a "just lookup, no insert" API). convertValToSlot could ask whether the object is already on the board, and if so it could use bo$BOARDID instead of allocating woNN. This would give the off-chain wallet more information to correlate with. This might integrate with user-provided petnames somehow.

Other Notifiers, More General-Purpose Translators

The on-chain wallet is clearly a user agent, and it makes a lot of sense to have this agent do additional data-gathering and translation work for the benefit of the off-chain user wallet.

The second use case, AMM balance announcements, is not scoped to a particular user. It's not clear that it would be appropriate to add code to the AMM contract to convert price updates into IAVL/Event publishing. The contract's responsibility should end with a publically-visible Notifier, that it updates after each trade.

But we can imagine a somewhat-generic service that takes a Notifier, starts watching it, and publishes each change to the IAVL tree (and makes an Event) in some well-known location. The index of this location is the question: earlier proposals add a method to the Notifier to fetch this index; this proposal would involve a service which allocates the index, publishes the updates, and can be queried (with the Notifier as an argument) to learn the index.

@michaelfig hinted that we might choose to integrate this with the Board. In this approach, every time you register a Notifier with the board (maybe with a distinct registration method), the Board itself subscribes, and updates a portion of the IAVL tree (/swingset/board/$BOARDID) with a boNNN/"bref" serialization of the new value. Any on-chain object with access to the AMM's price/trade Notifier and the Board (and the willingness to pay a fee?) could publish it, after which anyone off-chain could RPC-query /swingset/board/$BOARDID and pipe into |jq .amount to extract the price number, or something. Anyone with object access to the AMM's public facet could ask it for the Notifier, then ask the Board for its ID, to learn what the RPC query path should be, or to confirm that an alleged path is the right one.

And we might incorporate this registration into the standard AMM contract, so in addition to the normal public trading API, there would be a method like getBoardID, from which the RPC query path could be derived. The contract wouldn't know the details of how data gets published, but it knows that it's supposed to register useful Notifiers with the Board, and tell users about the resulting ID, after which all it needs to do is update the Notifier. On-chain objects can subscribe to the Notifier directly, and off-chain users can follow with RPC queries.

This approach gets us back to the more generic service end of the spectrum.

Mostly User-Space

As a kernel person, the biggest thing I like about this approach is that it doesn't involve a lot of kernel changes :). We need updates in cosmic-swingset to enable writing into the IAVL state vector and adding Events, but the rest of the changes are in userspace: the on-chain wallet vat, the AMM contract, maybe the Board vat. The off-chain wallet code needs to know how to make RPC queries, how to deserialize/interpret the results, and it needs to coordinate the format of that data with the on-chain publishers. But there aren't any kernel or comms changes that need to happen, and the client side doesn't use swingset at all. We aren't trying to let an object in ag-solo get Notifier updates cheaply, instead we're letting a non-vat off-chain wallet (web) application perform RPC/WebSocket queries to learn about changes to certain pieces of data. That removes a lot of the complexity from before.

[erights]

subscribes to notifiers (ok ok subscriptions?)

We should talk about this. @dtribble convinced me that for the cases we're probably concerned with, recent states rather than all events, the semantics we want is notifier-like is some ways. But it is still subscription-like in other ways.

It is notifier-like in that multiple state changes can be collapsed into the most recent one at the sending side, for example, because multiple updates happened with one turn or crank or block. However, it is subscription-like in that what information is published depends only on the sender. There is no adaptation to the frequency with which the receiver is interested. Only this lets a receiver receive without a transaction.

[Tartuffo]

Given we have bridge device, we don't need further Swingset changes.

[Tartuffo]

@warner please work with @gibson042 on adding this to the overall epic for this, and refining the estimate.

[Tartuffo]

@gibson042 The title of this ticket should be changed to reflect the issue here is to build the framework exposing info via RPC and notification, not to be specifically about chain-side balances. I've created the epic #4639 to track all the places this should be used. FYI @warner

[Tartuffo]

@gibson042 please rename this issue with a more accurate title.

[Tartuffo]

Need this for on-chain Wallet.

[dckc]

dup of #5356