Supporting IBC Casting

[michaelfig]

Motivation

The goal of IBC Casting is to support the model of:

• a single publishing IBC host
• many following clients, with no required submission of transactions to a chain
• a single Merkle proof for a given "stream head" is sufficient to link together the entire history of published packets

Implementations

• Agoric's Golang Casting streamer at https://github.com/Agoric/agoric-sdk/blob/mfig-vstream/golang/cosmos/x/vstream/spec/01_concepts.md#concepts (PR at Cosmos x/vstream implementation Agoric/agoric-sdk#5466)
• Agoric's (off-chain) Javascript Casting follower at https://github.com/Agoric/agoric-sdk/tree/master/packages/casting/README.md

IBC PubSub Discussion History

This is the unchanged initial post made by @michaelfig introducing IBC PubSub

I had an idea for a new IBC-based feature grounded in actual use cases on the Agoric chain.

Could we make a new connection primitive (other than the bidirectional "channel") for publish/subscribe applications? The API I envision is for a source host* to be able to publish messages on a given topic, and another host/client** to subscribe to that topic.
Unlike IBC channels, a pubsub would be unidirectional and not have any means to send data from the subscriber to the publisher (not even to handshake). Here are some loose requirements I'm thinking of:

* IBC host = chain or solo.
** including relayerless client libraries like CosmJS.

1. Topics need to have well-known names, and the namespace may be much larger than the current port IDs.
2. Publishers open a descriptor for a single topic per host, and can publish messages at any time.
3. Only the messages for the most recent published block needs to be in the current chain state. This is to eliminate the need for acks to clear old messages. If message history is desired, it can be obtained via historical queries.
4. Subscriptions name a connection and topic and may be multihop. This is not a shared bus, it is messages pulled by relayers from a single publisher to a specific subscriber.
5. An individual subscriber can open a subscription requester for a given publisher's IBC connection, and use it to create requests for relayers to fulfill with the last published message on given topic after given publisher block height. Maybe return all the messages in that block? Maybe separately?
6. Relayer incentivisation would be done as part of the subscription request, once the relayer can demonstrate that the relayed message satisfies the request.

I believe the provable data needed to store on the publishing side is per-host, per-topic, and consists of information about the last block that published something:

• block height
• array of messages (each containing base64 data) published in that block
• block height of prior publishing block (to be able to walk backwards through history)

Thoughts?

Tagging: @dtribble, @erights, @jackzampolin, @JimLarson, @warner, @zmanian for moral support.

[michaelfig]

Two more folks: @AdityaSripal, @dckc.

[zmanian]

Seems related to IBC queries.
#735

[michaelfig]

I agree that it is related, to IBC queries. Let me explain the differences:

1. Pubsub has a predicate to allow subscription to a future value (using events as gossip to announce an update) but queries are polling.
2. Pubsub handles streamed packets and preserved history where queries are based on single values.

I could implement pubsub as an application on top of queries, but only if not polling, and if the pubsub application could provide predicates to filter the acceptable query answers.

For this reason, I see pubsub as closer to a connection protocol than to a polling one-shot query.

@zmanian do you personally think it makes sense to add supporting features to ibc query?

[zmanian]

I think queries just need to be able to support a stream of values emitted over time rather than just a single value and then we can fuse both .

The infrastructure of pubsub and queries should be identical

[michaelfig]

I've figured out two orthogonal but composable features for Interchain Queries that would enable IBC PubSub to be built as an application:
- add a "different-than V" query limiter that requires the result not to match V in order to satisfy the query
- add an "after-height H" query limiter that can only satisfy the query with a result in some block after blockheight H
- compose them to get "different-than V AND after-height H"

Implementation is opt-in for relayers. Worst case, a relayer would poll periodically to see if new data arrives. If the source chain can raise on-change events for query results, that can help a relayer or other off-chain follower to satisfy the query more promptly/efficiently.

Here's a suggestion for change events:

• ibcpub.path=<store>:<key> the store and key's data has changed
• ibcpub.data=<base64-encoding, or 'null'> the new contents of the path

The ibcpub.data event is optional, and can be used as gossip for trusting clients (like simple event processors) to obtain the data quickly without having to issue a separate proved query.

[michaelfig]

I'm working on a PR at Agoric/agoric-sdk#5466 ! You're welcome to join.

[ethanfrey]

Interesting idea. What I would like to ask is the delivery semantics, which is one of the trickiest details of any messaging bus.

IBC channels guarantee "at most once" delivery. But also work hard to ensure that a response or timeout will eventually arrive on the sender, if the channel/connection remains alive (and push any liveness requirements to that lower level).

What about pubsub? If I am a subscriber, can I assume that I will get all messages published after my subscription is registered on the remote chain, until their is a liveness issue on the connection? If there are messages [A, B, C, D] published, and a subscriber connects and receives B as the first message - will it ever see A? Is it possible for it to receive only [B, D] and not C? Are there any ordering guarantees?

These are very important semantics to define before implementing anything. I am not sure what the right choices are to support the most applications, but for me allowing [B, D] to ever be valid seems like it opens up a lot of potential to abuse by a malicious relayer

[ethanfrey]

Nice link, I tried another link at the top (there were 3) and didn't find it nearly as illuminating.

Those still describe protocols and how to implement those. I wonder if we need to expose this all to the app level. Especially dealing with liveness.

Also, while I can see uses for the first 3, I cannot see a clear use for reverse iteration over every state change. It would be nice to explain sample apps there.

[AdityaSripal]

@michaelfig Seems to me like the proposal I had above would cover the use cases in your document, modulo minor changes?

I think the key property I'm proposing to enforce is that relayers are not allowed to skip values in the stream.

[michaelfig]

You are correct: your proposal is good for understanding the concepts relevant to IBC clients and relayers. However, I would encourage you to take a look at [the latest stream.proto file for my prototype implementation](. It is my attempt to bring the general casting feature to production, not just for IBC relayers.

It accommodates a few key features that your proposal currently lacks:

• termination (including error conditions)
• forking of a stream from a position in an earlier stream
• multiple published values per block

I understand that the IBC-specific design is important to you. The publisher/subscriber feature set is critical to me (and indeed, the non-IBC case is the initial motivation for the Agoric feature as a whole), and I added IBC support as an afterthought.

I'd like to see a proposed ICS standard with the relayer simplicity of your proposal merged with features of mine.

[michaelfig]

I forgot to attach the link to current prototype stream.proto: https://github.com/Agoric/agoric-sdk/blob/mfig-vstream/golang/cosmos/proto/agoric/types/stream.proto

[AdityaSripal]

Thank you! Attaching the streams.proto is indeed very helpful.

I think ideally we'd want to build the simplest version of queries such that your streams protocol can be built on top of it, even if we don't implement streams directly into the IBC part itself.

So given that IBC queries is data agnostic, the only thing that seems necessary to me to build into the IBC queries protocol is the the following feature: get next different value

The features you mention that is needed for streams:

termination (including error conditions)
multiple published values per block
forking of a stream from a position in an earlier stream

These all seem like they can be built at the application level above queries. If IBC queries is just handling raw bytes stored in a particular key. Then it is up to the query caller and storer to decide on a mutually intelligible format.

So the mutually intelligible format could well be a StreamCell, that stores a pointer to the previous position, multiple values, and the rest. The streams application can then just use IBC queries, to get the next change in the key-value, decode to a StreamCell and interpret the change as being multiple updates or a termination or whatever other stream-specific update.

If you agree that we can build Streams and everything you need at the application layer sitting above IBC queries (so long as we have get next value), then I would be in favor of getting that into the current queries spec.

My goal is to make sure that we can support the widest range of use cases with the simplest approach. Let me know if this works for you, and we can get started on it! cc: @angbrav

[AdityaSripal]

Thanks for the idea @michaelfig! Agreed with @zmanian that we should try to merge efforts here with Interchain Queries. A couple questions for my understanding.

1. Is it intended that the Publication by storing the KV pair in the IBC store or just by emitting an event whenever a particular key in the general state gets updated?
2. Is it the intention that the subscribing chain gets every single update to the value?
3. How would negotiation on data format work? Would it be implicit like current interchain queries.
4. Building on @ethanfrey 's comment, how does the subscribing chain respond if it receives update [B, D] but knows that there exists an update C in the middle which it has not yet received?

Unlike current Interchain queries, it should be noted that this will require changes to be implemented on the Publishing (Queried) chain since they must also store the height at which value updates as well as height of previous update.

Note that we can build a unidirectional PubSub on top of the bidirectional channel abstraction. We do something vey similar for ICA, where messages flow in one direction

[michaelfig]

Thanks for the idea @michaelfig! Agreed with @zmanian that we should try to merge efforts here with Interchain Queries. A couple questions for my understanding.

BTW, I'm rebranding this as "streams". Please see https://github.com/Agoric/agoric-sdk/blob/mfig-vstream/golang/cosmos/x/vstream/spec/01_concepts.md#concepts

1. Is it intended that the Publication by storing the KV pair in the IBC store or just by emitting an event whenever a particular key in the general state gets updated?

Storage is expected.

2. Is it the intention that the subscribing chain gets every single update to the value?

If it desires. For some streams, it may be desirable to skip to the latest value when possible.

3. How would negotiation on data format work? Would it be implicit like current interchain queries.

Yes, it would be implicit.

4. Building on @ethanfrey 's comment, how does the subscribing chain respond if it receives update [B, D] but knows that there exists an update C in the middle which it has not yet received?

It issues an interchain query for that specific update.

Unlike current Interchain queries, it should be noted that this will require changes to be implemented on the Publishing (Queried) chain since they must also store the height at which value updates as well as height of previous update.

Yes, it is intended as a layer on top of queries, not a replacement.

Note that we can build a unidirectional PubSub on top of the bidirectional channel abstraction. We do something vey similar for ICA, where messages flow in one direction

That doesn't work for me, since I want there to be no data flow (handshake or acks) from the client to the Publishing chain.

[jackzampolin]

This is very cool, will see how we can merge this with any ibc query efforts

[michaelfig]

I've updated the concepts document linked from the description to feature use cases prominently.

[michaelfig]

Inviting @albertchon to the conversation!