FIP-2: Flexible targets for messages

[pfletcherhill]

FIP-2: Flexible targets for messages

Title: Flexible targets for messages
Type: Implementation FIP
Author: Paul Fletcher-Hill (gh: @pfletcherhill , fc: @pfh)

UPDATE 1: The original FIP was separated into two proposals: —FIP-1 with a canonical URI spec and FIP-2 (this one).

UPDATE 2: This top-level post has been significantly edited from its original form to reflect the most recent proposal.

Abstract

A proposal to make Casts and Reactions more flexible by allowing them to "reply" to arbitrary strings instead of Casts, and to allow Casts to embed CastIds for more efficient quoting.

Problem

Users can broadcast new content to their followers by creating casts and reactions. But users may want to have a conversation on the side about a particular topic or item without introducing it into their main feed, and delivering it only when others express interest in that topic. For example, a discussion around an NFT or a recent sports event that doesn't need to be sent to all their followers. A second issue is that users commonly want to embed other casts, but using the embed string type is very inefficient in the Hubs.

Specification

Casts and Reactions should allow a string value for parents and targets which applications can use to detect that a message is relevant only in a particular context. Applications can use schemas like FIP-1 to interpret these strings. Additionally, references to casts within casts should be made with CastIds over FIP-1 strings where possible since they take up less space and allow for indexing and enable efficient gRPC queries.

Cast and reaction protobufs should be upgraded to have embeds, parents, and targets that can be either strings or CastIds.

1. CastId support for cast embeds

Casts previously contained a string in the embeds field at position 1. Now, they will contain an Embed type in the embeds field at position 6, and the older string embed field at position one is renamed and will be deprecated.

// Current Protobuf
message CastAddBody {
  repeated string embeds = 1; 
  ...
}

// Proposed Protobuf
message CastAddBody {
  repeated string embeds_deprecated = 1; 
  ...
  repeated Embed embeds = 6;
}

message Embed {
  oneof embed {
    string url = 1;
    CastId cast_id = 2;
  }
}

2. String support for cast parents and reaction targets

Cast parent and reaction target types should be extended to allow strings. Similar to embeds, hubs will validate the length in bytes of these strings but not their contents. This is a backwards compatible change that will not affect older message types. However, some gRPC apis will need to be updated.

// Current
message CastAddBody {
  ...
  oneof parent {
    CastId parent_cast_id = 3;
  };
  ...
}

// Upgrade
message CastAddBody {
  ...
  oneof parent {
    CastId parent_cast_id = 3;
    string parent_url = 7;
  };
  ...
}

// Current
message ReactionBody {
  ...
  oneof target {
    CastId target_cast_id = 2;
  }
}

// Upgrade
message ReactionBody {
  ...
  oneof target {
    CastId target_cast_id = 2;
    string target_url = 3;
  }
}

GetReactionsByCast

A new GetReactionsByTarget method is added and the older GetReactionsByCast method is deprecated. The response type of both methods will be upgraded to ReactionsByTargetRequest which contains a target_cast_id and target_url as fields.

// Current
rpc GetReactionsByCast(ReactionsByCastRequest) returns (MessagesResponse);

message ReactionsByCastRequest {
  CastId cast_id = 1;
  ...
}

// Upgrade
rpc GetReactionsByCast(ReactionsByTargetRequest) returns (MessagesResponse); // To be deprecated as of v2023.5.31
rpc GetReactionsByTarget(ReactionsByTargetRequest) returns (MessagesResponse);

message ReactionsByTargetRequest {
  oneof target {
    CastId target_cast_id = 1;
    string target_url = 6;
  }
  ...
}

GetReaction

GetReaction accepts target_cast_id and target_url instead of cast_id. Backwards compatibility is preserved for this API since the field number for the cast id property remains the same.

rpc GetReaction(ReactionRequest) returns (Message);

// Current
message ReactionRequest {
  ...
  CastId cast_id = 3;
}

// Upgrade
message ReactionRequest {
  ...
  oneof target {
    CastId target_cast_id = 3;
    string target_url = 4;
  }
}

GetCastsByParent

This method now accepts parent_cast_id and parent_url, whereas it previously just accepted cast_id. Apps using the new client should upgrade to use parent_cast_id, though the field number is the same so old clients can still use the method without problem.

rpc GetCastsByParent(CastsByParentRequest) returns (MessagesResponse);

// Current
message CastsByParentRequest {
  CastId cast_id = 1;
  ...
}

// Upgrade
message CastsByParentRequest {
  oneof parent {
    CastId parent_cast_id = 1;
    string parent_url = 5;
  }
  ...
}

Rationale

Why do CastIds need to be embedded in Casts as a native type?

While CastIds can be serialized and embedded in Casts as a string, using the native type allows more efficient storage and indexing and lookups in the Hubs in the future. This is particularly important since "quote casts" are likely a commonly used feature.

Release

FIP-2 will be included in the 2023.4.19 protocol release and implemented in v1.2.0 of Hubble which will be released on 4/19/23. Client developers will need to make several changes and it is recommended that they upgrade to the latest version of @farcaster/hub-nodejs or @farcaster/hub-web as quickly as possible.

Reading from Hubs

1. Calls to GetReactionsByCast must be changed to GetReactionsByTarget before version 2023.5.23
2. Calls to GetReactionsByTarget should provide target_cast_id or target_url in the request.
3. Calls to GetCastsByParent should provide parent_cast_id or parent_url in the request.
4. Calls to GetReaction should provide target_cast_id or target_url in the request
5. Check both embeds_deprecated and embeds when reading CastAdd messages since older formats will be around for a year.
6. Expect to receive strings when reading reaction targets
7. Expect to receive strings when reading cast parents

Most gRPC changes are backwards compatible and older clients can still read new messages from new hubs. However, they will fail validation and without changes to look for the new properties, important data might be missed.

Cast Messages that use parent_cast_id and reaction messages that use target_cast_id will be valid in old and new hubs and will not be deprecated.

Writing to Hubs

Using the latest version of the @farcaster/hub-nodejs or @farcaster/hub-web will handle most cases, but there are a few things to be aware of:

1. Cast Messages created with timestamps after 5/3/23 00:00 UTC (Farcaster time 73612800) must not include embeds_deprecated or hey will be rejected by Hubs. The cast builder in hub-nodejs will handle this automatically.

2. Cast Messages created with the new embeds property or parent_url and Reactions created with target_url will fail validation on Hubs running protocol version 2023.3.1. Such messages should only be broadcast to hubs implementing 2023.4.19 and old hubs will shut down at 4/27/23 00:00 UTC.

Appendix: Target Examples

Examples of targets we want to embed, reply to, or react to:

• Youtube video URL (string): https://www.youtube.com/watch?v=B51bLBdUt3w
• Jam long cast URL (string): http://longcast.jam.so/jjIT58
• News articles (string): https://www.nytimes.com/interactive/2023/04/06/travel/things-to-do-tokyo.html
• Farcaster cast (cast ID): { fid: 2, hash: "0x2e6bc2ef8a08157144c571baeebe450e28bd7534" }
• Ethereum NFT (string): Etherscan or OpenSea
• Ethereum NFT minting transaction (string): https://etherscan.io/tx/0xfa2bead71e628b6fcbcb45530b1197b52092981860e653dc5285f79a27a0b92a
• Tezos NFT (string): https://objkt.com/asset/fxhashgenesis/145979
• Solana NFT (string): SolanaFM, Solscan, or Solana Explorer
• Arweave transaction (string): https://viewblock.io/arweave/tx/rLiQEhElEWQlzzSUuWCiFPF7d6Zj2QwX03jB5oTKL34

[varunsrin]

Supportive of this direction! A few extensions I'm proposing:

1. Explicitly encourage the use of canonical identifiers over URL identifiers

Consider this cast which includes the embed: https://warpcast.com/v/0x99b0d8. When this is rendered in a client, clicking on the link will take you to warpcast which is not a good ux. Each client should be able to render the cast in its own UI if possible and can fall back to warpcast if necessary. Using a canonical identifier like the Cast ID : is much more flexible. The same applies to using CAIP-19 NFT identifiers over OpenSea links for NFTs.

2. Create a lightweight, self-organizing process for developers to introduce new target formats / embed schemas

People will want to add new extensions as soon as this is out. If it evolves organically without structure it will be difficult for developers to integrate. We should make a lightweight system for proposing, validation and avoiding conflicts between target types:

1. A form for proposal (Github Discussions Channel?)
2. A naming system (e.g. FRC-1)
3. A ruleset for specs (e.g. must start with ://, must not collide with existing mod, must contain a regex to validate, must specify how to canonically render).
4. An approval process (e.g. merge PR somewhere)

We could write the first proposal for URLs and Ethereum NFT's and let the community propose more over time. There's no guarantee that a client supports them, but it provides an interoperable way for people to build asynchronously.

3. Turning CastId into a form of string embed.

Instead of a oneof of CastId or string, can we just have target format string schema called cast id? Example castid:<fid>/<hash> which hubs can parse by peeking into the string? This would allow the schema to collapse into a strng without any "special treatment" for cast references.

[pfletcherhill]

I agree on (1) and (2). As for (3), I do like the simplicity of every embed, parent, and target being a string, but I don't think the benefits outweigh the storage costs. For an example cast ID (i.e. cast:34375/0xa0c5397ffab555cb466f5715103156e706fc2bfe), the hub would require 55 bytes to store the string (0x0a35636173743a33343337352f307861306335333937666661623535356362343636663537313531303331353665373036666332626665) and 26 bytes to store the structured version (0x08c78c021214a0c5397ffab555cb466f5715103156e706fc2bfe). What I'm not sure about is whether we should aim to "structure" other URIs over time at the protocol level once they are stable or only ever have this special case for casts.

[davidfurlong]

This is epic and has been top of mind for me recently, would love to be able to use this soon!

I strongly agree canonical identifiers are better than URL identifiers.

Some ideas:

• Could mentions or inline urls within cast also support arbitrary targets? Seems to give the cast creator a bit more control of the context within a cast in which something is mentioned. It would unlock expressing "[PURPLE nft url] is awesome, [other NFT url] is not." in a cast without specifically coupling to a url format like opensea
• This could be used to tag locations as well - like geo:22.300;-118.44 (https://datatracker.ietf.org/doc/rfc5870/) or some other scheme that makes sense to allow users to tag locations in their posts

[varunsrin]

1. i think this just works today? nothing stopping two clients from saying xyz:foo is a canonical identifier of something inside the text. the reason. but i also don't think its a direction we should pursue because it increases the client burden for rendering a piece of text - now you cannot simply assume that the text is something that can be shown to users, it must be pre-processed and every client must implement this logic and keep it up to date.

2. location metadata is a great idea. once @pfletcherhill adds a general system for making new proposals, we'd be very open to one for geo tags.

[davidfurlong]

while it is possible today, with warpcast being the overwhelming majority of the distribution today, if the fallback experience for warpcast users is worse then that's often sufficient incentive not to format in that way for the rest of clients

[michaelhly]

+1 on location metadata

[pfletcherhill]

Initial supported formats

After some further discussion, here are the two formats clients should initially support:

1. Web URL

• Regex: ^https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]+\.[a-zA-Z0-9()]{1,6}([-a-zA-Z0-9()@:%_\+.~#?&//=])*$
• Example: https://www.farcaster.xyz

2. Ethereum asset or collection

• CAIP-19 asset URI prefixed with chain:// scheme
• Regex: ^chain:\/\/eip155:(\d+)\/([a-zA-Z0-9]+):([a-zA-Z0-9]+)(\/([a-zA-Z0-9]+))?$
• Example (NFT): chain://eip155:1/erc721:0xa723a8a69d9b8cf0bc93b92f9cb41532c1a27f8f/11
• Example (NFT collection): chain://eip155:1/erc721:0xa723a8a69d9b8cf0bc93b92f9cb41532c1a27f8f

Client upgrade guide

Phase 1

In the next few days, Warpcast will start submitting casts with embeds structured as Ethereum asset URIs in the above format. Other Farcaster clients should ensure that their UIs will not break if cast embeds are not web URLs—and ideally render them as OpenSea or Etherscan links.

Phase 2

Once this spec is adopted by the protocol, the location of embeds in the CastAddBody protobuf will change. The protocol is upgrading from embeds as a list of strings (repeated string embeds) to a list of Embed messages that can mix-and-match strings and cast IDs (repeated Embed embeds). Once the new protocol version is released and hubs implement it, clients will have at least two weeks to start writing embeds to the new embeds field and in the new format. Here's an example:

# Old format
const oldCast = makeCastAdd(CastAddBody.create({ ..., embeds: ["https://www.farcaster.xyz"], ... }, ...);

# New format
const newCast = makeCastAdd(CastAddBody.create({ ..., embeds: [{ url: "https://www.farcaster.xyz" }, castId: { fid: 15, hash: ... }], ... }, ...);

The embeds in the old format after the upgrade will still be readable from a field called embedsDeprecated. Here's an example:

# CastAddBody
{
  embedsDeprecated: ["https://www.farcaster.xyz"],
  ...,
  text: "Hello world",
  embeds: [],
}

Once an upgrade cut-off is set (at least two weeks after hubs support the new format), casts that use the old embeds format and have a timestamp after the cut-off will be rejected by hubs.

Over time, casts using the old embeds format will be pruned from hubs, so clients should plan on rendering both formats for at least one year after the upgrade (casts are pruned after one year).

[pfletcherhill]

The reason to add the chain:// scheme to the Ethereum asset URI is for future-proofing. We expect to add other schemes, like location or something else, as well as more blockchains using the same CAIP-19 format (i.e. chain://tezos/nft:KT1KEa8z6vWXDJrVqtMrAeDVzsvxat3kHaCE/145979). By using the chain:// scheme, we don't have to worry about CAIP-19 namespaces conflicting with other URI schemes.

We should use chain:// rather than nft:// because we will likely want to reference on-chain transactions using the same scheme, which are not NFTs.

We should use chain:// rather than caip:// because CAIP does not have a standard for transactions, so by supporting transactions we'll be diverging from the existing standard so chain:// is still more apt.

[AMAN-BARBARIA]

This looks good .
@pfletcherhill Should we modify the regex to allow Ethereum NFT collections
Regex: ^chain://eip155:(\d+)/([a-zA-Z0-9]+):([a-zA-Z0-9]+)(/([a-zA-Z0-9]+))?$

[pfletcherhill]

Sure, that makes sense to me. I'll update the regex and examples.

[goksu]

Per latest proposal, here's a sample stub that can convert CAIP-19 asset URIs to OpenSea asset URLs. Sharing in case it is helpful for folks to plug & play right away:

const CAIP_19_REGEX = /^chain:\/\/eip155:1\/erc721:((?:0x)?[a-fA-F0-9]{1,})\/(\d{1,})/;

const convertCaip19UriToOpenSeaUrl = ({ uri }: { uri: string }) => {
  const matches = uri.match(CAIP_19_REGEX);

  if (matches === null) {
    throw new Error('Invalid CAIP-19 URI');
  }
  const [, contractAddress, tokenId] = matches;
  return `https://opensea.io/assets/ethereum/${contractAddress}/${tokenId}`;
};

convertCaip19UriToOpenSeaUrl({ uri: 'chain://eip155:1/erc721:0xcc3cac6d9f77681cbff754377ad4599f5d8afd30/37' });
// Returns: https://opensea.io/assets/ethereum/0xcc3cac6d9f77681cbff754377ad4599f5d8afd30/37

[betashop]

Will check this out, thanks

[varunsrin]

@pfletcherhill and I discussed offline and wanted to split this into two separate proposals:

Proposal 1: Canonical URIs, a scheme for referencing urls, onchain elements and more with a flexible, evolving standard

Proposal 2: URI Targets, a protocol upgrade that allows casts and reactions to point to URIs (this proposal)

[pfletcherhill]

Detailed implementation (updated 4/18)

This comment aims to cover exactly what upgrades will be made to the protobuf messages and gRPC hub service. The code implementing this proposal is here: farcasterxyz/hub-monorepo#847

Message protobuf upgrades

Note: fields in this note use the protobufs underscore syntax, though ts-proto generated code uses camelCase syntax (i.e. embeds_deprecated is embedsDeprecated in practice).

CastAddBody.embeds

// Current
message CastAddBody {
  repeated string embeds = 1; 
  ...
}

// Upgrade
message CastAddBody {
  repeated string embeds_deprecated = 1; 
  ...
  repeated Embed embeds = 6;
}

message Embed {
  oneof embed {
    string url = 1;
    CastId cast_id = 2;
  }
}

The upgraded version will store embeds (field number 6), each of which can be a string or a CastId.

⚠️ Upgrade compatibility

A message that includes an embed in the new format will fail validation in a hub or client running a prior protocol version, because the encoded CastAddBody will differ from the upgraded encoded version and therefore the hash will look invalid.

CastAddBody.embeds_deprecated

As shown above, the CastAddBody message will preserve embeds in the previous format and rename the field to embeds_deprecated.

⚠️ Upgrade compatibility

However, in order to incentivize using the new format, hubs will only accept messages that use embeds_deprecated with timestamps before 5/3/23 00:00 UTC (Farcaster time 73612800). That gives clients two weeks to start writing to the new format. Messages that use embeds_deprecated and have a timestamp after the cut-off will be rejected by hubs with a validation error.

CastAddBody.parent_url

// Current
message CastAddBody {
  ...
  oneof parent {
    CastId parent_cast_id = 3;
  };
  ...
}

// Upgrade
message CastAddBody {
  ...
  oneof parent {
    CastId parent_cast_id = 3;
    string parent_url = 7;
  };
  ...
}

The upgraded version will allow the parent of a cast to be either a CastId (current behavior) or a string.

⚠️ Upgrade compatibility

Messages that use parent_url will fail validation on non-upgraded hubs and clients because the hash will look invalid. Messages using parent_cast_id will continue to work in both environments and will not be deprecated.

ReactionBody.target_url

// Current
message ReactionBody {
  ...
  oneof target {
    CastId target_cast_id = 2;
  }
}

// Upgrade
message ReactionBody {
  ...
  oneof target {
    CastId target_cast_id = 2;
    string target_url = 3;
  }
}

The upgrade allows reactions to target casts (current behavior) or strings.

⚠️ Upgrade compatibility

Messages that use target_url will fail validation on non-upgraded hubs and clients for the same hash-related reason. Messages using target_cast_id will continue to work in both environments and will not be deprecated.

gRPC upgrades

The hub gRPC service will change with this upgrade though will be backwards compatible (i.e. clients using @farcaster/hub-nodejs@0.6.3 will be able to interact with upgraded hubs using the same methods as before). Keep in mind that messages returned from upgraded hubs that make use of the upgrades above will be readable but will fail validation.

GetReactionsByTarget(ReactionsByTargetRequest)

// Current
rpc GetReactionsByCast(ReactionsByCastRequest) returns (MessagesResponse);

message ReactionsByCastRequest {
  CastId cast_id = 1;
  ...
}

// Upgrade
rpc GetReactionsByCast(ReactionsByTargetRequest) returns (MessagesResponse); // To be deprecated as of v2023.5.31
rpc GetReactionsByTarget(ReactionsByTargetRequest) returns (MessagesResponse);

message ReactionsByTargetRequest {
  oneof target {
    CastId target_cast_id = 1;
    string target_url = 6;
  }
  ...
}

This method is a new name for the GetReactionsByCast method and the ReactionsByTargetRequest request message now includes target_cast_id and target_url as fields. Previously, the target_cast_id field was called cast_id, so apps using the new clients should use target_cast_id instead. GetReactionsByCast is preserved with this version, though it will be deprecated and removed in the next protocol version.

GetReaction(ReactionRequest)

// Current
rpc GetReaction(ReactionRequest) returns (Message);

message ReactionRequest {
  ...
  CastId cast_id = 3;
}

// Upgrade
rpc GetReaction(ReactionRequest) returns (Message);

message ReactionRequest {
  ...
  oneof target {
    CastId target_cast_id = 3;
    string target_url = 4;
  }
}

This method now accepts target_cast_id and target_url, whereas it previously just accepted cast_id. Apps using the new client should upgrade to use target_cast_id, though the field number is the same so old clients can still use the method without problem.

GetCastsByParent(CastsByParentRequest)

// Current
rpc GetCastsByParent(CastsByParentRequest) returns (MessagesResponse);

message CastsByParentRequest {
  CastId cast_id = 1;
  ...
}

// Upgrade
rpc GetCastsByParent(CastsByParentRequest) returns (MessagesResponse);

message CastsByParentRequest {
  oneof parent {
    CastId parent_cast_id = 1;
    string parent_url = 5;
  }
  ...
}

This method now accepts parent_cast_id and parent_url, whereas it previously just accepted cast_id. Apps using the new client should upgrade to use parent_cast_id, though the field number is the same so old clients can still use the method without problem.

[pfh]

Er. Greetings, fellow PFH. Github decided to tell me about this because you mentioned @pfh.

[varunsrin]

derp, that was referencing a username on another platform, but i guess GH auto links things now. sorry for the ping.