Sources transaction field: SUBMIT_POST

[HPaulson]

---

CIP: To be assigned
Title: Sources field for SUBMIT_POST transactions
Author: Hunter Paulson <@HPaulson>
Discussions URL: #3
Created: 2021-07-07
Note: This is a CIP duplication of bitclout/core #32

---

One-Line Description

Add a new Sources field added to SUBMIT_POST transaction types to allow applications to specify where a post is meant to display.

Explanation and Motivation

As the BitClout ecosystem expands, and many new applications are built on top of the protocol, having a standard for node-specific content will become vital. For example, if an engineer wanted to create an app built on top of the protocol which only displayed posts meant for that app (For example, if you wanted to make a Reddit-like app, it wouldn't make sense to display all posts from all subreddits on one twitter-like feed), the app could formulate a field like such in the request: Sources: ["myApp.com"]. BitClout.com, and any other nodes looking to display all content, could use the "Default" source to signify it displays regular content not specific to an application type. Those nodes could filter by this source. This could allow engineers to display posts only allowed for their source, without doing anything too hacky.

I think for the long-term success of BitClout we need many types of applications built on top of the protocol, and additions like this would be very very helpful in accomplishing such.

Specification

Backend

The new field, Sources: []string, should be added to typings for both PostEntryResponse and SubmitPostRequest. Additionally, the SubmitPost function should be updated to include the data from Sources in its call to CreateSubmitPostTxn.

In addition, the backend should add a conditional to check if a post is meant for the node it's running on -- though it would be at the discretion of the node operator to set such using a comma-separated list of the accepted sources in a backend flag. For example, --sources "prism.red, love4src.com" would active such a conditional to only return posts which include prism.red OR love4src.com in its Sources array, and no flag would include all posts regardless of its Sources. BitClout.com's node should be updated accordingly, displaying posts that include the Default source.

Core

The CreateSubmitPostTxn function should be updated to accept the Sources data as a parameter. The Source field should also be added to the SubmitPostMetadata struct with its appropriate []byte type. Finally, the usages of SubmitPostMetadata should be updated: Main, Test

Frontend

The SubmitPost method should be updated to include Sources: Array<string> as an parameter, and should include such as an argument when calling the Api post function. Additionally, the submitPost method should be updated accordingly, including Sources: ["Default"] in it's call to SubmitPost().

Other Solutions

While methods for accomplishing such a task already exist, for example hiding a post by default and making use of PostExtraData to identify the source application & if the post is actually hidden or not, however, I think having a standard method of doing such would be a much better, and long-lasting, solution.

Backwards Compatibility

There are no known issues with backward compatibility.

Test Cases

The only required test should be an update to TestSerializeSubmitPost, as described above.

Security Considerations

No known concerns.

[connerdouglass]

Hey @HPaulson, congrats for starting the first community proposal discussion in this repo! Overall this is an interesting idea. I have a few thoughts that I think are worth exploring:

• Why call it Sources? Because it's just a slice of strings added to the transaction meta, I think they're more like Tags.
• If you want it to be cryptographically enforceable, nodes need to have a public key associated with them, and they need to use the associated private key to affix a signature to every post originating from their node. I don't support this, just wanted to mention it.

My thinking currently is that adding a slice of strings to the transaction wouldn't achieve the desired filtering goals, but I would really like to hear more about this to clear up my understanding!

[HPaulson]

1. Originally my idea was to have it be one string, the "Source" node where the post came from, but realized having multiple makes much much more sense -- this was my rational, but quite honestly, the naming really doesn't matter.

2. I disagree entirely on this point. The purpose of the sources is to designate that a post is "supposed to" display on a particular node -- the goal is not to make it so only that node can add its designator to a post, or to make it enforceable in any way. For my specific use case, as an example: Prism is an application which is Instagram-like in nature. Considering this, having it share all posts with an application which is Twitter-like wouldn't make much sense, as often images on Instagram are much more high quality that those you may post on Twitter. For this purpose, posts on Prism won't (by default) display on other nodes, and posts from other nodes won't display on Prism, as having such wouldn't make sense. So, having a specified tag or source which confirms the post is meant for Prism makes a ton of sense -- and is more viable than using the hacky method we rely on now (see "Other Solutions"). It also allows for a more unified spec on the topic, as I'm aware of other applications looking to do the same. Anyone could add their post to Prism even while not using the app itself, by adding such a designator, so no need to make adding the source exclusive. The goal isn't to restrict who can add a post to what app, but to clarify that a post SHOULD display on a particular app. (Unsure if I explained this great, it's super late. Let me know if you need clarification!)

[bloated-devish]

Few questions:

1. I'd strongly suggest that you mention the size of this Sources backend field. What is the size of each string? How many different strings can be added?

2. I personally would prefer an integer enum so it uses less space (and possible to retain with a bitset array for multiple sources). This is especially relevant, since a not-so forward looking application can start making posts costlier by simply having large tags.

3. Do you care about adding a core / backend lookup of posts meeting specific source criteria? This might be useful for frontend nodes.

Also food for thought: Are you expecting this to grow into something more inter operable? Like tomorrow would this be something that a class of applications could interoperate? Do you want all prism type apps to consume this post. In that case, we could have the enum be grouped - assign some range for apps of various types and create a util func for group identification.

Thanks -Bloated

[HPaulson]

Sorry for the late reply here.

1. I didn't consider any specific size. Here's why: Since the body of a post (Both the content, and imageURLs array) is already "Unlimited" in size (Only restricted by transaction size to my knowledge), I assumed this would be no different
2. I'm not exactly sure what specifically you're suggesting. Could you please clarify?
3. I mention this is the "Backend" section of the specification

As far as food thought -- 100%. The intention is to allow groups of apps to work together and allow for cross-support between them. For example, if Application A wanted to display posts from Application B and C, it could make the appropriate implementations according to those app's spec (Likely has special data in PostExtraData), and then include such applications in its Source flag backend. This will allow for collaboration, but make it so apps aren't grouped -- as with grouping, important post data in PostExtraData may be ignored. This ensures the full value of posts is included while collaborating among projects, rather than being missed by accepting posts from a large group of apps. The only "Group" source I'd recommend is the Default group, mentioned above.

[dgsus]

i believe its important to be able to pinpoint where a transaction came from, apart from the use case Hunter mentioned, i see a at least one other good case -> forensic investigations of bad actors

not sure if the best thing would be a public key associated with them, or some sort of unique hash based on few properties of the node (physical/virtual , network, etc)

[tijno]

I think it would be very useful as a separate discussion - not necessarily for implementation right now - but its a complex topic that needs some good quality thought & time to mature. So lets get that discussion started?

[HPaulson]

@dgsus Yeah I think this is an amazing idea if done correctly (as Tijn mentioned, lot of privacy concerns), but is a separate topic. I'd love to see a separate Discussion created on this!

[dgsus]

Yes will submit something later tonight to start this discussion

[bloated-devish]

I have another concern. Could someone actually create tons of PRISM tagged posts, but just useless, just to make the PRISM app slow? In such a situation, would it be relevant to actually PRISM identify itself - that this is from the PRISM app. The solution there could be to encrypt the transaction/content or even some specific field with the node's private key and let the trust block provider decrypt using the node's public key to confirm and tag it as prism. This is a little complicated idea (while not sure if there is an underlying problem to solve), with some security implications - the node's key is in the config, but should also not get leaked (config file github submits are not unheard of), but can solve the non repudiation of node problem. Privacy implications not any different from current proposal.

[HPaulson]

@bloated-devish This isn't really relevant as it's already possible today. Someone could easily do the exact same thing without source tags, no change. Again, the idea here is not to have a bunch of proprietary posts -- that'd defeat the purpose of bitclout. Rather it's to allow engineers to communicate easily with nodes that a post authed by the user is meant for that node. This is it. Anything beyond this is entering either dangerous regulatory/decentralization territory, or is overly complex and solves a non-existent problem. e

[HPaulson]

@lazynina What exactly does "the community [...approving] your proposal" consist of? Is the above discussion adequate to request core team approval? If not, what further discussion shall occur? Is there an ideal time period a discussion should be open for before becoming a PR? (Asking more so in general, but applies to this discussion)

[lazynina]

@HPaulson I don't have an exact definition of "the community [...approving] your proposal" at this time. My hope is that it becomes more defined after CIPs gets rolling for a bit, but for now we don't want process to hold up progress.

Generally, I think a discussion should be open for at least a few days (unless it is a trivial CIP) before drafting a PR. This allows both core devs and community members ample time to provide their thoughts on the proposal. It also helps save the CIP drafter time by reducing revisions of the CIP PR.

In this instance, I think there is more discussion to be had - I've got a few questions/concerns that I need to type up 😅

[tijno]

for now we don't want process to hold up progress

nina not lazy - keep pushing.

(i owe you about 20 PR reviews)

[HPaulson]

@lazynina Sounds great! Thanks for clarifying :)

[lazynina]

Thanks for the great discussion. Bit of a brain dump here -- I think there's a way to make this work, but below are my concerns/recommendations/questions.

1. Adding this to the SubmitPostMetadata would create a fork as it would change how SubmitPostMetadata is serialized. The ExtraData field of transactions is meant to support these non-core attributes and is where the Source key should live in the transactions instead of the SubmitPostMetadata. ExtraData is effectively an arbitrary map to store data about a transaction.
2. The sources attribute could become really bloated in the future when there are 1000s of nodes. If I want my post to show up on all these nodes, I'm going to need to include the sources filtered on for each of those nodes. This would lead to higher fees and generally inefficient storage of posts.
3. Existing posts and new posts without this source attribute would not pass the filters of these other nodes and in a way leads to unintentional censoring of a post. This becomes confusing as a user -- I posted on bitclout.com, but my post will not appear on prism or another node and it's not clear to the lay user why this happens.
4. Any post transaction could set whatever it wants as the source (as mentioned above) so I could have a post with no image set to display on prism.
5. How does source work in comments? Do I need to have the same source as the parent in order to show up on the prism node? Or does the filtering only apply to the root level post? Just curious what the thoughts are on this.

Perhaps instead of source, we can have community agreed upon tags, which are a bit more general - e.g. "high-quality image", "nsfw", etc. However, the categorization of posts may not be efficient to maintain on-chain in the long run.

brain dump over - rebooting nina

[HPaulson]

@lazynina @maebeam How can we proceed further?

[HPaulson]

Still waiting on this, has been well over a month now. @maebeam @lazynina

[maebeam]

This was implemented recently! Marking DIP as approved.

[HPaulson]

@maebeam The other section of this, which is equally important IMO, is being able to serve posts only from X node. Any plans on implementing this? @maebeam

[maebeam]

While I'd love to implement that feature I don't have any immediate plans to. Feel free to implement it though, I would happily review and merge a PR.