Olympia membership input schema

[bedeho]

Background

The purpose of this issue is to serve as a discussion thread for the state of the current Olympia input schemas.

Here is a fully rendition of the current schema, lifted from

https://github.com/Lezek123/substrate-runtime-joystream/blob/olympia-integration-tests/query-node/schema.graphql

enum Network {
  BABYLON
  ALEXANDRIA
  ROME
  OLYMPIA
}

type Block @entity {
  "Block number as a string"
  id: ID!
  block: Int!
  executedAt: DateTime!
  network: Network!
}

enum MembershipEntryMethod {
  PAID
  INVITED
  GENESIS
}

"Stored information about a registered user"
type Membership @entity {
  "MemberId: runtime identifier for a user"
  id: ID!

  "The unique handle chosen by member"
  handle: String! @unique @fulltext(query: "membersByHandle")

  "Member's name"
  name: String

  "A Url to member's Avatar image"
  avatarUri: String

  "Short text chosen by member to share information about themselves"
  about: String

  "Member's controller account id"
  controllerAccount: String!

  "Member's root account id"
  rootAccount: String!

  "Blocknumber when member was registered"
  registeredAtBlock: Block!

  "Timestamp when member was registered"
  registeredAtTime: DateTime!

  "How the member was registered"
  entry: MembershipEntryMethod!

  "Whether member has been verified by membership working group."
  isVerified: Boolean!

  "Staking accounts bounded to membership."
  boundAccounts: [String!]

  "Current count of invites left to send."
  inviteCount: Int!

  "All members invited by this member."
  invitees: [Membership!] @derivedFrom(field: "invitedBy")

  "A member that invited this member (if any)"
  invitedBy: Membership

  "All members referred by this member"
  referredMembers: [Membership!] @derivedFrom(field: "referredBy")

  "A member that referred this member (if any)"
  referredBy: Membership
}

type MembershipSystem @entity {
  "Initial invitation count of a new member."
  defaultInviteCount: Int!

  "Current price to buy a membership."
  membershipPrice: BigInt!

  "Amount of tokens diverted to invitor."
  referralCut: BigInt!

  "The initial, locked, balance credited to controller account of invitee."
  invitedInitialBalance: BigInt!
}

Comments

NB: In all comments below I use relationships to Extrinsic and Block, but we currently cannot filter, order and aggregate through relationships fully, so there is some risk in doing this now. In general it also requires more efforts to get the mappings right. It may be wise to just inline hashes and block numbers for now.

If we do want to use them, this is what I currently have

"A blockchain genesis block."
enum Network {
  Babylon
  Sumer
  # Mainnet
}

"An event interface."
interface Event @entity {

  "Concatenation of block hash and index in block."
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  # Is there an index in extrinsic field worth adding?

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!
}

"A block in the blockchain."
type Block @entity {

  "Hash of block in hex encoding."
  id: ID!

  "Network in which block occurred."
  network: Network!

  "Height at which block is committed."
  height: BigInt!

  "Is block finalized. Note: we can have richer consensus information here in the future, like who signed, but its not a priority now."
  isFinalized: Boolean!

  "Timestamp in block."
  timestamp: BigInt!

  "Hash of block in hex encoding."
  hash: String!

  "Hash of parent block in hex encoding."
  parentHash: String!

  "State root hash in hex encoding."
  stateRoot: String!

  "Extrinsics merkle root in hex encoding."
  extrinsicRoot: String!

  "Time since last block."
  blockTime: BigInt!

  # All events triggered in block.
  extrinsics: [Extrinsic!] @derivedFrom(field: "inBlock")

  # All events triggered in block.
  events: [Event!] @derivedFrom(field: "inBlock")

}

"Substrate Root origin type."
type RootOrigin @variant {

  "No meaning, only here because GraphQL cannot support empty types."
  phantomField: Int
}

"Substrate signed origin type."
type SignedOrigin @variant {

  "Signing account in SS58 encoding."
  signingAccount: String!
}

"Substrate none origin type."
type NoneOrigin @variant {

  "No meaning, only here because GraphQL cannot support empty types."
  phantomField: Int
}

"Substrate extrinsic origin."
union ExtrinsicOrigin = RootOrigin | SignedOrigin | NoneOrigin

"A general blockchain extrinsic."
type Extrinsic @entity {

  "Hash in hex encoding."
  id: ID!

  "Name of runtime module to which extrinsic was targeted."
  moduleName: String!

  "Name of extrinsic, including module prefix."
  call: String!

  "Nonce"
  Nonce: BigInt!

  "Fee charged."
  fee: BigInt!

  "Tip provided."
  tip: BigInt!

  "Origin for extrinsic."
  origin: ExtrinsicOrigin!

  "Call parameters, unknown encoding currently."
  parameters: String

  "Index of extrinsic in block."
  indexInBlock: Int!

  "Block in which this extrinsic was included."
  inBlock: Block!

  "Whether extrinsic was successful."
  successful: Boolean!

  # All events triggered by this extrinsic.
  # Note that any on-chain event which does not have an entity implementing `Event` interface will be omitted here.
  # COMMENTED OUT FOR NOW DUE TO EVENT BEING INTERFACE
  #events: [Event!] @derivedFrom(field: "inExtrinsic")

}

Founding Member Field

Founding member state missing

  "Whether member is founding member."
  isFoundingMember: Boolean!

Avatar Representation

While currently not in the runtime, I think its safe to say that we will use same storage representation as in Sumer for avatars, as the storage system will be able to host them. So I suggest we switch to that, check the Sumerbranch, which currently has this

type Membership @entity {

  "...

  ### Cover photo asset ###

  # Channel's cover (background) photo. Recommended ratio: 16:9.

  "Asset's data object"
  coverPhotoDataObject: DataObject

  "URLs where the asset content can be accessed (if any)"
  coverPhotoUrl: String!
  
  "...
  
}

"The decision of the storage provider when it acts as liaison"
enum LiaisonJudgement {
  "Content awaits for a judgment"
  PENDING,

  "Content accepted"
  ACCEPTED,

  "Content rejected"
  REJECTED,
}
  
  "Manages content ids, type and storage provider decision about it"
type DataObject @entity {
  "Content owner"
  owner: DataObjectOwner!

  "Content added at"
  addedAt: BigInt!

  "Content type id"
  typeId: Int!

  "Content size in bytes"
  size: BigInt!

  "Storage provider id of the liaison"
  liaisonId: BigInt!

  "Storage provider as liaison judgment"
  liaisonJudgement: LiaisonJudgement!

  "IPFS content id"
  ipfsContentId: String!

  "Joystream runtime content"
  joystreamContentId: String!
}

"Owner type for storage object"
union DataObjectOwner = DataObjectOwnerMember
  | DataObjectOwnerChannel
  | DataObjectOwnerDao
  | DataObjectOwnerCouncil
  | DataObjectOwnerWorkingGroup

Richer representation of origin of membership

There is a way to represent how a membership has been established in a more compact & safe way, using a type like

"Membership creation genesis configuration."
type GenesisConfiguredMember @variant {

  "Network genesis at which a membership was established."
  network: Network!
}

"Membership creation via an invitation."
type MemberInvitation @variant {

  "Event corresponding to invitation of a membership."
  event: InvitedMemberEvent!
}

"A membership originating fro a purchase."
type MemberPurchase @variant {

  "Event corresponding to buying of a membership."
  event: BoughtMemberEvent!
}

"From which a membership can originate."
union MemberSource = GenesisConfiguredMember | MemberInvitation | MemberPurchase

This would replace registeredAtBlock, registeredAtTime and entry, however, this representation will make some queries harder, and requires more effort, so it may be useful to just stick to what you currently have. I just wanted to raise your awareness of this type of approach, as I think it will be a consistent question in the future for other schemas.

Metadata

Perhaps we should aggregate all the metadata fields together, so its more clear where they are coming from, and perhaps also include some comment about what version of the metadata for memberships this schema is using?

Events

We need to also index all of the events associated with each subsystem, this is needed not only to how a chronological list of events associated with a given member, worker or working group, as is needed in a few places in Pioneer, but it also allows us to compute various historical numbers by using aggregate functions (Joystream/hydra#195), without needing to have explicit query state for this. This could be things like, the total amount of money worker has earned or staked for example.

My approach has been to have an Event interface type, and then have that be implemented by an entity variation for each event in question, like this.

Event

"An event interface."
interface Event @entity {

  "Concatenation of block hash and index in block."
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  # Is there an index in extrinsic field worth adding?

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!
}

Membership events

type BoughtMemberEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "New membership created."
  newMember: Member!

  "New member root account in SS58 encoding."
  rootAccount: String!

  "New member controller in SS58 encoding."
  controllerAccount: String!

  "New member user name."
  name: String!

  "New member handle."
  handle: String!

  "New member avatar asset."
  #avatar: StorageAssetStatus!
  avatarURI: String

  "New member 'about' text."
  about: String!

  "Referrer member."
  referrer: Member
}

type InvitedMemberEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Inviting member created."
  invitingMember: Member!

  "New membership created."
  newMember: Member!

  "New member root account in SS58 encoding."
  rootAccount: String!

  "New member controller in SS58 encoding."
  controllerAccount: String!

  "New member user name."
  name: String!

  "New member handle."
  handle: String!

  "New member avatar asset."
  # avatar: StorageAssetStatus!
  avatarURI: String

  "New member 'about' text."
  about: String!
}

type MemberProfileUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership being updated."
  member: Member!

  "New member root account in SS58 encoding. Null means no new value was provided."
  newRootAccount: String

  "New member controller in SS58 encoding. Null means no new value was provided."
  newControllerAccount: String

  "New member user name. Null means no new value was provided."
  newName: String

  "New member handle. Null means no new value was provided."
  newHandle: String

  "New avatar asset. Null means no new value was provided."
  newAvatarURI: String

  "New member about text. Null means no new value was provided."
  newAbout: String!
}

type MemberAccountsUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New member root account in SS58 encoding. Null means no new value was provided."
  newRootAccount: String

  "New member controller in SS58 encoding. Null means no new value was provided."
  newControllerAccount: String
}

type MemberVerificationStatusUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  #"Worker updating status"
  #worker: Worker!

  "New status."
  isVerified: Boolean!
}

type ReferralCutUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New cut value."
  newValue: BigInt!
}

type InvitesTransferredEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership sending invites."
  sourceMember: Member!

  "Membership receiving invites."
  targetMember: Member!

  "Number of invites transferred."
  numberOfInvites: BigInt!
}

type MembershipPriceUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "The new membership price."
  newPrice: BigInt!
}

type InitialInvitationBalanceUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "New initial invitation balance."
  newInitialBalance: BigInt!
}

type LeaderInvitationQuotaUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New quota."
  newInvitationQuota: Int!
}

type InitialInvitationCountUpdatedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Initial invitation count for members."
  newInitialInvitationCount: Int!
}

type StakingAccountAddedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New staking account in SS58 encoding."
  account: String!
}

type StakingAccountRemovedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New staking account in SS58 encoding."
  account: String!
}

type StakingAccountConfirmedEvent implements Event @entity {

  # ------ Event <interface> ------

  "Event identifier"
  id: ID!

  "Possible extrinsic in which"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  # ------ Event <interface> ------

  "Membership in question."
  member: Member!

  "New staking account in SS58 encoding."
  account: String!
}

MembershipSystem

This entity, which has the following comment in my file

"State of the membership system. NB: There should only be one instance of this entity."

is a bit degenerate. It feels awkward to only have a single instance of this entity.

I think an alternative way to do this is to switch the meaning to something like MembershipSystemSnapshot, and add a block height field. Then a new entity is created for every block where at least one variable is changed, and initial entity is derived from genesis configuration. If there is more than one update in a block, then that just results in mutating the same entity. So for example, if you in the same block have a proposal to udpate the referral cut, and a separate one to update the membership price, then you still just ahve one snapshot. Now, if someone wants to get the latest state, they just order by block height and set limit to 1. Alternatively, they could access historical values for the system over time, which could be useful for various historical UI views.

What do you think?

[Lezek123]

NB: In all comments below I use relationships to Extrinsic and Block, but we currently cannot filter, order and aggregate through relationships fully, so there is some risk in doing this now. In general it also requires more efforts to get the mappings right. It may be wise to just inline hashes and block numbers for now.

Does this mean that for now we should just add the Extrinsic type containing only the extrinsic hash and relation to a block?

Founding member state missing

I don't think we have any extrinsics / events related to that state, should it just be a property that's always false for now?

While currently not in the runtime, I think its safe to say that we will use same storage representation as in Sumer for avatars, as the storage system will be able to host them. So I suggest we switch to that, check the Sumer branch, which currently has this

I was thinking about switching to that once it's supported by the runtime (after Sumer -> Olympia merge) in case there will be any changes in the mappings. Of course we can make the switch now, but there would be no way to test / make use of this part of the schema currently.

We need to also index all of the events associated with each subsystem, this is needed not only to how a chronological list of events associated with a given member, worker or working group, as is needed in a few places in Pioneer, but it also allows us to compute various historical numbers by using aggregate functions (Joystream/hydra#195), without needing to have explicit query state for this. This could be things like, the total amount of money worker has earned or staked for example.

I can see the utility of having this state accessible through query node, though I was wondering about the query node efficiency in case we decide to store all the events and extrinsics that get processed and then use aggregate queries (which will obviously have a much higher execution cost than a querying, for example, a single amount earned field in worker/member). So as I understand it comes with additional storage cost, query execution cost and implementation cost (having to create schemas, mappings and tests for all the events).

Looking at the wireframes I also didn't see where all of this event data would be needed (perhaps I overlooked something?)

My approach has been to have an Event interface type, and then have that be implemented by an entity variation for each event in question, like this.

It seems reasonable, I'm not sure what other options are there when it comes to GraphQL schemas and Hydra/warthog support. I was also considering separating generic event data from specific events data (perhaps it may have some value in terms of relationships where event type is not enforced), for example:

enum EventType {
  StakingAccountConfirmed,
  StakingAccountRemoved
}

type Event @entity {
  "Concatenation of block hash and index in block."
  id: ID!

  "Possible related extrinsic"
  inExtrinsic: Extrinsic

  "Block in which event was emitted."
  inBlock: Block!

  "Index of event in block from which it was emitted."
  indexInBlock: Int!

  "Type of the event"
  type: EventType!
}


type StakingAccountConfirmedEvent @entity {
  "Generic event data"
  event: Event!

  "Membership in question."
  member: Member!

  "New staking account in SS58 encoding."
  account: String!
}

type StakingAccountRemovedEvent @entity {
  "Generic event data"
  event: Event!

  "Membership in question."
  member: Member!

  "New staking account in SS58 encoding."
  account: String!
}

The best solution would probably be some kind of inheritance, but it's not obvious to me how this could be supported both on GraphQL schema and database/ORM side (I'm pretty sure it would be tough to implement)

[bedeho]

Does this mean that for now we should just add the Extrinsic type containing only the extrinsic hash and relation to a block?

No, it would mean that we do not use a relationship at all. So the Block and Extrinsic entities are removed from the schema, and the field of this type just use hash or block number, respectively.

I don't think we have any extrinsics / events related to that state, should it just be a property that's always false for now?

Correct, this will probably be introduced as an event that is triggered exclusively by the build code for the gensis block. Alternatively it will just be implied genesis value, like the total issuance, which is comitted to in the constitution.

I was thinking about switching to that once it's supported by the runtime (after Sumer -> Olympia merge) in case there will be any changes in the mappings. Of course we can make the switch now, but there would be no way to test / make use of this part of the schema currently.

I suggest we add it to the schema now, but do not write any mappings to manage it at all, that way it will be there for the Pioneer devs to see as early as possible, and it is not forgotten.

Looking at the wireframes I also didn't see where all of this event data would be needed (perhaps I overlooked something?)

The simplest example are the event logs showed on a the right hand side of different screens, such as

https://www.figma.com/file/pgNl83wgAub186krVkdiuo/Proposals?node-id=183%3A757

I was also considering separating generic event data from specific events data (perhaps it may have some value in terms of relationships where event type is not enforced)

Good question, for filtering it would appear to be required actually, so I think it is a must.