QueryNode: Support Event types not in latest runtime

[mnaamani]

When a new runtime removes an event variant, we can no longer reference it in the query-node/hydra manifest.yml as the metadata for the event runtime type is not found in the latest runtime. But we need to keep the mapping to handle the event when it was emitted in previous runtimes. (This is because we currently have a single set of mappings that need to work all the way from genesis)

Keeping around unused runtime types in the runtime is not ideal.

┆Issue is synchronized with this Asana task by Unito

[bedeho]

is this a query node issue, hydra issue? I'm adding nara label, correct me if wrong.

[mnaamani]

This is specifically a hydra issue,during the query-node build process when generating processor specific code/types.

joystream/query-node/build.sh

Line 18 in a03082e

yarn typegen # if this fails try to run this command outside of yarn workspaces

The typegen yarn script:
"typegen": "rm -rf ./mappings/generated && yarn hydra-typegen typegen manifest.yml --debug"

For example, when the events are removed and I rebuild the node,update chain metadata files and run the QN build this is what the error looks like:

hydra-typegen:imports-registry Built-in classes: BitVec,Bool,Bytes,Data,I128,I16,I256,I32,I64,I8,Null,OptionBool,StorageKey,Text,Type,U128,U16,U256,U32,U64,U8,USize,bool,i128,i16,i256,i32,i64,i8,u128,u16,u256,u32,u64,u8,usize,Json,Raw,BTreeMap,BTreeSet,CodecMap,CodecSet,Compact,DoNotConstruct,Enum,HashMap,Int,Json,Linkage,Map,Option,Range,RangeInclusive,Raw,Result,Set,Struct,U8aFixed,UInt,Vec,WrapperKeepOpaque,WrapperOpaque,GenericAccountId,GenericAccountIndex,GenericAddress,GenericBlock,GenericCall,GenericChainProperties,GenericConsensusEngineId,GenericEvent,GenericEventData,GenericLookupSource,GenericMultiAddress,GenericVote,GenericEthereumAccountId,GenericEthereumLookupSource,GenericExtrinsic,GenericExtrinsicEra,GenericExtrinsicPayload,GenericExtrinsicPayloadUnknown,GenericExtrinsicUnknown,GenericImmortalEra,GenericMortalEra,GenericSignerPayload,GenericExtrinsicPayloadV4,GenericExtrinsicSignatureV4,GenericExtrinsicV4 +0ms
    Error: No metadata found for the event content.ChannelDeletedByModerator

[mnaamani]

.. the manifest.yml has a section that refers to the chain metadata:

typegen:
  metadata:
    source: ./chain-metadata/
  events:
     - palletA.EventName1
     - palletB.EventName2

The folder contains multiple versions for each version of the runtime we support with the filenames as {runtime_spec_version}.json
This was introduced recently to be able to handle changes in the Event payload of some events in the last runtime upgrade.

The problem as @zeeshanakram3 described is that hydra-typegen seems to be only looking at the latest chain-metadata.

Possible solutions:
A. hydra-typegen should look at all the versions (starting with latest, and working backwards?)
B. modify manifest file structure to specify the spec version where event metadata can be found for a specific event if not the latest. eg.

typegen:
  metadata:
    source: ./chain-metadata/
  events:
    - palletA.EventName1,2001  # look in chain metadata for spec version 2001
    - palletB.EventName2          # look in latest chain metadata

C. ... ?