Strip metadata fix #1774
Strip metadata fix #1774
Conversation
metadata/src/utils/retain.rs
Outdated
| } | ||
| } | ||
|
|
||
| fn collect_types(&mut self, metadata: &Metadata, t: &PortableType) { |
There was a problem hiding this comment.
So this logic is basically recursing through all of the types linked from some type and adding them to the set. But this is the same thing that we do in PortableRegistry::retain here https://github.com/paritytech/scale-info/blob/master/src/portable.rs#L104; is it necessary?
There was a problem hiding this comment.
Yup, retain is called dead last before updating types, but we need to collect types ourselves to determine which outer type variants are safe to drop/which pallets can be stripped or dropped completetely.
There was a problem hiding this comment.
I'm not sure how I feel about this function, because it duplicates the logic in scale_info's retain.
a) On the one hand, we could also remove this function entirely (ie no longer recurse and building this type Ids map when we collect up our pallet/runtime etc types), and instead have a function which has the specific goal of searching through the metadata to look for the runtime types (and in that way, it doesn't need to allocate/collect anything).
b) On the other hand, keeping this function is "easier" because you need to recurse anyway to do the above searching, so maybe you end up with fewer lines of code by just keeping it around.
Personally I'd try (a) and see whether doing so makes things easier to follow or not, but not sure!
metadata/src/utils/retain.rs
Outdated
| || !matches!( | ||
| pallet, | ||
| PalletMetadataInner { | ||
| storage: None, | ||
| call_ty: None, | ||
| event_ty: None, | ||
| error_ty: None, | ||
| constants: map, | ||
| .. | ||
| } if map.is_empty() |
There was a problem hiding this comment.
Does this make any difference? I can't see why a pallet would exist that had nothing meaningful in it, but even if it did, is there any harm in leaving it there?
There was a problem hiding this comment.
Ah I am starting to see why this is kept but not understanding it yet; you are turning things to None above in some cases
metadata/src/utils/validation.rs
Outdated
| @@ -588,7 +584,7 @@ impl<'a> MetadataHasher<'a> { | |||
|
|
|||
| // Get the hashes of outer enums, considering only `specific_pallets` (if any are set). | |||
| // If any of the typed that represent outer enums are encountered later, hashes from `top_level_enum_hashes` can be substituted. | |||
| let outer_enum_hashes = OuterEnumHashes::new(metadata, self.specific_pallets.as_deref()); | |||
| let outer_enum_hashes = OuterEnumHashes::new(metadata); | |||
There was a problem hiding this comment.
So here we no logner care about the possibility of the enums like RuntimeCall etc being changed, but how can we avoid caring about this?
(either the variants and all of their types still exist, so the hashes are the same, or we have stripped some types and/or variants and then the hashes owuld be different unless we ignore them, wouldn't they?)
There was a problem hiding this comment.
list of pallets/variants types is completely dependant on the given set of pallets for a particular pallet or unpredictable
There was a problem hiding this comment.
I don't understand,
So what happens if Call::Balances(pallet_system::AccountData) variant exist and I only want retain the system pallet, does the Call::Balances get filtered out?
There was a problem hiding this comment.
Aight, I think this was opinionated and removed variants without checking that it was actually referenced by pallets, runtime apis?
We now only rely on TypeSet::retain_metadata(..) to achieve this
metadata/src/utils/retain.rs
Outdated
| fn collect_extrinsic_types(&mut self, extrinsic: &ExtrinsicMetadata) { | ||
| let mut ids = Vec::from([ | ||
| extrinsic.address_ty, | ||
| extrinsic.call_ty, | ||
| extrinsic.signature_ty, | ||
| extrinsic.extra_ty, | ||
| ]); | ||
|
|
||
| for signed in &extrinsic.signed_extensions { | ||
| ids.push(signed.extra_ty); | ||
| ids.push(signed.additional_ty); | ||
| } | ||
| for id in ids { | ||
| self.seen_ids.insert(id); | ||
| } | ||
| } | ||
|
|
||
| if let Some(ty) = pallet.event_ty { | ||
| type_ids.insert(ty); | ||
| /// Collect all type IDs needed to represent the runtime APIs. | ||
| fn collect_runtime_api_types(&mut self, api: &RuntimeApiMetadataInner) { | ||
| for method in api.methods.values() { | ||
| for input in &method.inputs { | ||
| self.seen_ids.insert(input.ty); | ||
| } | ||
| self.seen_ids.insert(method.output_ty); | ||
| } | ||
| } |
There was a problem hiding this comment.
These two functions don't recurse through any sub types. While I don't think anything should, if we did decide we wanted to recurse here then I guess these should, too?
There was a problem hiding this comment.
pushed a change that does traverse, a few more types added to retained set on average.
metadata/src/utils/retain.rs
Outdated
| let mut retained_set = TypeSet { | ||
| seen_ids: type_set | ||
| .seen_ids | ||
| .difference(&retained_set.seen_ids) | ||
| .copied() | ||
| .collect(), | ||
| }; |
There was a problem hiding this comment.
So it looks like you are:
- building a set of types to keep based on the pallets we want to keep (
retained_set) - cloning that set and then adding all of the other types we want to keep (runtime APIs etc) too (
type_set) - taking the difference between those (ie
type_set - retained_set? So just the other types without the pallet ones, which we could get by just not cloning above?). This difference is nowretained_set. - for pallets we want to filter, run
update_filtered_pallet, which roughly seems to keep things in the pallet if they reference types we've collected outside of the pallet stuff, and remove things if not. We then only discard pallets if no types that they directly reference are used elsewhere outside of pallets.
I might be wrong, but I don't really understand the reason for doing this? since you collected the outer enums, I can see that the first pallet we see that references those would actually be kept (which I guess has the effect of keeping the variant too?), but it feels like quite a long winded way to go about avoiding getting rid of variants if the outer enums are used somewhere, and I don't know if there is any other benefit :D
I also see that update_filtered_pallet calls retained_set.remove() each time it checks, so only the first pallet referencing eg RuntimeCall would end up having bits kept I think, and other pallets that reference RuntimeCall would be removed. Is that intentional?
Overall, I feel like we can remove a lot of this logic and rely on scale_info::PortableRegistry::retain for the recursive retaining of IDs we care about (because either that works ok already, or doesn't work and should be fixed). Then the logic can be something like:
- Add all of the type IDs from the pallets etc that we want to keep to a set
- For any return types or storage types in these things we are keeping, do a deep scan to see if we find the
RuntimeCall,RutimeEventorRuntimeErrorouter enums. If we find one, we can't strip any variants out of it - Now, call
PortableRegistry::retainto keep all of the type Ids we need (which will vary depending on whether those enums had variants stripped out of them or not and thus fewer IDs to recurse through).
All of that said though, if there is some value I'm not understanding in the way that we are keeping pallets etc here, then probably I'd need to be walked through it and we'd want to add a comment or two somewhere here to explain the approach and reasoning for it for our future selves :)
There was a problem hiding this comment.
taking the difference between those (ie type_set - retained_set? So just the other types without the pallet ones, which we could get by just not cloning above?). This difference is now retained_set.
The difference is the types present in the type_set and absent in retained_set. retained_set is not interesting in this case as types were already present in retained pallet. But i feel like we are talking about the same thing
but it feels like quite a long winded way to go about avoiding getting rid of variants if the outer enums are used somewhere, and I don't know if there is any other benefit :D
Its more about getting rid of pallets and outer enums, but only if they are not referenced anywhere. Its about stripping types that are not interesting to user without accidentally removing something that is required.
I also see that update_filtered_pallet calls retained_set.remove() each time it checks, so only the first pallet referencing eg RuntimeCall would end up having bits kept I think, and other pallets that reference RuntimeCall would be removed. Is that intentional?
I guess i'm misunderstanding something, shouldn't only one pallet with the type be enough or we can safely ignore those refs to runtime call/runtime error/runtime event and remove pallet based purely on storage/constant types? My intuition was to keep them as they might be referencing inner types inside outerenum variant, ie pallet.event_ty = 365 and metadata.outer_enum.event_ty = 22
All of that said though, if there is some value I'm not understanding in the way that we are keeping pallets etc here, then probably I'd need to be walked through it and we'd want to add a comment or two somewhere here to explain the approach and reasoning for it for our future selves :)
as above: My intuition was to keep them as they might be referencing inner types inside outerenum variant, ie pallet.event_ty = 365 and metadata.outer_enum.event_ty = 22
|
I had a look over and think I'm starting to understand the code more, though I still think that it's more complex than we need (but let's chat tomorrow and look at the code more to confirm). Essentially, So, the approach I'd lean towards with this is more like:
For working out whether some type contains an outer enum, you might end up with a recursive check which is a simplified version of yours, and doesn't need to allocate anything like: fn find_tys_in_id(id: u32, types: &PortableRegistry, unseen: &mut BTreeSet<u32>) {
unseen.remove(id)
let ty = types.resolve(id);
match ty.type_def {
TypeDef::Composite(TypeDefComposite { fields }) => {
for field in fields {
find_tys_in_id(field.ty.id, types, unseen)
}
}
TypeDef::Variant(TypeDefVariant { variants }) => {
for variant in variants {
for field in &variant.fields {
find_tys_in_id(field.ty.id, types, unseen)
}
}
}
TypeDef::Array(TypeDefArray { len: _, type_param })
| TypeDef::Sequence(TypeDefSequence { type_param })
| TypeDef::Compact(TypeDefCompact { type_param }) => {
find_tys_in_id(type_param.id, types, unseen)
}
TypeDef::Tuple(TypeDefTuple { fields }) => {
for field in fields {
find_tys_in_id(field.id, types, unseen)
}
}
TypeDef::Primitive(_) | TypeDef::BitSequence(_) => {
// End of recursion.
}
}
}I def think that the existing code can be made clearer w.r.t what steps it takes, and hopefully this extra check to see if we can strip outer enum types handles the edge case we've faced where we strip variants even though we need the fully types to decode something in Subxt. Note: If the outer enum types are inputs to some runtime API etc, we can still strip them I think. All this means is that users won't be able to express things about pallets that have been stripped when constructing such inputs. Everything should still encode ok. |
ok, so after fiddling this a bit more i think this suggested approach is the way to go. click me, wall of text`pallet_filter` - the pallet we want to keep. `name` - approach to traversal that was taken. `count` - amount of types retained in the registry after stripping.as visible from set below its the Note however that amount of types on average will increase compared to the old approach, e.g. for a dataset of 943 types it will be 644 instead of the 411 in the past. [
Collected {
pallets_filter: "System",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 581,
},
Info {
name: "traverse pallets",
count: 644,
},
Info {
name: "traverse extrinsics",
count: 581,
},
Info {
name: "traverse pallets & extrinsics",
count: 644,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 644,
},
],
},
Collected {
pallets_filter: "Babe",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 575,
},
Info {
name: "traverse pallets",
count: 575,
},
Info {
name: "traverse extrinsics",
count: 575,
},
Info {
name: "traverse pallets & extrinsics",
count: 575,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "Timestamp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Indices",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Balances",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 584,
},
Info {
name: "traverse pallets",
count: 584,
},
Info {
name: "traverse extrinsics",
count: 584,
},
Info {
name: "traverse pallets & extrinsics",
count: 584,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "Parameters",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "TransactionPayment",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Authorship",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Offences",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "Historical",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Session",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Grandpa",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "AuthorityDiscovery",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Treasury",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 634,
},
],
},
Collected {
pallets_filter: "ConvictionVoting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "Referenda",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 582,
},
Info {
name: "traverse pallets",
count: 582,
},
Info {
name: "traverse extrinsics",
count: 582,
},
Info {
name: "traverse pallets & extrinsics",
count: 582,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 645,
},
],
},
Collected {
pallets_filter: "FellowshipCollective",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 571,
},
Info {
name: "traverse pallets",
count: 571,
},
Info {
name: "traverse extrinsics",
count: 571,
},
Info {
name: "traverse pallets & extrinsics",
count: 571,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "FellowshipReferenda",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 581,
},
Info {
name: "traverse pallets",
count: 581,
},
Info {
name: "traverse extrinsics",
count: 581,
},
Info {
name: "traverse pallets & extrinsics",
count: 581,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 644,
},
],
},
Collected {
pallets_filter: "Origins",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Whitelist",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Claims",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Utility",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Identity",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 579,
},
Info {
name: "traverse pallets",
count: 579,
},
Info {
name: "traverse extrinsics",
count: 579,
},
Info {
name: "traverse pallets & extrinsics",
count: 579,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 643,
},
],
},
Collected {
pallets_filter: "Society",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 585,
},
Info {
name: "traverse pallets",
count: 585,
},
Info {
name: "traverse extrinsics",
count: 585,
},
Info {
name: "traverse pallets & extrinsics",
count: 585,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 648,
},
],
},
Collected {
pallets_filter: "Recovery",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 632,
},
],
},
Collected {
pallets_filter: "Vesting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Scheduler",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "Proxy",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 572,
},
Info {
name: "traverse pallets",
count: 572,
},
Info {
name: "traverse extrinsics",
count: 572,
},
Info {
name: "traverse pallets & extrinsics",
count: 572,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 636,
},
],
},
Collected {
pallets_filter: "Multisig",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Preimage",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 573,
},
Info {
name: "traverse pallets",
count: 573,
},
Info {
name: "traverse extrinsics",
count: 573,
},
Info {
name: "traverse pallets & extrinsics",
count: 573,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 637,
},
],
},
Collected {
pallets_filter: "AssetRate",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Bounties",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "ChildBounties",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Nis",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "NisCounterpartBalances",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 584,
},
Info {
name: "traverse pallets",
count: 584,
},
Info {
name: "traverse extrinsics",
count: 584,
},
Info {
name: "traverse pallets & extrinsics",
count: 584,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "ParachainsOrigin",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Configuration",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "ParasShared",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParaInclusion",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParaInherent",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "ParaScheduler",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 635,
},
],
},
Collected {
pallets_filter: "Paras",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 575,
},
Info {
name: "traverse pallets",
count: 575,
},
Info {
name: "traverse extrinsics",
count: 575,
},
Info {
name: "traverse pallets & extrinsics",
count: 575,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 639,
},
],
},
Collected {
pallets_filter: "Initializer",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Dmp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Hrmp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "ParaSessionInfo",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "ParasDisputes",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParasSlashing",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "MessageQueue",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 571,
},
Info {
name: "traverse pallets",
count: 571,
},
Info {
name: "traverse extrinsics",
count: 571,
},
Info {
name: "traverse pallets & extrinsics",
count: 571,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 634,
},
],
},
Collected {
pallets_filter: "OnDemandAssignmentProvider",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "CoretimeAssignmentProvider",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 573,
},
Info {
name: "traverse pallets",
count: 573,
},
Info {
name: "traverse extrinsics",
count: 573,
},
Info {
name: "traverse pallets & extrinsics",
count: 573,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 638,
},
],
},
Collected {
pallets_filter: "Registrar",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "Slots",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Auctions",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 632,
},
],
},
Collected {
pallets_filter: "Crowdloan",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Coretime",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "XcmPallet",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 583,
},
Info {
name: "traverse pallets",
count: 583,
},
Info {
name: "traverse extrinsics",
count: 583,
},
Info {
name: "traverse pallets & extrinsics",
count: 583,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "Beefy",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Mmr",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "MmrLeaf",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "IdentityMigrator",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "ParasSudoWrapper",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "AssignedSlots",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "ValidatorManager",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "StateTrieMigration",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "RootTesting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Sudo",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
] |
| @@ -214,7 +214,10 @@ impl Metadata { | |||
| MetadataHasher::new(self) | |||
| } | |||
|
|
|||
| /// Filter out any pallets that we don't want to keep, retaining only those that we do. | |||
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | |||
| /// Note: | |||
There was a problem hiding this comment.
I would rather create a dev-note using // to avoid this to end-up in the API docs
metadata/src/lib.rs
Outdated
| /// Filter out any pallets that we don't want to keep, retaining only those that we do. | ||
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | ||
| /// Note: | ||
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. |
There was a problem hiding this comment.
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. | |
| /// runtime api filtering will not lead to significant metadata size reduction because the return types are kept to ensure that those can be decoded. |
There was a problem hiding this comment.
runtimeApi filtering will lead to significant metadata size reductions.
If a runtimeApi returns an outerEnum type we no longer can strip the enum, because we don't know which variants are used.
metadata/src/lib.rs
Outdated
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | ||
| /// Note: | ||
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. | ||
| /// This happens because we will keep the types returned by runtime_api methods to guarantee that we can safely decode them. |
There was a problem hiding this comment.
why do we need to keep the return types if those are not used/trimmed?
There was a problem hiding this comment.
we can never know if they are used fully or partially, therefore we always keep the full type
metadata/src/utils/retain.rs
Outdated
| match entry.entry_type { | ||
| StorageEntryType::Plain(ty) => { | ||
| type_ids.insert(ty); | ||
| use alloc::collections::{BTreeMap, BTreeSet, VecDeque}; |
There was a problem hiding this comment.
Do you mind document the algorithm we use to filter out pallets/runtime APIs as top-level docs in this module?
Such as the tree/types, outer enums etc are traversed and in which order...
metadata/src/utils/validation.rs
Outdated
| // test that the hashes are the same: | ||
| let hash = MetadataHasher::new(&metadata) | ||
| .only_these_pallets(&["First"]) | ||
| .only_these_pallets(&name_set) |
There was a problem hiding this comment.
I guess we can revert this to "First" now to simplify?
There was a problem hiding this comment.
removed the test completetely as the behaviour in OuterEnumHashes relied on the fact that some of the variants in outernums are removed unconditionally.
| { | ||
| let set = "Balances,Timestamp,Contracts,ContractsEvm,System" | ||
| .split(",") | ||
| .collect::<BTreeSet<&str>>(); | ||
| move |s| set.contains(&s) | ||
| }, |
There was a problem hiding this comment.
I'd simplify this to just |s| ["Balances","Timestamp","Contracts","ContractsEVM","System"].contains(s); no need for the set etc really (and with so few items it's prob not even slower anyways, but being a test it's not a big deal)
There was a problem hiding this comment.
i written it like this only because repro command is a string and i was lazy to type
metadata/src/utils/retain.rs
Outdated
|
|
||
| #[test] | ||
| fn issue_1659() { | ||
| let metadata_cache = load_metadata("../artifacts/regressions/1659.scale"); |
There was a problem hiding this comment.
Nit: rename to full_metadata?
There was a problem hiding this comment.
which full_metadata? maybe we can name it 1659.full_metadata.scale?
There was a problem hiding this comment.
Sorry, I wasn't clear at all! I meant to rename metadata_cache to full_metadata :)
metadata/src/utils/retain.rs
Outdated
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn retain_one_runtime_api() { | ||
| let metadata_cache = load_metadata(); | ||
| let metadata_cache = load_metadata_polkadot(); |
There was a problem hiding this comment.
Nit: I'd call this just metadata too (not really this PR but hey :))
There was a problem hiding this comment.
renamed load_metadata_polkadot to load_metadata and introduced load_metadata_custom to load different files
There was a problem hiding this comment.
Sorry, I wasn't clear at all there! I was referring to the variable metadata_cache; it's not really a cache or anything ,it is just the metadata :)
|
One of the things that was bugging me a bit in this code prior to this PR is that we have separate Something that I find a bit hard to do still in this PR is following exactly the steps that we take to strip types. I think this was also a bit messy before the PR too. Anyway, I got to wondering whether something like the following would be ultimately much clearer and easier to follow, while avoiding duplicating the logc around collect vs update: pub fn retain_metadata<F, G>(
metadata: &mut Metadata,
mut pallets_filter: F,
mut runtime_apis_filter: G,
) where
F: FnMut(&str) -> bool,
G: FnMut(&str) -> bool,
{
// 1. Delete pallets we don't want to keep.
metadata.pallets.retain(|pallet| pallets_filter(&pallet.name));
metadata.pallets_by_index = metadata
.pallets
.values()
.iter()
.enumerate()
.map(|(pos, p)| (p.index, pos))
.collect();
// 2. Delete runtime APIs we don't want to keep.
metadata.apis.retain(|api| runtime_apis_filter(&api.name));
// 3. For each outer enum type, strip it if possible, ie if it is not returned by any
// of the things we're keeping (because if it is, we need to keep all of it so that we
// can still decode values into it).
let outer_enums = metadata.outer_enums();
for outer_enum_ty_id in [outer_enums.call_enum_ty(), outer_enums.error_enum_ty(), outer_enums.event_enum_ty()] {
if !find_type_id_in_return_pos(metadata, outer_enum_ty_id) {
retain_variants_in_enum_type(metadata, &pallets_filter);
// ^ prob clearer renamed to "strip_variants_in_enum_type"
}
}
// 4. Collect all of the type IDs we still want to keep after deleting.
let mut keep_these_type_ids: BTreeSet<u32> = iterate_metadata_types(metadata).map(|id| *id).collect();
// 5. Additionally, subxt depends on the `DispatchError` type existing; we use the same
// logic here that is used when building our `Metadata` to ensure we keep it too.
let dispatch_error_ty = metadata
.types
.types
.iter()
.find(|ty| ty.ty.path.segments == ["sp_runtime", "DispatchError"])
.expect("Metadata must contain sp_runtime::DispatchError");
keep_these_type_ids.insert(dispatch_error_ty.id);
// 5. Strip all of the type IDs we no longer need, based on the above set.
let map_ids = metadata.types.retain(|id| keep_these_type_ids.contains(&id));
// 6. Now, update the type IDs referenced in our metadata to reflect this.
for id in iterate_metadata_types(metadata) {
if let Some(new_id) = map_ids.get(id) {
*id = *new_id;
} else {
panic!("Type id {id} was not retained. This is a bug");
}
}
}
/// return an iterator handing back a mutable reference to each type ID seen in the metadata (not recursively).
/// This will iterate over every type referenced in the metadata outside of `metadata.types`.
fn iterate_metadata_types(metadata: &mut Metadata) -> impl Iterator<Item = &mut u32> {
unimplemented!()
}
/// Look for a type ID anywhere that we can be given back, ie in constants, storage or runtime API return types.
/// This will recurse deeply into those type IDs to find them (but doesn't need to allocate).
fn find_type_id_in_return_pos(metadata: &mut Metadata, type_id: u32) -> bool {
unimplemented!()
}I think this would be something like my "north star" for a readable and hopefullyfairly easy to follow implementation of stripping pallets/runtime APIs. That all said, I think we can aim to get this PR merged and havea go as a follow up step to do something like this. |
metadata/src/utils/retain.rs
Outdated
|
|
||
| /// return an Vec handing back a mutable reference to each type ID seen in the metadata (not recursively). | ||
| /// This will iterate over every type referenced in the metadata outside of `metadata.types`. | ||
| fn iterate_metadata_types(metadata: &mut Metadata) -> alloc::vec::Vec<&mut u32> { |
There was a problem hiding this comment.
This looks good for now, as actually iterating is much more ocmplicated and requires you to keep an internal state machine etc!
One thing I'd be tempted to do, since it's called iterate_metadata_types and only needed for iterating is to change the return type to -> impl Iterator<Item=&mut u32> and then call types.into_iter() at the end ofthe body to make that work.
(That way we can also optimise. the internals at some point without caring about anything else)
metadata/src/utils/retain.rs
Outdated
|
|
||
| /// Look for a type ID anywhere that we can be given back, ie in constants, storage, extrinsics or runtime API return types. | ||
| /// This will recurse deeply into those type IDs to find them. | ||
| pub fn collect_return_types<F, G>( |
There was a problem hiding this comment.
Nit: rename to something like keep_outer_enum_type (because it's specifically handing back a function for that)?
Also all of the TypeSet stuff above is only used in here now I think? Personally I'd prob move the functions from there into here so we end up with something like:
fn collect_return_types(..) {
fn collect_types_recursively(&mut set: BTreeSet, metadata, id) {
// TypeSet::collect_types impl
}
fn collect_extrinsic_types(&mut set: BTreeSet, extrinsic) {
// TypeSet::collect_extrinsic_types impl
}
// same for collect_runtime_api_types and collect_pallet_types
// Gather all of the types that could contain an outer enum:
let mut types = BTreeSet::new();
for pallet in metadata.pallets.values() {
if pallets_filter(&pallet.name) {
collect_pallet_types(&mut types, pallet, metadata);
}
}
for api in metadata.apis.values() {
if runtime_apis_filter(&api.name) {
collect_runtime_api_types(&mut types, metadata, api);
}
}
type_set.collect_extrinsic_types(&mut types, &metadata.extrinsic);
// Return a function to check whether some outer enum type is seen
move |type_id| types.contains(type_id)
}Bit "messier" but it hides all of the "internal" logic of how we do it away behind your nice function signature, and thus lets us tweak it in the future without any fear that anything else will break.
metadata/src/utils/retain.rs
Outdated
| fn new() -> Self { | ||
| Self { | ||
| seen_ids: BTreeSet::new(), | ||
| work_set: VecDeque::with_capacity(22), |
There was a problem hiding this comment.
No particular reason besides that the biggest struct I've seen in the wild had ~20 fields
There was a problem hiding this comment.
Worth a comment about this perhaps!
There was a problem hiding this comment.
yeah, would be good with a comment because I will ask the same everytime I see that :D
There was a problem hiding this comment.
added a comment, also simplified a bit to keep workset size smaller, now its usually around 30-50 elements
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
|
|
||
| /// Returns an iterator that allows modifying each type ID seen in the metadata (not recursively). | ||
| /// This will iterate over every type referenced in the metadata outside of `metadata.types`. | ||
| fn iterate_metadata_types(metadata: &mut Metadata) -> impl Iterator<Item = &mut u32> { |
There was a problem hiding this comment.
I wonder whether we can name this function to something better to explain that you get a mutable iterator to modify the metadata.
What I can come up with is iter_mut(&mut Metadata), metadata_types_iter_mut(&mut Metadata) 🤷
Description:
See #1659
This fixes the error described inside the issue, however it will strip types in a bit different way which will result in slightly less savings on average.
Biggest difference between the new and old approaches is that:
outerenumenums, unless there are some variants that do not reference any types we intend to retain.Size comparison with the previous stripping approach:
size reduction Comparison:
Comparison for old and new stripping methods. be warned, it's 1K lines