List of partial instances

[raphCode]

I want to have a list of Sub-Configs where Python classes are referenced and turned into partial instances.
I figured out the structured config part so far:

from omegaconf import OmegaConf

@define
class PartialInstance:
    _target_: str
    _partial_: bool = True

@define:
class ConfigSchema:
    instances: list[PartialInstance]

structured = OmegaConf.structured(ConfigSchema)

I want to call hydra.utils.instantiate once on the whole config so everything is instantiated or transformed into partials. For this I obviously need to merge the structured config with the user config.
How can I achieve this in hydra / omegaconf so that:

• the _target_ key is enforced so that user has to provide it
• new keys can be inserted, like arguments to the instantiated class
• the _partial_ key can't be overriden by the user (like to False)

I tried experimenting with the merge order (omegaconf) / defaults list (hydra):
related: omry/omegaconf#412 (comment)

• structured config first:
  • can't add new keys (related: Deprecate subclassing Dict omry/omegaconf#663)
    • OmegaConf.set_struct(structured, False) did not help
  • _partial_ can be overridden
    • can be restored by iterating the config and merging the default values back in from the dataclasses
• user config first:
  • The structured config PartialInstance is not retained in the list: OmegaConf.get_type(cfg.instances[0]) reports dict
    • no default values like _partial_ in the merged config
    • the restore trick does not work because get_type() does not report the dataclass

[raphCode]

OmegaConf.set_struct(structured, False) does not work because the merge checks only the DictConfig corresponding to the structured config for the struct flag, not inheriting the flag settings from the parent.
I also cannot set the flag on the config beforehand, because the DictConfig corresponding to PartialInstance gets created in the merge process. In fact, it has to be, because only the merge knows the length of the list!

[raphCode]

merging user config first
The structured config PartialInstance is not retained in the list

I opened a bug about that: omry/omegaconf#1058

I think this would solve my problem, since it would allow extra keys from the user config and carry type information into the resulting config. Default values are still not merged, but this is by design: omry/omegaconf#412 (comment)
With the types available, I can then iterate through the config and reset all values to the defaults from the dataclasses (see omry/omegaconf#412 (comment))