How to configure optimizer that uses model parameters for further instantiation?

[chekhovana]

I would like to instantiate both model and optimizer directly from config, but can't get how to pass model parameters to optimizer constructor. Below is my attempt that raises the error.

model:
  _target_: timm.create_model
  model_name: resnet18
  pretrained: False
  num_classes: 1

optimizer:
  _target_: torch.optim.AdamW
  lr: 3e-4
  params:
    _target_: ${model}.parameters

Is it possible to achieve?

[Jasha10]

Hi @chekhovana,
I'm sorry to say that currently there is no clean way to do this with Hydra.
Doing this would require singleton support to be implemented.

Without singleton support, referencing ${model} inside of optimizer would likely result in two different instances of model being instantiated (instead of the same model instance being used in two places).

In the future, singleton support may be implemented. For future reference, I'll give some details on how this could be done if singleton support were available.

---

If singleton support were available, passing in parameters to optimizer might look like this:

model:
  _target_: timm.create_model
  model_name: resnet18
  pretrained: False
  num_classes: 1
  _singleton_: true  # requires https://github.com/facebookresearch/hydra/issues/1393

optimizer:
  _target_: torch.optim.AdamW
  lr: 3e-4
  params: ${getattr:${model},"parameters"}

In this example, getattr is a custom resolver that looks up the "parameters" attribute on the model instance:

OmegaConf.register_new_resolver("getattr", getattr)

[chekhovana]

@Jasha10, thanks for the detailed response.

[omry]

What about utilizing partial instantiation support?
This use case was, iirc, the primary reason for that feature.

[chekhovana]

Sorry, I'v missed your reply, thanks for the advice.