Fix broken remove_parameterization in firefly

[med1844]

Is this PR adding new feature or fix a BUG?

This is a developer-side bug fix that addresses an issue with VQGAN parameterization removal. This bug does not affect end users under normal circumstances, as remove_parameterization is not used in any other part of the codebase.

The following code (forked from tools/vqgan/inference.py) triggers the error:

model: FireflyArchitecture = load_model(config_name, checkpoint_path, device=device)
model.remove_parametrizations()

Error message:

ValueError: Module FishTransConvNet(
  (conv): ParametrizedConvTranspose1d(
    512, 256, kernel_size=(16,), stride=(8,)
    (parametrizations): ModuleDict(
      (weight): ParametrizationList(
        (0): _WeightNorm()
      )
    )
  )
) does not have a parametrization on weight

Is this pull request related to any issue? If yes, please link the issue.

No

[leng-yue]

LGTM