[LOCA] Projection head pre-trained weights in loca_vbase_imnet1k are almost zeros. Is this a correct behavior?

[KJ-rc]

Hi,
In project LOCA, I found the pre-trained weights of the projection head are almost zeros.
The minimum reproducible code with the downloaded checkpoint is as below.

import numpy as np
from flax.training import checkpoints

checkpoint_path = 'loca_vbase_imnet1k'
jax_weights = checkpoints.restore_checkpoint(checkpoint_path, target=None)
for k, w in jax_weights['teacher_weights']['output_projection'].items():
    for n, v in w.items():
        print(k, n, v.shape, abs(v).min(), v.mean())

And, the corresponding results are

Dense_0 bias (2048,) 0.0 0.0
Dense_0 kernel (768, 2048) 0.0 -3.5370087e-30
Dense_1 bias (2048,) 0.0 0.0
Dense_1 kernel (2048, 2048) 1.9844132e-33 5.522772e-30
Dense_2 bias (256,) 0.0 0.0
Dense_2 kernel (2048, 256) 5.721843e-33 -1.2184184e-29
prototypes kernel (256, 4096) 4.5964833e-32 -2.4790736e-29

Are they supposed to be such small?