Finetuning Fast Vit model

[Tanmay98]

How can i download the pretrained model weights file .pth for Fast Vit model similar to Deit that looks like (https://dl.fbaipublicfiles.com/deit/deit_tiny_patch16_224-a1311bcf.pth)?

Because when I go in the fastvit.py file the url link is missing in _cfg dictionary

[rwightman]

many new weights are only on the Hugging Face hub with a hf_hub entry that defaults to timm/model_name path, ex https://huggingface.co/timm/fastvit_t12.apple_in1k

[Tanmay98]

I tried to call the model in such a fashion but I am getting an unpickling error while loading the model from checkpoint. Below is the code and the error. Can you help me?

Code:

from timm.models.fastvit import FastVit as TimmFastVit
from timm.models.fastvit import _cfg as _cfg2

m2 = TimmFastVit(layers=(2, 2, 6, 2), embed_dims=(64, 128, 256, 512), mlp_ratios=(3, 3, 3, 3), token_mixers=("repmixer", "repmixer", "repmixer", "repmixer"))
m2.default_cfg = _cfg2
checkpoint = torch.hub.load_state_dict_from_url( url="https://huggingface.co/timm/fastvit_t12.apple_in1k", map_location="cpu", check_hash=True ) m.load_state_dict(checkpoint["model"])

Error:

UnpicklingError Traceback (most recent call last)
/home/tbaweja/811/model.ipynb Cell 20 line 6
1 m2 = TimmFastVit(layers=(2, 2, 6, 2),
2 embed_dims=(64, 128, 256, 512),
3 mlp_ratios=(3, 3, 3, 3),
4 token_mixers=("repmixer", "repmixer", "repmixer", "repmixer"))
5 m2.default_cfg = _cfg2
----> 6 checkpoint = torch.hub.load_state_dict_from_url(
7 url="https://huggingface.co/timm/fastvit_t12.apple_in1k",
8 map_location="cpu", check_hash=True
9 )
10 m.load_state_dict(checkpoint["model"])

File /home/tbaweja/anaconda3/envs/project/lib/python3.8/site-packages/torch/hub.py:770, in load_state_dict_from_url(url, model_dir, map_location, progress, check_hash, file_name, weights_only)
768 if _is_legacy_zip_format(cached_file):
769 return _legacy_zip_load(cached_file, model_dir, map_location, weights_only)
--> 770 return torch.load(cached_file, map_location=map_location, weights_only=weights_only)

File /home/tbaweja/anaconda3/envs/project/lib/python3.8/site-packages/torch/serialization.py:1028, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
1026 except RuntimeError as e:
1027 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1028 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File /home/tbaweja/anaconda3/envs/project/lib/python3.8/site-packages/torch/serialization.py:1246, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
...
-> 1246 magic_number = pickle_module.load(f, **pickle_load_args)
1247 if magic_number != MAGIC_NUMBER:
1248 raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, '<'.`