🆕 Integrate Foundation Models Available VIA timm: UNI, Virchow, Hibou, H-optimus-0
#856
Conversation
|
Only added UNI and Prov-GigaPath for now. Will add more after initial comments. I do not like that |
|
I found this file for testing: https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/tests/models/test_arch_vanilla.py There might be a problem regarding memory and compute resources when running some of the larger feature extractors through GitHub actions, e.g. Prov-GigaPath needs a considerable amount of memory just to be loaded. |
As this is just testing the functionality and not loading weights, I hope this would work. |
In that case, you can inherit PatchPredictor uses all the functionalities of EngineABC other than the ones defined explicitly.
|
allows to reuse the `infer_batch` method of `CNNBackbone`
shaneahmed
changed the title
timm: UNI, Virchow, Hibou, H-optimus-0
|
I have updated this branch to make sure that tests pass on Ubuntu-24 before we merge it with develop. |
- remove explicit assert statement for `timm` version - add `timm` version into in if statement for prov-gigapath - add comment about `timm` version for timm-based models Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
for more information, see https://pre-commit.ci
was not removed while accepting suggested changes
| "hf_hub:prov-gigapath/prov-gigapath", pretrained=True | ||
| ) | ||
| else: | ||
| msg = f"Architecture {arch_name} not supported. If you are loading timm models, only timm version > `1.0.3` are supported." |
There was a problem hiding this comment.
This line is too long and causing the tests to fail. If you do pre-commit install in your development environment then it will test formatting before making a commit. You can also run ruff manually in your terminal before making a commit.
| msg = f"Architecture {arch_name} not supported. If you are loading timm models, only timm version > `1.0.3` are supported." | |
| msg = f"Architecture {arch_name} not supported." | |
| f"If you are loading timm models, only timm version > `1.0.3` are supported." |
Making a pull request as discussed in issue #855
Copied from the issue:
I think it would be useful to integrate pre-trained foundation models from other labs into tiatoolbox.models.architecture.vanilla.py.
Currently, the
_get_architecture()function allows the use of models fromtorchvision.models.But another function
_get_timm_architecture()could be made to incorporate foundation models which are available fromtimmwith weights on HuggingFace Hub. All the models fromtimmthat I've used require users to sign the licence agreement with the authors, so the licencing question seems to be solved itself since there is no way users will get access to the model weights just through Tiatoolbox without getting the access request approved by the authors first.