Integration of OpenCLIP into FiftyOne Model Zoo

[AdonaiVera]

Hello FiftyOne Team and Community,

I'm thrilled to announce the integration of OpenCLIP into the FiftyOne Model Zoo. This addition enriches the Model Zoo with the capabilities of OpenCLIP, a powerful vision foundation model.

About the Integration:

• OpenCLIP, accessible via MLFoundations' GitHub repository, is now integrated into FiftyOne. This allows users to leverage OpenCLIP's advanced vision capabilities within the familiar FiftyOne workflows.
• The integration has been carefully implemented to ensure compatibility and ease of use with existing FiftyOne functions.

Demo Notebook:

• To demonstrate the capabilities and usage of OpenCLIP within FiftyOne, I have prepared a comprehensive Jupyter notebook. This notebook guides users through various functionalities, showcasing how OpenCLIP can be utilized effectively in vision tasks. Demo FiftyOne + OpenClip

Available Models in OpenCLIP:

• The integration includes support for various models available in OpenCLIP. A detailed list and their potential applications are included in the demo notebook.

Installation and Usage:

• Installation instructions and usage guidelines are provided in the notebook, ensuring users can easily get started with OpenCLIP in FiftyOne.

Closing:

• I am open to feedback and further discussions to refine this integration. Please feel free to reach out for any clarifications or suggestions.
• I'm excited to contribute to the FiftyOne community and look forward to seeing how this integration facilitates advanced vision research and applications.

Cheers,
Adonai Vera

[swheaton]

Hi @AdonaiVera thank you for the contribution, we appreciate it!! OpenCLIP has been near the top of our list of must-haves in the model zoo!

It looks like you've included most of the source code from open_clip. We should not directly copy the source code but rather import the package.

Could you try that, then push again here? It will make it much easier to review this PR also 😄

[AdonaiVera]

Hi @swheaton 👋
Thanks a ton for the feedback! Totally agree, using the pip version of OpenCLIP is way simpler. I've already made the switch to importing the package instead of using the source code directly.

I've pushed the changes to the PR. Could you take a look when you have a moment? Also, I'm ready to integrate other methods into FiftyOne if that's helpful.

Just let me know 🚀

[swheaton]

Awesome thank you 🙇🏼
We will take a look at it.

Also, I'm ready to integrate other methods into FiftyOne if that's helpful.

Do you mean other OpenCLIP methods or other models into model zoo?

[AdonaiVera]

I'm open to contributing in any way that best serves the computer vision community. This could include adding more OpenCLIP methods or introducing new models to the model zoo.

[codecov]

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (4716506) 15.87% compared to head (4e3d9df) 15.86%.
Report is 49 commits behind head on develop.

Files	Patch %	Lines
...core/src/plugins/SchemaIO/utils/generate-schema.ts	0.00%	15 Missing ⚠️
.../src/components/Sidebar/Entries/PathValueEntry.tsx	0.00%	11 Missing ⚠️
...e/src/plugins/SchemaIO/components/InferredView.tsx	0.00%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3925      +/-   ##
===========================================
- Coverage    15.87%   15.86%   -0.01%     
===========================================
  Files          731      731              
  Lines        81557    81665     +108     
  Branches      1093     1093              
===========================================
+ Hits         12944    12959      +15     
- Misses       68613    68706      +93     

Flag	Coverage Δ
app	15.86% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brimoor]

@AdonaiVera thank you for this contribution! Amazing! 🎉 💪

[brimoor]

are model_name and pretrained required? If the OpenCLIP package defaults to something, you could update to this:

self.model_name = self.parse_string(d, "model_name", default=None)
self.pretrained = self.parse_string(d, "pretrained", default=None)

so that users could just do this:

model = foz.load_zoo_model("open-clip-torch")

without needing to know a particular pretrained model to load.

[AdonaiVera]

Hi @brimoor,

Thanks a lot for your feedback! I'm working on the adjustments right now. Just to clarify, the 'pretrained' parameter is optional, but specifying the model name is necessary. To streamline the process, I'm setting the default model to 'ViT-B-32'. This should make it more user-friendly while ensuring essential parameters are in place.

[AdonaiVera]

Hi @swheaton @brimoor , I've applied the suggestions and made the necessary updates. I've also refined the demo for a smoother experience. Please let me know if there are any other suggestions or improvements needed.

Thanks for the guidance 🚀

[Seedmanc]

waiting for the merge, I need this

[brimoor]

@AdonaiVera I'm going to go ahead and merge this so that we can include it in fiftyone==0.23.2 which is releasing today! I'll make the model_name -> clip_model_name change for you 😄

Thanks again for this great contribution!

[brimoor]

@AdonaiVera just a heads up: in testing this PR, I found that I needed to provide a pretrained value or else the model produced nonsense results (probably randomly initialized weights?)

So I updated the defaults in the version that will ship with fiftyone==0.23.2 to:

        clip_model ("ViT-B-32"): the Open CLIP model to use
        pretrained ("openai"): the pretrained version to use

[AdonaiVera]

Actually, that pre-trained value is optional according to the documentation but it's okay to set OpenAI as the default value. Thank you for making the modification 💪

Looks amazing, thank you @brimoor 🚀 💯