New OpenCLIP Ecommerce & Retail Models - Image Classification Example

[ellie-sleightholm]

New OpenCLIP Ecommerce & Retail Models - Image Classification Example

Introduction

Marqo have recently announced two new state-of-the-art embedding models for multimodal fashion search and recommendations: Marqo-FashionCLIP & Marqo FashionSigLIP. These models are supported in open-clip so I thought I'd create a Show and Tell of these models in action 🔥

For the full code and a guided walk-through visit this article or try them out yourself in Google Colab!

1. Install Libararies

pip install open_clip_torch
pip install datasets

2. Load a Dataset

from datasets import load_dataset

# Load the dataset
ds = load_dataset('ceyda/fashion-products-small')

image = entry['image']
dataset = ds['train']

3. Load Marqo-FashionCLIP Model and Preprocessing

import open_clip
import torch

# Load the Marqo/marqo-fashionCLIP model and preprocessors
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

4. Perform Image Classification with Marqo-FashionCLIP

import matplotlib.pyplot as plt

# Select indices for three example images
indices = [1, 17, 23]

# Get the list of possible subcategories from the dataset
subcategories = list(set(example['subCategory'] for example in dataset))

# Preprocess the text descriptions for each subcategory using the tokenizer
text_inputs = tokenizer([f"a photo of {c}" for c in subcategories]).to(device)

# Create a figure with subplots
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# Loop through the indices and process each image
for i, idx in enumerate(indices):
    # Select an example image from the dataset
    example = dataset[idx]
    image = example['image']
    subcategory = example['subCategory']

    # Preprocess the image
    image_input = preprocess_val(image).unsqueeze(0).to(device)

    # Calculate image and text features
    with torch.no_grad():
        image_features = model.encode_image(image_input)
        text_features = model.encode_text(text_inputs)

    # Normalize the features
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    # Calculate similarity between image and text features
    similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)
    values, indices = similarity[0].topk(1)

    # Display the image in the subplot
    axes[i].imshow(image)
    axes[i].set_title(f"Predicted: {subcategories[indices[0]]}, Actual: {subcategory}")
    axes[i].axis('off')

# Show the plot
plt.tight_layout()
plt.show()

Conclusion

In this post, we've used Marqo-FashionCLIP for image classification on a small fashion dataset. We performed image classification for three specific images from the dataset and computed the predicted sub-category for each image and compared these with the true sub-category.

Additional Resources

Marqo-Fashion CLIP on Hugging Face
Marqo-Fashion CLIP on GitHub
Article: Marqo Launches Family of Embedding Models for Ecommerce and Retail