product_matching_with_shelf_images

Problem Statement

Given a set of images captured in grocery stores(shelf image), and a set of close up images of products in those stores, your task is as follows.

1. For every product image, find the location of that product in all shelf images in which it appears.
2. For every shelf image, locate all products and assign the name from given set of product images.

Find the whole soluction look for master branch

Dataset: https://www.kaggle.com/datasets/amanindiamuz/store-shelf-images-and-product-images-for-retial

Product Image

Shelf Image

Approach : Feature Extraction and Matching

Feature Matching is one of the widely used area of Computer vision with countless applications. In this approach, I have used "Brute-Force Matching with ORB Descriptors" for solving the probem statement. Well, I have tried other methods too, Including:

• Harris Corner
• SIFT(Scale Invariant Feature Transform)
• SURF(Speeded Up Robust Feature)
• FAST(Features from Accelerated Segment Test)

But, Satisfactory results were from "Brute-Force Matching with ORB Descriptors". Here are the steps included in the approach.

Steps included in the approach are :
• Reading Product and Shelf Images
• Cropping product images
• Removing Background for feature Extraction
• Extracting features using ORB Descriptors
• Matching features using BFMatcher
• Visualization
• Saving detected product cordinates in soluction.txt file

Approach: Deep Learning

As per the problem statement, the major/logical part of the problem is object recognition.
Recognizing the product on the shelf (Image), saving the coordinates of bounding boxes, and vice-versa.

Well,
One approach can be Feature mapping.

Using SIFT algorithm to match the features of the object in the image and based on that Saving the coordinates of the same.
Tough, there are limitations of Feature mapping technique, like different lightning conditions, Image clarity, Size of Image and others.

Thus,
For a robust model and higher recognition accuracy, there needed something complex which works with a most possible scenarios by reducing the chance of error.

Thus,
Training deep learning neural Network is prefrable.

Over here,
My approach is to have a neural network trained on product images and then using pretrained model of object detection from shelf images to recognize the product.

So,
For training neural network, I used Transfer learning. Use pretrained model and trained it further on the product dataset, Where dataset was upscaled using data Argumentation. Then using pre-trained YOLOv5 network that detects items from shelf images, Then further object matching.

Here are the Steps in detail that involve in building the soluction.

Steps included in the approach are :

• Reading Product and Shelf Images
• Exploratory Data Analysis
• Labelling the Product Data
• Image Augmentation
• Splitting the Data for Training and Testing
• Training pretrained Neural Network - ResNet (Transfer learning)
• Testing and accuracy Measure
• Using a pre-trained YOLOv5 network that detects items from shelf images.
• Use similarity between products to match product in shelf image : Resnet network has been used for this purpose.
• Writing in the soluction.txt file

For Further details, Kindly Look for Jupyter Notebook Attached.