Dataset class improperly handles images with multiple objects

[TasinIshmam]

Describe the bug

If a single image contains more than one object, then the dataset class treats each object as a separate item in the dataset. As a result, if there are 'n' objects for any image 'i', then the same image 'i' is repeated in the dataset 'n' times, one time for each object.

This happens for both instantiation methods (with csv and without csv file). Looking at the detecto source code, this happens because the __getitem__ method of the Dataset class treats each entry in the csv or dataframe as a separate image instance (even though the same image may be in the csv/dataframe multiple times if there are multiple objects in the image).

Code and Data

I am using the Dhaka-AI traffic dataset. The dataset is labelled in PASCAL VOC format, with labels and annotations both residing inside the train folder.

I ran the following code to verify the bug

from detecto import utils, core, visualize

utils.xml_to_csv('train', 'train.csv')
dataset = core.Dataset('train.csv', 'images/')

for i in range ( 0, 5) 
    image, target = dataset[i]
    visualize.show_labeled_image(image, target['boxes'], target['labels'])  
# all 5 images are of the same first image in the dateset, each with bounding box around one of the annotated objects. 

Environment:

• OS: Ubuntu
• Python version: 3.6.9
• Detecto version: detecto-1.1.1 (installed from pip) AND detecto 1.1.6 (cloned from github)
• torch version: 1.6.0
• torchvision version: 0.7.0

I'd be happy to work on this issue. I have already modified the detecto source code locally to handle this exact problem so I can make a PR to fix it.

Additional context

I'm attaching a sample annotation file from the dateset for reference.

<?xml version='1.0' encoding='utf8'?>
<annotation>
	<folder>train</folder>
	<filename>01.jpg</filename>
	<path>./train/01.jpg</path>
	<source>
		<database>Unknown</database>
	</source>
	<size>
		<width>1200</width>
		<height>800</height>
		<depth>3</depth>
	</size>
	<segmented>0</segmented>
	<object>
		<name>bus</name>
		<pose>Unspecified</pose>
		<truncated>1</truncated>
		<difficult>0</difficult>
		<bndbox>
			<xmin>833</xmin>
			<ymin>390</ymin>
			<xmax>1087</xmax>
			<ymax>800</ymax>
		</bndbox>
	</object>
	<object>
		<name>bus</name>
		<pose>Unspecified</pose>
		<truncated>0</truncated>
		<difficult>0</difficult>
		<bndbox>
			<xmin>901</xmin>
			<ymin>284</ymin>
			<xmax>1018</xmax>
			<ymax>395</ymax>
		</bndbox>
	</object>
	<object>
		<name>bus</name>
		<pose>Unspecified</pose>
		<truncated>0</truncated>
		<difficult>0</difficult>
		<bndbox>
			<xmin>909</xmin>
			<ymin>241</ymin>
			<xmax>1010</xmax>
			<ymax>287</ymax>
		</bndbox>
	</object>
	<object>
		<name>motorbike</name>
		<pose>Unspecified</pose>
		<truncated>0</truncated>
		<difficult>0</difficult>
		<bndbox>
			<xmin>863</xmin>
			<ymin>241</ymin>
			<xmax>882</xmax>
			<ymax>268</ymax>
		</bndbox>
	</object>
</annotation>

[langejoh]

I just realized this problem too. How much of the source code has to be rewritten to fix it?

[alankbi]

Thanks for bringing this up. @TasinIshmam if you have the code for this, definitely feel free to submit a PR :)

[TasinIshmam]

I just realized this problem too. How much of the source code has to be rewritten to fix it?

@langejoh this might not be comprehensive but from my understanding the following changes need to be made.

detecto.utils

• xml_to_csv() function needs to be modified. Each unique object needs to have some kind of "object_id" index field associated in the csv so that it can be used to identify all bounding boxes associated with that specific object from the generated csv.

detecto.core

• getitem() in Dataset class needs to be modified to reflect changes in the csv format. Additionally, code for handling some of the transforms also need to be slightly changed.
• _convert_to_int_labels() in Model class needs to be modified to handle multiple labels per object.

There might be other changes required in detecto.visualize as well. I will have to look further into the code to find out.

Thanks for bringing this up. @TasinIshmam if you have the code for this, definitely feel free to submit a PR :)

@alankbi alright thanks. Please give me a few days to document the changes and clean up my code. I'l submit a PR afterwards.

[TasinIshmam]

Hi @alankbi, I wanted to verify my changes are okay with all existing tests. However, I'm not able to run test_core.test_model_predict() because I don't have the "model.pth" file in detecto/tests/static.

I could generate a model of my own but I wanted to keep the tests as consistent as possible. Could you please provide me that file in some way?

[alankbi]

Sure - here is a link to the model file (this will automatically download the file when you click on it):