Add tests for datasets

[fmassa]

We should add unit tests for datasets. This will enable the codebase to be more reliable, and I expect that adding new datasets to torchvision will be a more straightforward procedure for the reviewer, as he will be able to rely on CI for a lot of the code checking.

We need to define how we will structure the tests for the datasets in order to avoid having to download the full datasets in memory, which will probably make the CI flaky due to network issues.

Here are a few alternatives:

1 - Add dummy test files for each dataset

Every dataset that has a download option should use a (newly introduced) function that performs the downloading and the extraction. The dummy test files should have the same folder structure as the original files after extraction. This way, any postprocessing of the extracted files and be tested by the testing suite.

The dummy files would be fake images (e.g., of 2x3 pixels), but that enables to test both the __init__ and the __getitem__ of the dataset in a straightforward way, which are the only functions that we guarantee that should be implemented for datasets.

Drawback

While this is a very flexible alternative, adding a new dataset adds an extra burden, as now the contributor also need to create a set of fake data that should be commited into the repo.

2 - Add an option to load images / data from a buffer

There is a common pattern in torchvision datasets.
1 - download / extract the original dataset
2 - get a mapping of image_id -> image_path / class_id / etc
3 - load image (either from memory or from the image_path) and annotations

the download / extract parts should be factored out into a common function, which handles zip / tar / etc.

If we add an extra method to the datasets get_paths_for_idx that gives the mapping image_id -> image_path / class_id / etc, we could then patch this method during testing so that instead of returning an image_path, we return instead a file-like object, which can be read by PIL, scipy.io.loadmat, xml.etree.ElementTree and other Python libs without problems.
This will require that we specify somewhere what is the type of values that get_paths_for_idx return, so that we how how to generate random data that fits to what the dataset expect.

Drawback

We might end up adding extra complexity while defining a dataset, and plus we might not even test all possible codepaths.

3 - Factor out dataset-specific functions and test only those

This would be an indirect testing, where we would split the dataset into standalone functions and test each function with some dummy inputs.
For example, in

vision/torchvision/datasets/mnist.py

Lines 311 to 328 in d4a126b

	def read_label_file(path):
	with open(path, 'rb') as f:
	data = f.read()
	assert get_int(data[:4]) == 2049
	length = get_int(data[4:8])
	parsed = np.frombuffer(data, dtype=np.uint8, offset=8)
	return torch.from_numpy(parsed).view(length).long()
	def read_image_file(path):
	with open(path, 'rb') as f:
	data = f.read()
	assert get_int(data[:4]) == 2051
	length = get_int(data[4:8])
	num_rows = get_int(data[8:12])
	num_cols = get_int(data[12:16])
	parsed = np.frombuffer(data, dtype=np.uint8, offset=16)
	return torch.from_numpy(parsed).view(length, num_rows, num_cols)

we could generate on-the-fly a dummy object that can test those functions, without going into testing all MNIST itself.

Drawback

We wouldn't be having end-to-end tests, but only be testing parts of the functionality. Having the tests.

Wrapping up

There are other potential ways of implementing unit tests for the datasets that I'm potentially missing in this list.

One thing that is pretty clear to me is that we should provide a function to perform the download + extract for zip / tar / etc that is tested, and use it everywhere in the codebase.

I'm currently inclined towards going for approach number 1, but it will involve adding a number of (dummy and very small) files to the repo. It's currently the only one that tests end-to-end all the functionality that is required by the dataset API, and thus is the most robust IMO.

Thoughts?

[soumith]

you can do things like mocking, where a fake OS / fake filesystem / fake files are read and processed, and we assert that the right things happened when interacting with this fake OS.

See https://www.toptal.com/python/an-introduction-to-mocking-in-python for example and look for rm and how the test for file removal

[soumith]

alternatively,

1. for small datasets, full download and test
2. for large datasets, tiny fake dataset

i think that's reasonable compromise

[pmeier]

@soumith Do you have a specific size in mind up to that we consider a dataset to be small?

[fmassa]

I'm currently implementing the "small" dataset downloading locally, and for MNIST / EMNIST / KMNIST / FashionMNIST, it takes 3 min to run a trivial test on my laptop.

I think that the small-dataset size might need to be smaller than EMNIST for tests to run reasonably fast.

I'll soon push a PR to test how long it takes CI to download it.

[fmassa]

We are going for an approach that generates random dataset files on-the-fly, so that they can be tested more easily.

Here is an example: https://github.com/pytorch/vision/blob/master/test/test_datasets.py

We have context-managers that generate datasets on the fly for the testing, and helps us catch bugs

vision/test/fakedata_generation.py

Lines 18 to 46 in 4886ccc

	@contextlib.contextmanager
	def mnist_root(num_images, cls_name):
	def _encode(v):
	return torch.tensor(v, dtype=torch.int32).numpy().tobytes()[::-1]
	def _make_image_file(filename, num_images):
	img = torch.randint(0, 255, size=(28 * 28 * num_images,), dtype=torch.uint8)
	with open(filename, "wb") as f:
	f.write(_encode(2051)) # magic header
	f.write(_encode(num_images))
	f.write(_encode(28))
	f.write(_encode(28))
	f.write(img.numpy().tobytes())
	def _make_label_file(filename, num_images):
	labels = torch.zeros((num_images,), dtype=torch.uint8)
	with open(filename, "wb") as f:
	f.write(_encode(2049)) # magic header
	f.write(_encode(num_images))
	f.write(labels.numpy().tobytes())
	with get_tmp_dir() as tmp_dir:
	raw_dir = os.path.join(tmp_dir, cls_name, "raw")
	os.makedirs(raw_dir)
	_make_image_file(os.path.join(raw_dir, "train-images-idx3-ubyte"), num_images)
	_make_label_file(os.path.join(raw_dir, "train-labels-idx1-ubyte"), num_images)
	_make_image_file(os.path.join(raw_dir, "t10k-images-idx3-ubyte"), num_images)
	_make_label_file(os.path.join(raw_dir, "t10k-labels-idx1-ubyte"), num_images)
	yield tmp_dir

[wolterlw]

Have you reached consensus on how to test contributed datasets yet?

I would like to contribute wrappers for several hand pose estimation datasets, like RHD, but haven't found a distinctive guide on how to structure a dataset wrapper or a transform.
In practice I usually structure data access as follows:

1. Dataset subclass that basically takes care of indexing.
   it's __getitem__() returns {'img_': Path, 'anno': annotation}
2. Transforms that take sample and load necessary files be it images, videos or files in any other format

This approach has following benefits:

• Dataset initialization time is minimal
• Filesystem caching takes care of keeping used files in memory
• Flexibility in file interpretation (useful for working with videos and sequential data generally)

Writing tests would also be separated into Database tests that interact with the directory system and separate Transform tests that need a single dummy sample.

[wolterlw]

resumed in #1080

[fmassa]

@andfoy could you work on this? The first dataset that would be good to have is UCF101 for #2475

Let me know if you have questions

[pmeier]

Tests for datasets and extraction logic with mocked archives

• CIFAR10
• CIFAR100
• Caltech101 ([POC] Base class for dataset tests #3402)
• Caltech256 (Add tests for Caltech101 #3412)
• CelebA (add tests for CelebA #3413)
• Cityscapes
• CocoCaptions (add tests for Coco(Captions|Detection) #3416)
• CocoDetection (add tests for Coco(Captions|Detection) #3416)
• DatasetFolder (add tests for (Dataset|Image)Folder #3477)
• EMNIST (add tests for MNIST and variants #3423)
• FakeData (add tests for FakeData dataset #3468)
• FashionMNIST
• Flickr30k (add tests for Flickr(8|30)k datasets #3489)
• Flickr8k (add tests for Flickr(8|30)k datasets #3489)
• HMDB51 (add tests for HMDB51 dataset #3458)
• ImageFolder (add tests for (Dataset|Image)Folder #3477)
• ImageNet
• Kinetics400 (add tests for Kinetics400 #3457)
• KMNIST
• LSUN (add tests for LSUN #3454)
• MNIST
• Omniglot (add tests for Omniglot dataset #3461)
• PhotoTour (Add tests for the PhotoTour dataset #3486)
• Places365
• QMNIST (add tests for MNIST and variants #3423 )
• SBDataset (add tests for SBDataset #3467)
• SBU (add tests for SBU dataset #3464)
• SEMEION (add tests for SEMEION dataset #3465)
• STL10 (Add tests for the STL10 dataset #3345)
• SVHN
• UCF101 (Add tests for UCF101 #3411)
• USPS (add tests for USPS dataset #3466)
• VOCDetection (Add tests for VOC(Segmentation|Detection) and fix existing bugs #3415)
• VOCSegmentation (Add tests for VOC(Segmentation|Detection) and fix existing bugs #3415)
• WIDERFace

---

Download tests

• CIFAR10
• CIFAR100
• Caltech101
• Caltech256
• CelebA (Add download tests for remaining datasets #3338)
• Cityscapes
• CocoCaptions
• CocoDetection
• DatasetFolder
• EMNIST (Add download tests for remaining datasets #3338)
• FakeData
• FashionMNIST (Add download tests for FashionMNIST #3337)
• Flickr30k
• Flickr8k
• HMDB51
• ImageFolder
• ImageNet
• KMNIST (Add download tests for remaining datasets #3338)
• Kinetics400
• LSUN
• MNIST (Add download tests for MNIST #3336)
• Omniglot (Add download tests for remaining datasets #3338)
• PhotoTour (Add download tests for remaining datasets #3338)
• Places365
• QMNIST (Add download tests for remaining datasets #3338)
• SBDataset (Add download tests for remaining datasets #3338)
• SBU (Add download tests for remaining datasets #3338)
• SEMEION (Add download tests for remaining datasets #3338)
• STL10 (Add download tests for remaining datasets #3338)
• SVHN (Add download tests for remaining datasets #3338)
• UCF101
• USPS (Add download tests for remaining datasets #3338)
• VOCDetection
• VOCSegmentation
• WIDERFace (Add download tests for remaining datasets #3338)