[Bug Report] Automatic download of MNIST dataset fails

[alar0330]

Link to the notebook
https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-experiments/mnist-handwritten-digits-classification-experiment/mnist-handwritten-digits-classification-experiment.ipynb

Describe the bug
Automatic download of MNIST dataset fails.

[In]

# this will not only download data to ./mnist folder, but also load and transform (normalize) them
train_set = datasets.MNIST('mnist', train=True, transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))]), 
    download=True)
                           
test_set = datasets.MNIST('mnist', train=False, transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))]),
    download=False)

[Out]

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to mnist/MNIST/raw/train-images-idx3-ubyte.gz
0it [00:00, ?it/s]
Extracting mnist/MNIST/raw/train-images-idx3-ubyte.gz

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-8-98ced4b6e1eb> in <module>
      4         transforms.ToTensor(),
      5         transforms.Normalize((0.1307,), (0.3081,))]), 
----> 6     download=True)
      7 
      8 test_set = datasets.MNIST('mnist', train=False, transform=transforms.Compose([

/opt/conda/lib/python3.7/site-packages/torchvision/datasets/mnist.py in __init__(self, root, train, transform, target_transform, download)
     66 
     67         if download:
---> 68             self.download()
     69 
     70         if not self._check_exists():

/opt/conda/lib/python3.7/site-packages/torchvision/datasets/mnist.py in download(self)
    144             file_path = os.path.join(self.raw_folder, filename)
    145             download_url(url, root=self.raw_folder, filename=filename, md5=None)
--> 146             self.extract_gzip(gzip_path=file_path, remove_finished=True)
    147 
    148         # process and save as torch files

/opt/conda/lib/python3.7/site-packages/torchvision/datasets/mnist.py in extract_gzip(gzip_path, remove_finished)
    125         print('Extracting {}'.format(gzip_path))
    126         with open(gzip_path.replace('.gz', ''), 'wb') as out_f, \
--> 127                 gzip.GzipFile(gzip_path) as zip_f:
    128             out_f.write(zip_f.read())
    129         if remove_finished:

/opt/conda/lib/python3.7/gzip.py in __init__(self, filename, mode, compresslevel, fileobj, mtime)
    166             mode += 'b'
    167         if fileobj is None:
--> 168             fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
    169         if filename is None:
    170             filename = getattr(fileobj, 'name', '')

FileNotFoundError: [Errno 2] No such file or directory: 'mnist/MNIST/raw/train-images-idx3-ubyte.gz'

To Reproduce
Run all cells in order.

Possible reason of error
The error is probably due to the download links for MNIST at https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py#L33-L36 are hosted on yann.lecun.com and that server has moved under CloudFlare protection.

Possible fix
Update the hardcoded version of PyTorch in the Notebook to a newer Pytorch version, where the MNIST download issue was fixed. See also this: pytorch/vision#1938

[hongshanli23]

I didn't encounter this issue, so the dataset should still be available. Did you try to upgrade torchvision

!pip install -qU torchvision

[alar0330]

Thanks for a prompt reply.

In order to reproduce, you need to exec all cells in order on a clean slate Python 3 (Data Science) container - as it is described in the header of the Notebook.

And, yes, I expect that upgrading torchvision will solve the problem (as I implicitly mentioned earlier by suggesting to upgrade the hardcoded versions of pip-installed packages in the Notebook), but just upgrading torchvision, as you suggested, will pull and upgrade all dependent packages including torch, which may break the training and deployment steps if one is not careful with specifying the framework_version (e.g. pytorch/pytorch#25214). I feel that a more careful fix might be needed.

More info and the torchvision fix: pytorch/vision#1938

Alternatively: download MNIST from an alternative location, e.g. a public 'sagemaker-sample-files' S3 bucket (as, e.g., done here: https://github.com/aws/amazon-sagemaker-examples/blob/master/frameworks/pytorch/get_started_mnist_train.ipynb).

[hongshanli23]

The issue you linked is for torchvision 0.4.2 but torchvision 0.3.0 is still fine. I can still use to download mnist data. If we use download it from a public bucket, then we will need to convert it to the format that can be used by torchvsion.datasets.MNIST in the notebook as well as the script mnist.py. It will makes the code convoluted and it obliterate the purpose of this notebook. Could you confirm the problem is still there if you use torchvision 0.3.0 and torch 1.1.0

[alar0330]

Resolved with #2091