Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

remote training service: mnist-keras example download data failed #2216

Closed
chicm-ms opened this issue Mar 21, 2020 · 1 comment
Closed

remote training service: mnist-keras example download data failed #2216

chicm-ms opened this issue Mar 21, 2020 · 1 comment
Assignees
Labels

Comments

@chicm-ms
Copy link
Contributor

chicm-ms commented Mar 21, 2020

when multiple trial jobs on same machine, the downloading can be failed:

2020-03-21T16:02:00.2296265Z Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz
2020-03-21T16:02:00.2297049Z 
2020-03-21T16:02:00.2298385Z     8192/11490434 [..............................] - ETA: 0s
2020-03-21T16:02:00.2300214Z  3088384/11490434 [=======>......................] - ETA: 0s
2020-03-21T16:02:00.2301403Z 10387456/11490434 [==========================>...] - ETA: 0s
2020-03-21T16:02:00.2302636Z 11493376/11490434 [==============================] - 0s 0us/step
2020-03-21T16:02:00.2303356Z Using TensorFlow backend.
2020-03-21T16:02:00.2303910Z Bad magic number for file header
2020-03-21T16:02:00.2304485Z Traceback (most recent call last):
2020-03-21T16:02:00.2305442Z   File "mnist-keras.py", line 128, in <module>
2020-03-21T16:02:00.2306870Z     train(ARGS, PARAMS)
2020-03-21T16:02:00.2308104Z   File "mnist-keras.py", line 93, in train
2020-03-21T16:02:00.2309169Z     x_train, y_train, x_test, y_test = load_mnist_data(args)
2020-03-21T16:02:00.2310368Z   File "mnist-keras.py", line 66, in load_mnist_data
2020-03-21T16:02:00.2311171Z     (x_train, y_train), (x_test, y_test) = mnist.load_data()
2020-03-21T16:02:00.2312896Z   File "/usr/local/lib/python3.5/dist-packages/keras/datasets/mnist.py", line 25, in load_data
2020-03-21T16:02:00.2314205Z     x_train, y_train = f['x_train'], f['y_train']
2020-03-21T16:02:00.2315858Z   File "/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py", line 228, in __getitem__
2020-03-21T16:02:00.2316586Z     bytes = self.zip.open(key)
2020-03-21T16:02:00.2317230Z   File "/usr/lib/python3.5/zipfile.py", line 1266, in open
2020-03-21T16:02:00.2317717Z     raise BadZipFile("Bad magic number for file header")
2020-03-21T16:02:00.2318262Z zipfile.BadZipFile: Bad magic number for file header
2020-03-21T16:02:00.2318708Z Traceback (most recent call last):
2020-03-21T16:02:00.2319995Z   File "mnist-keras.py", line 128, in <module>
2020-03-21T16:02:00.2320377Z     train(ARGS, PARAMS)
2020-03-21T16:02:00.2320906Z   File "mnist-keras.py", line 93, in train
2020-03-21T16:02:00.2321320Z     x_train, y_train, x_test, y_test = load_mnist_data(args)
2020-03-21T16:02:00.2321899Z   File "mnist-keras.py", line 66, in load_mnist_data
2020-03-21T16:02:00.2322372Z     (x_train, y_train), (x_test, y_test) = mnist.load_data()
2020-03-21T16:02:00.2323670Z   File "/usr/local/lib/python3.5/dist-packages/keras/datasets/mnist.py", line 25, in load_data
2020-03-21T16:02:00.2324931Z     x_train, y_train = f['x_train'], f['y_train']
2020-03-21T16:02:00.2326812Z   File "/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py", line 228, in __getitem__
2020-03-21T16:02:00.2328780Z     bytes = self.zip.open(key)
2020-03-21T16:02:00.2330012Z   File "/usr/lib/python3.5/zipfile.py", line 1266, in open
2020-03-21T16:02:00.2331793Z     raise BadZipFile("Bad magic number for file header")
2020-03-21T16:02:00.2333246Z zipfile.BadZipFile: Bad magic number for file header
2020-03-21T16:02:00.2335105Z [2020-03-21 16:01:51.138132] INFO subprocess terminated. Exit code is 1. Quit
2020-03-21T16:02:00.2338708Z [2020-03-21 16:01:51.152957] INFO NNI trial keeper exit with code 1
@chicm-ms chicm-ms changed the title mnist-keras example download data failed remote training service: mnist-keras example download data failed Mar 21, 2020
@chicm-ms chicm-ms self-assigned this Mar 21, 2020
@chicm-ms
Copy link
Contributor Author

Fixed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants