test_gluon_data.test_dataloader_context broken on g4

[leezu]

Description

test_gluon_data.test_dataloader_context broken on g4

Error Message

======================================================================
ERROR: test suite for <module 'test_gluon_data' from 'C:\\jenkins_slave\\workspace\\ut-python-gpu\\tests\\python\\unittest\\test_gluon_data.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\nose\suite.py", line 229, in run
    self.tearDown()
  File "C:\Python37\lib\site-packages\nose\suite.py", line 352, in tearDown
    self.teardownContext(ancestor)
  File "C:\Python37\lib\site-packages\nose\suite.py", line 368, in teardownContext
    try_run(context, names)
  File "C:\Python37\lib\site-packages\nose\util.py", line 471, in try_run
    return func()
  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\common.py", line 310, in teardown
    mx.nd.waitall()
  File "c:\jenkins_slave\workspace\ut-python-gpu\python\mxnet\ndarray\ndarray.py", line 211, in waitall
    check_call(_LIB.MXNDArrayWaitAll())
  File "c:\jenkins_slave\workspace\ut-python-gpu\python\mxnet\base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "C:\jenkins_slave\workspace\build-gpu\src\common\../common/cuda_utils.h", line 414
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: invalid device ordinal

----------------------------------------------------------------------
Ran 7 tests in 1

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-17808/runs/59/nodes/108/steps/154/log/?start=0

To Reproduce

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. On g4 instance with Windows: python -m nose --verbose .\tests\python\unittest\test_gluon_data.py

What have you tried to solve it?

Run all other tests but the broken tests works:

python -m nose --verbose .\tests\python\unittest\test_gluon_data.py -m "test_array_dataset|test_recordimage_dataset|test_recordimage_dataset_with_data_loader_multiworker|test_sampler|test_datasets|test_image_folder_dataset|test_list_dataset|test_multi_worker|test_multi_worker_shape|test_multi_worker_forked_data_loader|test_multi_worker_dataloader_release_pool|test_dataset_filter|test_dataset_shard|test_dataset_take"