Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

marcoabreu · 2018-02-21T08:18:33Z

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/395/pipeline

======================================================================

FAIL: test_random.test_exponential_generator

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest

    self.test(*self.arg)

  File "/workspace/tests/python/unittest/common.py", line 155, in test_new

    orig_test(*args, **kwargs)

  File "/workspace/tests/python/unittest/test_random.py", line 374, in test_exponential_generator

    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)

  File "/workspace/python/mxnet/test_utils.py", line 1891, in verify_generator

    str(buckets), str(probs)))

AssertionError: Generator test fails, Chi-square p=[0.21031861623067064, 0.023120840905593496, 0.02566284972432512, 0.0396308101152387, 0.031248307960405564], obs_freq=[array([200561, 200496, 199515, 200039, 199389]), array([198895, 200007, 200598, 200780, 199720]), array([200509, 200524, 199203, 200596, 199168]), array([199922, 200497, 200741, 199937, 198903]), array([199609, 200258, 199725, 201138, 199270])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].

buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.5108256237659907), (0.5108256237659907, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]

-------------------- >> begin captured stdout << ---------------------

ctx=cpu(0), dtype=float16, Scale=0.1:

ctx=cpu(0), dtype=float16, Scale=1:



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1690699840 to reproduce.

--------------------- >> end captured logging << ---------------------

The text was updated successfully, but these errors were encountered:

zheng-da · 2018-02-22T22:15:25Z

It seems this problem is related to #9856

sxjscience · 2018-02-22T22:47:56Z

As shown in the log, the chi-square value is above the 0.05 threshold (indicates the check has passed) once in the 5 runs, i.e, 20% success rate. Currently we force the success rate to be 25% (https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/test_utils.py#L1848), we could lower it to 10% to fix the test.

marcoabreu · 2018-02-22T23:29:16Z

That might fix the test, but could we risk masking a problem that way? I'm afraid I'm unfamiliar with the topic, so would you mind explaining how an actual failure would look like and how we could select the best threshold?

sxjscience · 2018-02-23T01:25:24Z

@marcoabreu A wrongly implemented random number generator would typically have success_rate = 0, i.e, producing p < 0.05 for all the 5 runs. However, since this tests the randomness, it's really tricky to set the threshold and my previous design is to force the success rate to be at least 25%.

zheng-da · 2018-02-28T07:59:36Z

This test also fails constantly with a special seed.

$ export MXNET_TEST_SEED=548415857
$ for i in {1..10}; do nosetests -v tests/python/gpu/test_operator_gpu.py:test_exponential_generator; done
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.485s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.569s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.078s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.074s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.156s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.483s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.407s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.576s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.255s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.210s

FAILED (failures=1)

sxjscience · 2018-02-28T17:49:31Z

Should I revise the code to use a smaller threshold?

azai91 · 2018-06-27T19:20:35Z

@marcoabreu the success rate has been lowered. do we know if this test is still causing us problems?

marcoabreu · 2018-06-27T22:45:57Z

Haven't seen any problems recently. Thanks for following up!

azai91 · 2018-06-27T22:56:11Z

kk ran this test 500x as well and didn't see a problem

ubuntu@ip-172-31-11-93:~/incubator-mxnet-original$ for i in {1..500}; do nosetests tests/python/unittest/test_random.py:test_exponential_generator; if [ $? -ne 0 ]; then break; fi; done || exit 1

piyushghai · 2019-01-07T22:37:57Z

Found a failure of this test on the CI pipeline on Windows-CPU.

Details of failure captured here : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-cpu/detail/PR-13678/30/pipeline

marcoabreu added Test Flaky labels Feb 21, 2018

marcoabreu changed the title ~~Flaky test_random.test_exponential_generator~~ Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU Feb 21, 2018

pengzhao-intel mentioned this issue Feb 21, 2018

Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU #9820

Closed

sxjscience mentioned this issue Feb 28, 2018

[Test] smaller success rate for test random #9927

Merged

6 tasks

marcoabreu closed this as completed Jun 27, 2018

anirudh2290 self-assigned this Jun 27, 2018

perdasilva mentioned this issue Feb 18, 2019

Flaky test_random.test_exponential_generator @ Python3: Centos 7 CPU #14187

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

marcoabreu commented Feb 21, 2018

zheng-da commented Feb 22, 2018

sxjscience commented Feb 22, 2018 •

edited

Loading

marcoabreu commented Feb 22, 2018

sxjscience commented Feb 23, 2018

zheng-da commented Feb 28, 2018

sxjscience commented Feb 28, 2018

azai91 commented Jun 27, 2018

marcoabreu commented Jun 27, 2018

azai91 commented Jun 27, 2018

piyushghai commented Jan 7, 2019

Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

Comments

marcoabreu commented Feb 21, 2018

zheng-da commented Feb 22, 2018

sxjscience commented Feb 22, 2018 • edited Loading

marcoabreu commented Feb 22, 2018

sxjscience commented Feb 23, 2018

zheng-da commented Feb 28, 2018

sxjscience commented Feb 28, 2018

azai91 commented Jun 27, 2018

marcoabreu commented Jun 27, 2018

azai91 commented Jun 27, 2018

piyushghai commented Jan 7, 2019

sxjscience commented Feb 22, 2018 •

edited

Loading