Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Flaky test test_softmin failed CI #13600

Closed
zachgk opened this issue Dec 10, 2018 · 3 comments · Fixed by #15116
Closed

Flaky test test_softmin failed CI #13600

zachgk opened this issue Dec 10, 2018 · 3 comments · Fixed by #15116

Comments

@zachgk
Copy link
Contributor

zachgk commented Dec 10, 2018

test_operator.test_softmin failed the CI at http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-13364/8/pipeline on stage "Python 3: MKLDNN-GPU Win" on unrelated PR #13364.

See output below:

======================================================================

FAIL: test_operator.test_softmin

----------------------------------------------------------------------

Traceback (most recent call last):

  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest

    self.test(*self.arg)

  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\common.py", line 173, in test_new

    orig_test(*args, **kwargs)

  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_operator.py", line 4408, in test_softmin

    rtol=rtol, atol=atol, grad_req=req, dtype=dtype)

  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 1131, in check_symbolic_backward

    equal_nan=equal_nan)

  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 495, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 1.119420 exceeds tolerance rtol=0.010000, atol=0.005000.  Location of maximum error:(0, 1, 0, 0), a=0.000000, b=-0.005660

 EXPECTED_softmin9_data: array([[[[ 0.,  0.,  0.],

         [ 0.,  0.,  0.]],

...

 BACKWARD_softmin9_data: array([[[[ 0.00006208, -0.00026137,  0.00015508],

         [ 0.00023366,  0.00007945,  0.00000728]],

...

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=350762988 to reproduce.

--------------------- >> end captured logging << ---------------------
@zachgk
Copy link
Contributor Author

zachgk commented Dec 10, 2018

@mxnet-label-bot add [Test, Flaky, Python]

@jlcontreras
Copy link
Contributor

@perdasilva
Copy link
Contributor

perdasilva commented May 21, 2019

Seems to be happening against the v1.4.x branch:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/v1.4.x/56/pipeline

I'll back port the PR disabling the test.

Please backport solution, once it is found. Perhaps also to v1.5.x.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants