Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Nightly test fail with "test_large_vector.test_nn" on v1.7.x branch #18246

Closed
ciyongch opened this issue May 6, 2020 · 4 comments
Closed

Nightly test fail with "test_large_vector.test_nn" on v1.7.x branch #18246

ciyongch opened this issue May 6, 2020 · 4 comments
Labels

Comments

@ciyongch
Copy link
Contributor

ciyongch commented May 6, 2020

Description

Nightly test fail with "test_large_vector.test_nn" on v1.7.x branch, please check log here

Error Message

05:29:54  + nosetests-3.4 tests/nightly/test_large_vector.py:test_nn
05:44:46  E
05:44:46  ======================================================================
05:44:46  ERROR: test_large_vector.test_nn
05:44:46  ----------------------------------------------------------------------
05:44:46  Traceback (most recent call last):
05:44:46    File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
05:44:46      self.test(*self.arg)
05:44:46    File "/work/mxnet/tests/nightly/test_large_vector.py", line 159, in test_nn
05:44:46      check_layer_norm()
05:44:46    File "/work/mxnet/tests/nightly/test_large_vector.py", line 89, in check_layer_norm
05:44:46      mx_out = nd.LayerNorm(data, gamma, beta, axis, eps)
05:44:46    File "<string>", line 81, in LayerNorm
05:44:46    File "/work/mxnet/python/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke
05:44:46      ctypes.byref(out_stypes)))
05:44:46    File "/work/mxnet/python/mxnet/base.py", line 246, in check_call
05:44:46      raise get_last_ffi_error()
05:44:46  mxnet.base.MXNetError: MXNetError: Shape inconsistent, Provided = [4300000000], inferred shape=[5032704]
05:44:46  
05:44:46  ----------------------------------------------------------------------
05:44:46  Ran 1 test in 896.356s
05:44:46  
05:44:46  FAILED (errors=1)

@ChaiBapchya @szha @TaoLv

@ciyongch ciyongch added the Bug label May 6, 2020
@ChaiBapchya
Copy link
Contributor

layer norm and batchnorm have been fixed in master by #17805

But they aren't cherry-picked in 1.7.x as of today.
Cherry-picked them now and tested locally
Verified the large tensor tests pass

@ciyongch
Copy link
Contributor Author

ciyongch commented May 8, 2020

Hi @ChaiBapchya , thanks for your kindly help to identify and point out the fix:)

@ciyongch ciyongch changed the title Night test fail with "test_large_vector.test_nn" on v1.7.x branch Nightly test fail with "test_large_vector.test_nn" on v1.7.x branch May 10, 2020
@ChaiBapchya
Copy link
Contributor

ChaiBapchya commented May 12, 2020

@ciyongch
Copy link
Contributor Author

Close this issue as it's fixed now.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants