Test large vector mean operator and fix a few bugs #16079

apeforest · 2019-09-03T05:59:15Z

Description

Add test for mean operator with large vector.
Fixed a few bugs that may affect other large array operators.
Fixed a few tests in test_large_vector to make it more memory efficient.

access2rohit · 2019-09-03T06:34:25Z

src/operator/tensor/broadcast_reduce-inl.h

  #pragma unroll
-  for (int i = ndim-1, j = mdim, s = 1; i >= 0; --i) {


nice catch ! :)

src/common/tensor_inspector.h

access2rohit · 2019-09-03T06:37:24Z

tests/nightly/test_large_vector.py

@@ -168,6 +168,12 @@ def test_topk():
    val = nd.topk(b, k=1, axis=0, dtype=np.int64, ret_typ="value")
    assert val.sum() == (LARGE_X - 1)

+
+def test_mean():
+    a = nd.arange(-LARGE_X / 2, LARGE_X / 2 + 1, dtype=np.int64)


nit: even though LARGE_X is divisible by 2 but can we use integer division here LARGE_X//2

…y-mean

ChaiBapchya

LGTM!

access2rohit · 2019-09-04T20:16:15Z

tests/nightly/test_large_vector.py

 def test_clip():
    a = create_vector(LARGE_X)
    res = nd.clip(a, a_min=100, a_max=1000)
-    assert np.sum(res[-1].asnumpy() == 1000) == 1
+    assert res[-1] == 1000


good ... previous one was bad

access2rohit · 2019-09-04T20:17:59Z

Nice work !
Removal of all numpy functions should speedup the processes

apeforest · 2019-09-04T23:19:21Z

@access2rohit My test test_mean runs through. However, test_sequence_last seems to run out of memory on an instance with 480G memory.

test_large_vector.test_slice ... ok
test_large_vector.test_ndarray_zeros ... ok
test_large_vector.test_ndarray_ones ... ok
test_large_vector.test_ndarray_random_uniform ... ok
test_large_vector.test_ndarray_random_randint ... ok
test_large_vector.test_ndarray_empty ... ok
test_large_vector.test_elementwise ... ok
test_large_vector.test_clip ... ok
test_large_vector.test_argmin ... ok
test_large_vector.test_take ... ok
test_large_vector.test_slice_assign ... ok
test_large_vector.test_expand_dims ... ok
test_large_vector.test_squeeze ... ok
test_large_vector.test_broadcast_div ... ok
test_large_vector.test_Dense ... ok
test_large_vector.test_argsort ... ok
test_large_vector.test_sort ... ok
test_large_vector.test_topk ... ok
test_large_vector.test_mean ... ok
test_large_vector.test_ndarray_random_exponential ... ok
test_large_vector.test_ndarray_random_gamma ... ok
test_large_vector.test_ndarray_random_generalized_negative_binomial ... ok
test_large_vector.test_ndarray_random_multinomial ... ok
test_large_vector.test_ndarray_random_negative_binomial ... ok
test_large_vector.test_ndarray_random_normal ... ok
test_large_vector.test_ndarray_random_poisson ... ok
test_large_vector.test_ndarray_random_randn ... ok
test_large_vector.test_ndarray_random_shuffle ... ok
test_large_vector.test_exponent_logarithm_operators ... ok
test_large_vector.test_power_operators ... ok
test_large_vector.test_sequence_mask ... ok
test_large_vector.test_sequence_reverse ... ok
test_large_vector.test_sequence_last ...
Segmentation fault: 11

Segmentation fault: 11

Stack trace:
[bt] (0) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x8f96a9) [0x7fc345d596a9]
[bt] (1) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fc366cc94b0]
[bt] (2) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x1909334) [0x7fc346d69334]
[bt] (3) /usr/lib/x86_64-linux-gnu/libgomp.so.1(+0xf43e) [0x7fc352a0543e]
[bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fc3670656ba]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fc366d9b41d]
Stack trace:
[bt] (0) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x8f96a9) [0x7fc345d596a9]
[bt] (1) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fc366cc94b0]
[bt] (2) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x1909334) [0x7fc346d69334]
[bt] (3) /usr/lib/x86_64-linux-gnu/libgomp.so.1(GOMP_parallel+0x3f) [0x7fc352a01cbf]
[bt] (4) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::op::SequenceLastOp<mshadow::cpu, float, float>::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x50b) [0x7fc346d8838b]
[bt] (5) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::op::OperatorState::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0xab0) [0x7fc345eca990]
[bt] (6) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::Resource, std::allocatormxnet::Resource > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<unsigned int, std::allocator > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#3}::operator()(mxnet::RunContext, mxnet::engine::CallbackOnComplete) const+0x2aa) [0x7fc345d5241a]
[bt] (7) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::Resource, std::allocatormxnet::Resource > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<unsigned int, std::allocator > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext)#4}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&)+0x1d) [0x7fc345d533ed]
[bt] (8) /home/ubuntu/src/mxnet/python/mxnet/../../build/libmxnet.so(+0x84c3de) [0x7fc345cac3de]

apeforest · 2019-09-05T04:34:09Z

@wkcn Can you help to review and merge this for me?

wkcn

Thank you for the update!

In the unittest test_ndarray_random_randint,
it will be specific to use 2 ** 32 rather than 4294967296.

tests/nightly/test_large_vector.py

…y-mean

wkcn

LGTM. I will merge it after the CI pass.

* add test_mean and fix a few potential bugs * address reviewer commnet * fix and refactor test * fix topk test * address reviewer comment

apeforest added 2 commits September 2, 2019 22:56

add test_mean and fix a few potential bugs

04fab9c

Merge branch 'master' into bugfix/large-array-mean

32f9800

access2rohit reviewed Sep 3, 2019

View reviewed changes

src/common/tensor_inspector.h Show resolved Hide resolved

access2rohit reviewed Sep 3, 2019

View reviewed changes

address reviewer commnet

0cdf325

access2rohit approved these changes Sep 3, 2019

View reviewed changes

apeforest added 2 commits September 4, 2019 19:30

fix and refactor test

a30fb91

Merge remote-tracking branch 'upstream/master' into bugfix/large-arra…

d4208b2

…y-mean

ChaiBapchya approved these changes Sep 4, 2019

View reviewed changes

access2rohit reviewed Sep 4, 2019

View reviewed changes

access2rohit approved these changes Sep 4, 2019

View reviewed changes

fix topk test

58dda55

apeforest added the pr-awaiting-merge Review and CI is complete. Ready to Merge label Sep 5, 2019

wkcn reviewed Sep 5, 2019

View reviewed changes

tests/nightly/test_large_vector.py Outdated Show resolved Hide resolved

tests/nightly/test_large_vector.py Outdated Show resolved Hide resolved

apeforest added 2 commits September 4, 2019 22:28

Merge remote-tracking branch 'upstream/master' into bugfix/large-arra…

4fdba4a

…y-mean

address reviewer comment

aed27ea

wkcn approved these changes Sep 5, 2019

View reviewed changes

wkcn merged commit d0fa8c0 into apache:master Sep 5, 2019

apeforest deleted the bugfix/large-array-mean branch September 5, 2019 09:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test large vector mean operator and fix a few bugs #16079

Test large vector mean operator and fix a few bugs #16079

apeforest commented Sep 3, 2019 •

edited

Loading

access2rohit Sep 3, 2019 •

edited

Loading

access2rohit Sep 3, 2019

ChaiBapchya left a comment

access2rohit Sep 4, 2019

access2rohit commented Sep 4, 2019

apeforest commented Sep 4, 2019

apeforest commented Sep 5, 2019

wkcn left a comment

wkcn left a comment

		#pragma unroll
		for (int i = ndim-1, j = mdim, s = 1; i >= 0; --i) {

Test large vector mean operator and fix a few bugs #16079

Test large vector mean operator and fix a few bugs #16079

Conversation

apeforest commented Sep 3, 2019 • edited Loading

Description

access2rohit Sep 3, 2019 • edited Loading

Choose a reason for hiding this comment

access2rohit Sep 3, 2019

Choose a reason for hiding this comment

ChaiBapchya left a comment

Choose a reason for hiding this comment

access2rohit Sep 4, 2019

Choose a reason for hiding this comment

access2rohit commented Sep 4, 2019

apeforest commented Sep 4, 2019

apeforest commented Sep 5, 2019

wkcn left a comment

Choose a reason for hiding this comment

wkcn left a comment

Choose a reason for hiding this comment

apeforest commented Sep 3, 2019 •

edited

Loading

access2rohit Sep 3, 2019 •

edited

Loading