operator: Support Clip op for cpu, wasm, and webgl backends

[hariharans29]

Resolve #92

[hariharans29]

No use of templatizing as unary-op in WASM backend as there is not much to templatize (no core implementation here). In fact, the overhead in dealing with types and attributes parsing (which is much more prevalent than in binary-op case) makes it an unattractive option. So, keep the kernels separate even for unary-op like implementations

[hariharans29]

No use of templatizing like binary_imp as in binary ops there is a lot to templatize on (for example - the broadcasting logic, etc.). Here - there is only a single for loop to templatize on and it becomes cumbersome to deal with special-casing of attributes in unary-op case. So, only templatizing on different types of the op and instantiating as necessary.

[fs-eire]

I think it is OK for now. If we are going to have 10+ unary-op implementations it will be the time to consider this problem.

[hariharans29]

I think we can potentially support all unary-ops in WASM. Even then I would find it hard to templatize the core implementation as each has separate attributes. Since there is no native "attribute" object in our WASM layer, it becomes really hard to templatize this.

[fs-eire]

Generally looks good to me. there is one small bug in WASM implementation.

About WASM:
Compare WASM vs. javascript on element-wise unary op for performance:

• possible perf gain: avoid float32<->float64 convert for each element (because javascript number is float64, and the underlying data is usually Float32Array)
• possible drawback: extra cost in interop (ccallSerialize() and ccallDeserialize())

And this is also a JIT(V8)-vs-emcc battle, so the perf result may be very different in different browser version.

Due to our previous perf data (as we tried Relu before) in resnet50 there is no performance gain from implementation of simple element-wise unary op in WASM. But it is possible that WASM implementation can be faster than javascript. So in my opinion it's a good idea to have WASM unary-op implementation.

[fs-eire]

output[i] = (val < min) ? min : (val > max) ? max : val;

[hariharans29]

Thanks for catching this. Will fix.

[hariharans29]

This is weird, I am not sure how this is not being caught by node tests. I tried adding unit tests, but I realized that our test data sits in some other repo making it hard to add tests :(

[fs-eire]

op test is in the repo and node tests are not. do you want to add test case into node tests?

[hariharans29]

I had not resolved this op in the wasm backend and so it was passing (despite the bug) all tests by falling back to the cpu backend. I added wasm native clip in the operators resolve function and this caught the bug (as expected)

[hariharans29]

Generally looks good to me. there is one small bug in WASM implementation.

About WASM:
Compare WASM vs. javascript on element-wise unary op for performance:

• possible perf gain: avoid float32<->float64 convert for each element (because javascript number is float64, and the underlying data is usually Float32Array)
• possible drawback: extra cost in interop (ccallSerialize() and ccallDeserialize())

And this is also a JIT(V8)-vs-emcc battle, so the perf result may be very different in different browser version.

Due to our previous perf data (as we tried Relu before) in resnet50 there is no performance gain from implementation of simple element-wise unary op in WASM. But it is possible that WASM implementation can be faster than javascript. So in my opinion it's a good idea to have WASM unary-op implementation.

Another thing to consider for JS vs WASM unary-ops is the actual operation being performed on each element. For example, let's consider exp(x) - I do not know how slow exp(x) is for JS when compared to C++. My hypothesis is that (I do not have proof yet) for certain large number of elements, the interop cost + float32 <-> float64 + c++ element-wise op performance <= pure JS element-wise op.

[fs-eire]

Generally looks good to me. there is one small bug in WASM implementation.
About WASM:
Compare WASM vs. javascript on element-wise unary op for performance:

possible perf gain: avoid float32<->float64 convert for each element (because javascript number is float64, and the underlying data is usually Float32Array)
possible drawback: extra cost in interop (ccallSerialize() and ccallDeserialize())

And this is also a JIT(V8)-vs-emcc battle, so the perf result may be very different in different browser version.
Due to our previous perf data (as we tried Relu before) in resnet50 there is no performance gain from implementation of simple element-wise unary op in WASM. But it is possible that WASM implementation can be faster than javascript. So in my opinion it's a good idea to have WASM unary-op implementation.

Another thing to consider for JS vs WASM unary-ops is the actual operation being performed on each element. For example, let's consider exp(x) - I do not know how slow exp(x) is for JS when compared to C++. My hypothesis is that (I do not have proof yet) for certain large number of elements, the interop cost + float32 <-> float64 + c++ element-wise op performance <= pure JS element-wise op.

It is actually interop cost + WASM c++ element-wise op performance <= float32 <-> float64 + pure JS element-wise op.

For float32 tensor, the data is stored in Float32Array. For javascript, accessing an element in that typed array will cause a float32 -> float64, and write an element back to the typed array will cause a float64 -> float32.

The real underlying math function is always native so that if they are same function it should not make difference. however exp(double)(happened in js) and exp(float32)(happened in wasm) are expected to have difference.