[microNPU] Add the infrastructure for lookup table and TANH #9547

ekalda · 2021-11-22T16:44:19Z

Some activation functions like TANH and SIGMOID are implemented
by calculating the values based on the QNN parameters and
recording the values into a lookup table (LUT).

This patch adds the LUT functionality alongside with the TANH
activation function and the tests.

ekalda · 2021-11-22T16:46:09Z

@manupa-arm @mbaret @lhutton1 @NicolaLancellotti @dchauhan-arm

lhutton1

Thanks @ekalda, mostly minor things

lhutton1 · 2021-11-22T21:54:23Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+    def create_op_with_lut(self, call):
+        """Extract the parameters and attributes from the NPU operator and create
+        a new operator with LUT.
+        ----------


Nit: missing Parameters and I think we need spacing for these doc strings?

lhutton1 · 2021-11-22T21:56:49Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        lut_activations = ["TANH", "LUT"]
+
+        if (
+            call.op.name == "contrib.ethosu.identity"


Should there be a check here to make sure call is an op?

From my digging of the code, it looks like tvm.relay.expr.Call always has an op attribute, so the equality check shouldn't fail with an error (but correct me if I'm wrong there)

I think that while it will always have an op attribute, op may not be a tvm.ir.Op and could, for example, be a Function. See here:

tvm/include/tvm/relay/expr.h

Line 269 in 52edc9a

* - It can be tvm::Op which corresponds to the primitive operators.

Ok yes, I added the op type check :)

lhutton1 · 2021-11-22T22:05:25Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+
+@relay.transform.function_pass(opt_level=1, name="LutOptimizer")
+class LUTsOptimizer(Pass):
+    """Register LutOptimizer as a relay pass."""


Nit: LUTsOptimizer and the same for name=LUTsOptimizer above.

lhutton1 · 2021-11-22T22:08:06Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        Parameters
+        ----------
+        func : tvm.relay.function.Function
+            The function to apply the layout optimization pass to.


Suggested change

The function to apply the layout optimization pass to.

The function to apply the optimization pass for multiple LUTs to.

lhutton1 · 2021-11-22T22:11:17Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        Returns
+        -------
+        mod : tvm.IRModule
+            New module with augmented layouts.


Suggested change

New module with augmented layouts.

New module with optimized LUTs.

lhutton1 · 2021-11-22T22:17:24Z

python/tvm/relay/backend/contrib/ethosu/legalize.py

@@ -123,6 +124,80 @@ def __call__(self, *args, **kwargs):
        pass


+def round_away_zero(f):


Could we reuse round_away_zero from util.py?

Ah nice, I didn't know it was there, thanks for pointing this out! Changed it to use the one from util.py

lhutton1 · 2021-11-22T22:25:22Z

python/tvm/relay/backend/contrib/ethosu/legalize.py

+        input_zp = int(dequantize_args[2].data.asnumpy())
+
+        lut_values = find_tanh_values(input_scale, input_zp, output_scale, output_zp)
+        lut = relay.const(lut_values, dtype="uint8")


Just curious, can LUT be any other dtype?

Currently, the values are calculated for an int8 activation and then cast into uint8 due to that can of worms dtype problem in the tir_to_cs_translator.py

Nice, glad to see a fix for that issue will improve something else :)

lhutton1 · 2021-11-22T22:38:39Z

tests/python/contrib/test_ethosu/test_lookup_table.py

+    ifm_shape = (1, 55, 55, 3)
+
+    def create_tflite_graph():
+        tf.config.run_functions_eagerly(True)


Do we need to enable eager execution?

No we don't :D Removed it!

lhutton1 · 2021-11-23T09:56:45Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
+        new_args.append(lut)


Suggested change

new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]

new_args.append(lut)

new_args = ethosu_op.args[:-1] + [lut]

NicolaLancellotti · 2021-11-24T16:24:56Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        a new NPU operator.
+        Parameters
+        ----------
+        call : tvm.relay.expr.Call
+            The current call node being visited.
+        Returns
+        -------


Suggested change

a new NPU operator.

Parameters

----------

call : tvm.relay.expr.Call

The current call node being visited.

Returns

-------

a new NPU operator.

Parameters

----------

call : tvm.relay.expr.Call

The current call node being visited.

Returns

-------

NicolaLancellotti · 2021-11-24T16:29:29Z

python/tvm/relay/backend/contrib/ethosu/codegen.py

+        """Visit relay nodes in the given module.
+        Parameters


Suggested change

"""Visit relay nodes in the given module.

Parameters

"""Visit relay nodes in the given module.

Parameters

ekalda · 2021-11-26T11:48:09Z

Thanks @lhutton1 and @NicolaLancellotti for the reviews! :)

NicolaLancellotti

LGTM!

lhutton1

LGTM!

Some activation functions like TANH and SIGMOID are implemented by calculating the values based on the QNN parameters and recording the values into a lookup table (LUT). This patch adds the LUT functionality alongside with the TANH activation function and the tests. Change-Id: Ibe49759fd02724af869826663ff0babd352e5894

manupak

LGTM!

I was hoping we could merge #9597 before this as it get conflicted :) -- but it got conflicted anyway. Therefore getting this in. Lets prioritize getting a version of #9597 otherwise it would be perma-conflicted.
cc : @leandron @mbaret

manupak · 2021-12-01T07:31:35Z

Thanks all!

) Some activation functions like TANH and SIGMOID are implemented by calculating the values based on the QNN parameters and recording the values into a lookup table (LUT). This patch adds the LUT functionality alongside with the TANH activation function and the tests.

ekalda requested review from anijain2305, areusch, comaniac, icemelon, jroesch, junrushao, jwfromm, MarisaKirisame, mbrookhart, merrymercy, slyubomirsky, tqchen, vinx13, wweic, yzhliu, zhiics and ZihengJiang as code owners November 22, 2021 16:44

lhutton1 reviewed Nov 23, 2021

View reviewed changes

ekalda mentioned this pull request Nov 23, 2021

[Tracking Issue] Arm(R) Ethos(TM)-U NPU operator support #9410

Closed

15 tasks

NicolaLancellotti reviewed Nov 24, 2021

View reviewed changes

ekalda force-pushed the lut_upstream branch from 56b9d84 to d6cad0c Compare November 26, 2021 11:38

NicolaLancellotti approved these changes Nov 26, 2021

View reviewed changes

lhutton1 approved these changes Nov 26, 2021

View reviewed changes

ekalda force-pushed the lut_upstream branch from c10f7ca to 27743a7 Compare November 29, 2021 10:13

ekalda added 3 commits November 29, 2021 10:39

Responding to the reviews

46b4602

Checking that tvm.relay.expr.Call.op exists

2218102

ekalda force-pushed the lut_upstream branch from 27743a7 to 2218102 Compare November 29, 2021 10:43

manupak approved these changes Dec 1, 2021

View reviewed changes

manupak merged commit 9ada371 into apache:main Dec 1, 2021

driazati mentioned this pull request Jul 14, 2022

TVM v0.9.0.rc0 Release Candidate Notes #12102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[microNPU] Add the infrastructure for lookup table and TANH #9547

[microNPU] Add the infrastructure for lookup table and TANH #9547

ekalda commented Nov 22, 2021

ekalda commented Nov 22, 2021

lhutton1 left a comment

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 26, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 26, 2021

lhutton1 Nov 22, 2021

ekalda Nov 26, 2021

lhutton1 Nov 23, 2021

ekalda Nov 26, 2021

NicolaLancellotti Nov 24, 2021

ekalda Nov 26, 2021

NicolaLancellotti Nov 24, 2021

ekalda Nov 26, 2021

ekalda commented Nov 26, 2021

NicolaLancellotti left a comment

lhutton1 left a comment

manupak left a comment •

edited

Loading

manupak commented Dec 1, 2021

	The function to apply the layout optimization pass to.
	The function to apply the optimization pass for multiple LUTs to.

	New module with augmented layouts.
	New module with optimized LUTs.

		@@ -123,6 +124,80 @@ def __call__(self, args, *kwargs):
		pass


		def round_away_zero(f):

		new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
		new_args.append(lut)

	new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
	new_args.append(lut)
	new_args = ethosu_op.args[:-1] + [lut]

[microNPU] Add the infrastructure for lookup table and TANH #9547

[microNPU] Add the infrastructure for lookup table and TANH #9547

Conversation

ekalda commented Nov 22, 2021

ekalda commented Nov 22, 2021

lhutton1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekalda commented Nov 26, 2021

NicolaLancellotti left a comment

Choose a reason for hiding this comment

lhutton1 left a comment

Choose a reason for hiding this comment

manupak left a comment • edited Loading

Choose a reason for hiding this comment

manupak commented Dec 1, 2021

manupak left a comment •

edited

Loading