[Frontend] [Tensorflow2] Added test infrastructure for TF2 frozen models #8074

rohanmukh · 2021-05-19T00:16:58Z

Test infrastructure for TF2 models that are frozen inside the script. Both ways of creating models in TF2 are tested -

Using tf.function calls as available in tests/frontend/tensorflow2/test_functional.py
Using tf.keras.Sequential API as available in tests/frontend/tensorflow2/test_sequential.py

Co-authored-by: David Huang davhuan@amazon.com
Co-authored-by: Rohan Mukherjee mukrohan@amazon.com
Co-authored-by: Wei Xiao weix@amazon.com

rohanmukh · 2021-05-19T00:29:17Z

@yongwww @zhiics @zxy844288792 @wx3000 @srkreddy1238 @dvhg

rohanmukh · 2021-05-19T18:39:29Z

@trevor-m

yongwww

One more question, does the tf2 model go with the same from_tensorflow as tf1?

yongwww · 2021-05-19T22:49:37Z

tests/python/frontend/tensorflow2/common.py

+from tensorflow.python.eager.def_function import Function
+
+
+def vmobj_to_list(o):


there is another vmobj_to_list in tests/python/frontend/tensorflow/test_forward.py, it would be good to merge this same-name function together, looks they are quite similar.

Thanks @yongwww for the comments. I addressed them in the new commits.

In this commit I do not introduce a new frontend code for TF2 models. That I will address in a new PR that is designed to handle TF2 style control flows and will also inherit some code from TF1 frontend code, mostly the ops. The structure of the PRs should follow the pattern as discussed in this thread, especially in this comment as below:

#4102 (comment)

yongwww · 2021-05-19T22:52:28Z

tests/python/frontend/tensorflow2/common.py

+    return vmobj_to_list(_out)
+
+
+def run_graph(lib, input_, ctx=tvm.cpu(0)):


personally, I like namerun_graph_runtime more

yongwww · 2021-05-19T22:55:45Z

tests/python/frontend/tensorflow2/common.py

+
+    Parameters
+    ----------
+    func: tf function. can be from saved model or not. different ways to pass input


don't see func in the arg list of the function compare_tf_tvm, please update them to keep docstring and definition identical

yongwww · 2021-05-19T22:58:17Z

tests/python/frontend/tensorflow2/common.py

+    return _out
+
+
+def compare_tf_tvm(gdef, input_, output_, vm=True, output_sig=None):


how about changing vm to "runtime_mode" or runtime, since we have vm, graphruntime, interpreter

trevor-m

Thanks @rohanmukh
I made some suggestions

trevor-m · 2021-05-19T23:44:27Z

tests/python/frontend/tensorflow2/common.py

+
+
+def compile_graph_runtime(
+    mod, params, target="llvm", target_host="llvm", opt_level=3, output_sig=None


output_sig is unused here and in compile_vm(), is this intended? What is output_sig?

It is a mistake from refactoring, output_sig are the outputs that should only be passed to frontend parser code, not to compiler

trevor-m · 2021-05-19T23:45:05Z

tests/python/frontend/tensorflow2/common.py

+    mod, params, target="llvm", target_host="llvm", opt_level=3, disabled_pass=None, output_sig=None
+):
+    with tvm.transform.PassContext(opt_level, disabled_pass=disabled_pass):
+        mod = relay.transform.InferType()(mod)


I don't think InferType is required here

trevor-m · 2021-05-19T23:45:34Z

tests/python/frontend/tensorflow2/common.py

+    return a
+
+
+def compile_graph_runtime(


Graph Runtime is now Graph Executor, may want to rename these functions

trevor-m · 2021-05-19T23:49:38Z

tests/python/frontend/tensorflow2/test_functional_models.py

+from common import run_tf_code
+
+
+class AddOne(tf.Module):


For all the tests in this file, I think it would be best to encapsulate the tf module class(es) inside the test function. For example:

def test_expand_dims(): class ExpandDims(tf.Module): def get_input(self): return np.ones((1, 30), dtype=np.float32) @tf.function(input_signature=[tf.TensorSpec(shape=(1, 30), dtype=tf.float32)]) def func(self, x): return tf.expand_dims(x, axis=2) run_all(ExpandDims)

trevor-m · 2021-05-19T23:50:54Z

tests/python/frontend/tensorflow2/test_sequential_models.py

+    compare_tf_tvm(*model_graph(model_fn, input_shape), vm=True, output_sig=None)
+
+
+def dense_model(input_shape, num_units=128):


Same here, I would put these model functions inside the test functions which they are used.

Thanks @trevor-m I addressed the comments in the new commits.

Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com>

trevor-m

Thanks for implementing the changes! LGTM

yongwww

LGTM

zhiics · 2021-05-24T22:56:26Z

tests/python/frontend/tensorflow2/common.py

+
+
+def run_tf_code(func, input_):
+    print(type(func))


remove or change to log

zhiics · 2021-05-24T22:58:23Z

python/tvm/relay/testing/tf.py

@@ -73,6 +75,46 @@ def convert_to_list(x):
    return x


+def vmobj_to_list(o):


is this copied from somewhere? We can add this to testing to remove the other places

This is copied from tests/python/frontend/tensorflow/test_forward.py
Do you mean I should remove it from there and import its invocations from here (i.e. relay.testing.tf ) in this PR? Or do you mean that should be done in another PR? Similar code exists in other places as well like test_vm.py, test_tensorrt.py etc.

zhiics

lgtm

zhiics · 2021-05-25T04:39:36Z

Thanks @rohanmukh @yongwww @trevor-m

…els (apache#8074) * added test infrastructure for frozen TF2 models * linting with black * removing some comments * change in comment in sequential test * addressed the comments * refactored to place vmobj_to_list in a common file * Added helper function in python/tvm/relay/testing/tf.py Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com> * Refactor tf according to CI error Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com> * Added docstring Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com> * removing print Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Xiao <weix@amazon.com>

rohanmukh added 3 commits May 18, 2021 23:55

added test infrastructure for frozen TF2 models

2d30945

linting with black

edb62ca

removing some comments

79a73ec

rohanmukh marked this pull request as ready for review May 19, 2021 00:29

change in comment in sequential test

689ac3b

yongwww reviewed May 19, 2021

View reviewed changes

trevor-m requested changes May 19, 2021

View reviewed changes

rohanmukh and others added 5 commits May 20, 2021 02:33

addressed the comments

6637481

refactored to place vmobj_to_list in a common file

0b04d40

Added helper function in python/tvm/relay/testing/tf.py

4dd6167

Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com>

Refactor tf according to CI error

a14c386

Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com>

Added docstring

512ab4e

Co-authored-by: David Huang <davhuan@amazon.com> Co-authored-by: Rohan Mukherjee <mukrohan@amazon.com> Co-authored-by: Xiao <weix@amazon.com>

trevor-m approved these changes May 20, 2021

View reviewed changes

yongwww approved these changes May 24, 2021

View reviewed changes

zhiics reviewed May 24, 2021

View reviewed changes

removing print

fd8f8e5

zhiics approved these changes May 25, 2021

View reviewed changes

zhiics merged commit 65cd19f into apache:main May 25, 2021

junrushao mentioned this pull request Nov 1, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Frontend] [Tensorflow2] Added test infrastructure for TF2 frozen models #8074

[Frontend] [Tensorflow2] Added test infrastructure for TF2 frozen models #8074

rohanmukh commented May 19, 2021 •

edited

Loading

rohanmukh commented May 19, 2021

rohanmukh commented May 19, 2021

yongwww left a comment

yongwww May 19, 2021

rohanmukh May 20, 2021

yongwww May 19, 2021

yongwww May 19, 2021

yongwww May 19, 2021

trevor-m left a comment

trevor-m May 19, 2021

rohanmukh May 20, 2021

trevor-m May 19, 2021

trevor-m May 19, 2021

trevor-m May 19, 2021

trevor-m May 19, 2021 •

edited

Loading

rohanmukh May 20, 2021

trevor-m left a comment

yongwww left a comment

zhiics May 24, 2021

zhiics May 24, 2021

rohanmukh May 24, 2021 •

edited

Loading

zhiics left a comment

zhiics commented May 25, 2021

		from tensorflow.python.eager.def_function import Function


		def vmobj_to_list(o):

		return vmobj_to_list(_out)


		def run_graph(lib, input_, ctx=tvm.cpu(0)):

		return _out


		def compare_tf_tvm(gdef, input_, output_, vm=True, output_sig=None):



		def compile_graph_runtime(
		mod, params, target="llvm", target_host="llvm", opt_level=3, output_sig=None

		compare_tf_tvm(*model_graph(model_fn, input_shape), vm=True, output_sig=None)


		def dense_model(input_shape, num_units=128):

		@@ -73,6 +75,46 @@ def convert_to_list(x):
		return x


		def vmobj_to_list(o):

[Frontend] [Tensorflow2] Added test infrastructure for TF2 frozen models #8074

[Frontend] [Tensorflow2] Added test infrastructure for TF2 frozen models #8074

Conversation

rohanmukh commented May 19, 2021 • edited Loading

rohanmukh commented May 19, 2021

rohanmukh commented May 19, 2021

yongwww left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevor-m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevor-m May 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevor-m left a comment

Choose a reason for hiding this comment

yongwww left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanmukh May 24, 2021 • edited Loading

Choose a reason for hiding this comment

zhiics left a comment

Choose a reason for hiding this comment

zhiics commented May 25, 2021

rohanmukh commented May 19, 2021 •

edited

Loading

trevor-m May 19, 2021 •

edited

Loading

rohanmukh May 24, 2021 •

edited

Loading