New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MetaSchedule][M4b] Testcases for TensorRT builder/runner #10055

Merged

junrushao merged 50 commits into apache:main from sunggg:meta-trt-testcase

Jan 29, 2022

Contributor

sunggg commented Jan 24, 2022

This PR includes BYOC builder/runner infra and its test case for TensorRT.
Thanks for your time to review this.
cc: @junrushao1994

Please note that previous PR is closed due to the overlap with previous merge.

sunggg requested review from areusch, comaniac, icemelon, jroesch, junrushao, merrymercy, tqchen and yzhliu as code owners

January 24, 2022 23:54

Member

junrushao commented Jan 25, 2022

CC @zxybazh would love you guys to review each other’s code :-)

junrushao mentioned this pull request

[RFC][Tracking Issue] Meta Schedule (AutoTIR) #8473

Closed

62 tasks

junrushao changed the title ~~[MetatSchedule] testcase for TensorRT builder/runner~~ [MetatSchedule][M4b] Testcases for TensorRT builder/runner

junrushao changed the title ~~[MetatSchedule][M4b] Testcases for TensorRT builder/runner~~ [MetaSchedule][M4b] Testcases for TensorRT builder/runner

junrushao reviewed

View reviewed changes

Member

junrushao left a comment

Just some minor nitpicks

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

Comment on lines 30 to 32

+              # from tvm import script
+              # from tvm._ffi import register_func
+              # from tvm.runtime import Module

Member

junrushao Jan 28, 2022

remove this?

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

+              ):
+                  if use_meta_sched:
+                      # With meta_schedule
+                      dev = "nvidia/geforce-rtx-2080"

Member

junrushao Jan 28, 2022

Suggested change

      
                    dev = "nvidia/geforce-rtx-2080"
          
                    dev = "cuda"

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

Comment on lines 102 to 115

+                          def relay_build_with_tensorrt(
+                              mod: Module,
+                              target: Target,
+                              params: dict,
+                          ) -> List[BuilderResult]:
+                              from tvm.relay.op.contrib.tensorrt import partition_for_tensorrt
+                              mod, config = partition_for_tensorrt(mod, params)
+                              with tvm.transform.PassContext(
+                                  opt_level=3, config={"relay.ext.tensorrt.options": config}
+                              ):
+                                  return tvm.relay.build_module._build_module_no_factory(
+                                      mod, "cuda", "llvm", params
+                                  )

Member

junrushao Jan 28, 2022

maybe we should refactor these functions, put them under python/tvm/meta_schedule/testing/byoc_trt.py, so that others could conveniently reuse these cool stuff

Member

junrushao Jan 28, 2022

example: https://github.com/junrushao1994/tvm/blob/meta-schedule/python/tvm/meta_schedule/testing/byoc_trt.py

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

+                              target: Target,
+                              params: dict,
+                          ) -> List[BuilderResult]:
+                              # @Sung: Weird. Cannot pass keyword arg

Member

junrushao Jan 28, 2022

if you have time, you may add a proxy function to _build_module_no_factory to allow kwargs

@register_func("tvm.relay.build")
def _build_module_no_factory_impl(mod, target, target_host, params, mod_name):
    target, target_host = Target.check_and_update_host_consist(target, target_host)
    return build(mod, target, params=params, mod_name=mod_name).module


def _build_module_no_factory(mod, target=None, target_host=None, params=None, mod_name="default"):
    """A wrapper around build which discards the Python GraphFactoryRuntime.
    This wrapper is suitable to be used from other programming languages as
    the runtime::Module can be freely passed between language boundaries.
    """
    return _build_module_no_factory_impl(mod, target, target_host, params, mod_name)

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

+                  "model_name",
+                  ["resnet-50", "mobilenet"],
+              )
+              @pytest.mark.parametrize("batch_size", [1, 8, 16])

Member

junrushao Jan 28, 2022

Suggested change

      
            @pytest.mark.parametrize("batch_size", [1, 8, 16])
          
            @pytest.mark.parametrize("batch_size", [1])

tests/python/unittest/test_meta_schedule_byoc_tensorrt.py Outdated

		)


		# @sunggg: memory verification error at test_relay_model("resnet-50", 1, use_meta_sched=False, use_trt=True)

Member

junrushao Jan 28, 2022

cannot reproduce this, so let's double confirm :-) If there is no problem, let's remove this line

junrushao force-pushed the meta-trt-testcase branch from 8f9b886 to eb21e55 Compare

January 28, 2022 23:32

sunggg and others added 16 commits

January 28, 2022 15:45


          [MetatSchedule] testcase for TensorRT builder/runner

eb344fd


          [Runtime][PipelineExecutor] Add Pipeline Executor Interface (apache#1…

03646ac

…0010)

Adding interfaces into Pipeline Executor to "run", "stop","set input",
and "get input" from the pipeline executor,

In this patch, we also implemented the "BackendRuntime" structure to
wrap the graph runtime interface in order to support  pipeline executor
interface and implement data copy method. This method is used to
transfer data between two backend runtimes.


          [skip ci][Docker, CI] Update DGL installation, temp disable DGL tutor…

29a77e3

…ial (apache#10067)


          [CUTLASS] Profile only the largest-possible alignment by default (apa…

452e168

…che#10036)

* introduce profile_all_alignments option

* add profile_all_alignment option to API

* wip

* fixed dynamic case

* black

* update gen_gemm too

* minor improvement

* fix

* all tests work

* add doc

* fixed for sm = 75 case

* fix typo

* remove unused import

* profile_all -> find_first_valid

* fix


          [Meta Schedule] Add ApplyHisotryBest Meta Schedule Context (apache#…

d2ac944

…10049)

* Add ApplyHisotryBest.

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Wuwei Lin <wuwei@apache.org>
Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>

* Retrigger CI.

* Update integration.py

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Wuwei Lin <wuwei@apache.org>
Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>


          [MetaSchedule] Mutator Rule: Mutate Unroll (apache#10045)

15e0624

* mutate-unroll

* mutate-unroll


          [TIR][Schedule] Blockize and Tensorize (apache#9871)

f9e1ff8

* WIP

* WIP

* WIP

* test cases

* add examples

* lint

* Amend co-authors information

Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>

* WIP

* address comments and changed tensorized comparator

* update

* nit

* fix example

* lint

* lint

* lint

* remove unused

* trigger ci

* clang-format

* fix

* rebase

Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>


          [microTVM][tutorial] Add ENV variable to enable testing on physical h…

fa00dc6

…ardware (apache#9993)

* Add env variable to micro tflite tutorial

* Address @gromero comments

* address @areusch comment

* fix scope

* trigger

* trigger


          [microNPU] Refactor base address determination to codegen (apache#9929)

c90311a

This commit introduces BaseAddress ObjectRef to determine
base addresses in the codegen for microNPU. This is
required when multiple memory pools become available. Thus,
base addresses could not be statically determined in the
source module.


          Add FP requantize flow. Set float32 flow by default for llvm x86 targ…

b3fcb4b

…ets with (apache#9637)

sse4.1 support


          [Relay][DefuseOps pass] bug fix: To support function body types other…

a0c95f8

… than call node (apache#10069)

Co-authored-by: pranav jonnalagadda-SJ1 Eng_ML <pjonnalagadd@sj1mach1.caveonetworks.com>


          [Fix Bug]fix the bug of tensorflow frontend when parsing Range layer (a…

e7705d7

…pache#9999)

Co-authored-by: wangjiuyang <wang.jiuyang@intellif.com>


          [MetaSchedule][M4a] Schedule Rule: Multi-Level-Tiling (apache#10043)

887779b

* multi level tiling

* remove tensor core related code

* pylint

* fix

Co-authored-by: Junru Shao <junrushao1994@gmail.com>


          Revert "[Frontend] Add Span filling for frontends to Relay (apache#9723…

aa44d7b

…)" (apache#10072)

Because of the failure of LSTM conversion from Pytorch


          Improve the tensorflow frontend _test_spop_resource_variables to supp…

4a15db2

…ort tensoflow 2.6 (apache#9978)

On tensorflow 2.4 the test is expected to fail as the generated graph is not forzen.
On tensorflow 2.6 the generated graph is identified as frozen, therefore the test is not needed


          [MetaSchedule] postproc: rewrite_parallel_vectorize_unroll (apache#10071

b1812bb

)

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Wuwei Lin <wuwei@apache.org>

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Wuwei Lin <wuwei@apache.org>

sunggg requested review from mbaret, mbrookhart, siju-samuel, slyubomirsky, srkreddy1238, tmoreau89, trevor-m, vinx13, wweic, zhiics, ZihengJiang and a team as code owners

January 29, 2022 00:08

sunggg added 9 commits

January 28, 2022 16:54


          Rebase to pass CI and reflect suggestions

4bad153


          Add ASF header for the new file

e38b24d


          [MetatSchedule] testcase for TensorRT builder/runner

a269c8c


          add pytest condition to pass CI. rename test name to be consistent.

8b87320


          add pyteset decorator to pass CI

46325d0


          [MetatSchedule] testcase for TensorRT builder/runner

ca52292


          add pytest condition to pass CI. rename test name to be consistent.

419d756


          Rebase to pass CI and reflect suggestions

c45d16a


          Add ASF header for the new file

68fd695

junrushao force-pushed the meta-trt-testcase branch from e38b24d to 68fd695 Compare

January 29, 2022 01:27

sunggg added 3 commits

January 28, 2022 17:30


          add pylint, docstring


          Merge branch 'meta-trt-testcase' of https://github.com/sunggg/tvm int…

0fda196

…o meta-trt-testcase


          fix lint and wish for the best

04cb4ca

junrushao approved these changes

View reviewed changes

junrushao merged commit ba65197 into apache:main

Member

junrushao commented Jan 29, 2022

Thanks @sunggg! It's finally merged :-)

ylc pushed a commit to ylc/tvm that referenced this pull request


          [MetaSchedule][M4b] Testcases for TensorRT builder/runner (apache#10055)

a1e5ccd

Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>
Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>

driazati mentioned this pull request

TVM v0.9.0.rc0 Release Candidate Notes #12102

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

junrushao junrushao approved these changes

areusch Awaiting requested review from areusch

comaniac Awaiting requested review from comaniac

icemelon Awaiting requested review from icemelon

jroesch Awaiting requested review from jroesch

merrymercy Awaiting requested review from merrymercy

tqchen Awaiting requested review from tqchen

yzhliu Awaiting requested review from yzhliu

anijain2305 Awaiting requested review from anijain2305

Huyuwei Awaiting requested review from Huyuwei

Hzfengsy Awaiting requested review from Hzfengsy

jcf94 Awaiting requested review from jcf94

jwfromm Awaiting requested review from jwfromm

kazum Awaiting requested review from kazum

kevinthesun Awaiting requested review from kevinthesun

kparzysz-quic Awaiting requested review from kparzysz-quic

Laurawly Awaiting requested review from Laurawly

leandron Awaiting requested review from leandron

liangfu Awaiting requested review from liangfu

manupak Awaiting requested review from manupak

MarisaKirisame Awaiting requested review from MarisaKirisame

masahi Awaiting requested review from masahi

mbaret Awaiting requested review from mbaret

mbrookhart Awaiting requested review from mbrookhart

siju-samuel Awaiting requested review from siju-samuel

slyubomirsky Awaiting requested review from slyubomirsky

srkreddy1238 Awaiting requested review from srkreddy1238

tmoreau89 Awaiting requested review from tmoreau89

trevor-m Awaiting requested review from trevor-m

vinx13 Awaiting requested review from vinx13

wweic Awaiting requested review from wweic

zhiics Awaiting requested review from zhiics

ZihengJiang Awaiting requested review from ZihengJiang

Labels

None yet

25 participants