[CUTLASS][Ansor] Combine CUTLASS and Ansor

[qingchanghan]

Description

This PR adds a test script to combine CUTLASS and Ansor, which can use TensorCore kernels by CUTLASS and keep op fusion and automatic tuning of Ansor.

Modifications

1. Add a test script to show how to combine CUTLASS and Ansor.
2. Add the other_targets parameter for Ansor's extract_tasks function, to pass cutlass target to call_all_topi_funcs.

Performance

• Bert-large
• A10, CUDA 11.8
• FP16
• Input shape: (8, 128)

Latency(ms)

Ansor (n=3000)	CUTLASS+TOPI	CUTLASS+Ansor (n=3000)	MetaSchedule (n=3000)	CUTLASS+MetaSchedule (n=3000)
55.8870	20.2297	17.2543	19.2774	17.5876

Test scripts

https://github.com/qingchanghan/tvm-cutlass-eval/tree/combine-cutlass-ansor/bert

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• No users to tag found in teams: cutlass, ansor _{See #10317 for details}

_{Generated by tvm-bot}

[junrushao]

This is interesting and extremely helpful! Just curious - have you tried out CUTLASS + MetaSchedule?

[qingchanghan]

This is interesting and extremely helpful! Just curious - have you tried out CUTLASS + MetaSchedule?

Thank you for your advice! I tried CUTLASS + MetaSchedule and completed the test results. The performance diff between CUTLASS+Ansor and CUTLASS+MetaSchedule is negligible. I guess that the time ratio of non-MatMul operations is low in bert model and most tasks are optimized by CUTLASS.

Test script of CUTLASS + MetaSchedule: https://github.com/qingchanghan/tvm-cutlass-eval/blob/combine-cutlass-ansor/bert/cutlass_ms.py

[qingchanghan]

Please help review. @masahi @merrymercy @Laurawly @jwfromm @comaniac @tqchen @vinx13 @Hzfengsy @zhiics

[masahi]

Please merge your test into test_cutlass.py (while disabling the tuning test by default), and remove code duplications.

[junrushao]

LGTM!

[qingchanghan]

Please merge your test into test_cutlass.py (while disabling the tuning test by default), and remove code duplications.

Thanks for your review! The merge is done. BTW, I found that there are some errors when calling conv2d test in test_cutlass.py. I guess it is the compatibility problem introduced by tvm update. I will try to fix them later.

[Hzfengsy]

You could set a follow-up PR to fix that issue, and we can get it in quickly.

BTW, we are going to make further improvements to Meta-schedule with TensorCore recently, beginning with #13891

[masahi]

BTW, I found that there are some errors when calling conv2d test in test_cutlass.py

Is that an accuracy error from hardswish? I'm aware that assert_allclose in test_cutlass.py can fail depending on CUDA version / card etc.

[masahi]

ok residual block fusion seems to be broken. The "residual" tensor is supposed to be passed as a different tensor than the input tensor to conv2d. So there should be 4 inputs, data, weight, bias, residual_tensor (which is the same as data).

But now we are getting only 3 inputs.

[masahi]

@qingchanghan Here is my fix for the broken residual block fusion test submitted to a different repo tlc-pack/relax@13a73e5. It would be great if you can improve on it (e.g., removing TODO I included) and send it to main.

[qingchanghan]

BTW, we are going to make further improvements to Meta-schedule with TensorCore recently, beginning with #13891

This is cool! I will also learn Meta-schedule and keep an eye on it. Looking forward to further improvements!

[qingchanghan]

It would be great if you can improve on it (e.g., removing TODO I included) and send it to main.

OK. I'll try.