[MI200][NCHW][SWDEV-312112] The find-db NCHW records for ConvAsmImplicitGemmGTCDynamic*XdlopsNHWC must be updated

[atamazov]

The find-db NCHW records for

• ConvAsmImplicitGemmGTCDynamicBwdXdlopsNHWC
• ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC
• ConvAsmImplicitGemmGTCDynamicWrwXdlopsNHWC

must be refreshed.

Originally posted by @atamazov in #1324 (comment)

[atamazov]

#1324 increases WS up to 3 bytes, but the system find-db remains the same. The library may run into fatal errors from time to time.

The problem may manifest in develop, Staging tests and in 5.0 Release Staging tests. The latter is potentially the most problematic.

How we can fix this

We do not need to actually run tuning sessions to update find-db files. Adding 3 to the "required workspace size" field for ConvAsmImplicitGemmGTCDynamic*XdlopsNHWC solvers in NCHW records should be enough. However it is better to accommodate for #1327 in advance and increase requires WS size not by 3, but by (255*3).

Alternatives:

• Regenerate find-db for MI100/MI200 (@JehandadKhan) from the existing Tuna db.
• Patch existing find-db files using some script (@DrizztDoUrden )

Let's decide which way we prefer (assuming that the goal is to have the fix ASAP).

[JehandadKhan]

@atamazov Thanks for highlighting this. Can we hold off on merging #1324 until January ?

[atamazov]

Sure. But the immediate problem is that we have this issue in 5.0 staging.

[junliume]

AFAIK, we tune after the branch is cut, and will need to regenerate find-db then anyway?

[atamazov]

Of course, but we do not want to receive bug reports before that, right?

[atamazov]

...And currently we have this problem in Staging branch. After promotion it will be replicated to Master branch and will remain there (i.e. in all Mainline builds) until 5.0 tuning is done, backported to develop, promoted to Staging, tested, and finally promoted to Mainline.

[junliume]

@atamazov @JehandadKhan : which way is "faster", a.k.a. can be completed by next Tuesday? I will vote for "Patch existing find-db files using some script (@DrizztDoUrden )" since Chris is off and patching looks like a more "direct" way to ensure problems are fixed. What do you think?

[atamazov]

Either variant works for me.

[JehandadKhan]

I vote for @DrizztDoUrden to update it since that sounds the safest to me as well.

[atamazov]

@DrizztDoUrden Let's update MI100/MI200 fdb as per #1349 (comment). I am available for a call at any time.

[atamazov]

@shaojiewang FYI. Please share your opinion about this. Thanks.

[shaojiewang]

@atamazov Thanks for findings and updates. I think that patching the existing find-db is a better way.

In the future, for the find db part, could it be possible to remove the Workspace_size from the db and just let it be computed on the fly? Or else, is it possible to have some reserved space for WS?

[atamazov]

@JehandadKhan FYI There are none records with ConvAsmImplicitGemmGTCDynamic*XdlopsNHWC in MI100 find-db files.

[atamazov]

@JehandadKhan FYI There are none records with ConvAsmImplicitGemmGTCDynamic*XdlopsNHWC in MI100 find-db files.

I mean, is it expected in develop?

[JehandadKhan]

@JehandadKhan FYI There are none records with ConvAsmImplicitGemmGTCDynamic*XdlopsNHWC in MI100 find-db files.

I mean, is it expected in develop?

Not according to my data

[atamazov]

@JehandadKhan Still unclear... ;) So there is nothing to fix in MI100 System find-db files, right?

[JehandadKhan]

@atamazov That is correct, no records for gfx908 of the above/mentioned solvers

[JehandadKhan]

Perhaps I could have worded my earlier response better !