Install a Config.cmake file

[ax3l]

Hi,

we are combining a couple of ROCm libraries in U.S. Exascale Computing Project libraries and applications.

We would like to include roctracer in ECP AMReX and dependent applications. To make this inclusion via CMake easier, would it be possible to add and install CMake *config.cmake files on install for roctracer?

The workflow we would prefer would export the roctracer CMake targets via such a file for simpler inclusion, similar as we do with rocprim & rocrand/hiprand:
https://github.com/AMReX-Codes/amrex/blob/db8e47927065fcf283f6aae631f91a07b222876b/Tools/CMake/AMReXParallelBackends.cmake#L210-L213

ROCm CMake package ref:
https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#rocm-cmake-packages

[ax3l]

Hi, is anyone working on this problem? :)

[ax3l]

Hi, I am still interested in making roctracer usage from CMake less awkward :)

https://github.com/AMReX-Codes/amrex/blob/f270b3d5db8f8b7ab010bc9134632361b8a9009c/Tools/CMake/AMReXParallelBackends.cmake#L251-L261

[skyreflectedinmirrors]

Hi @ax3l -- I can confirm there is a ticket (and work internally) at AMD for this, if you'd like I'd be happy to mirror this to the ORNL JIRA for tracking.

[skyreflectedinmirrors]

cc: @lmoriche

[ax3l]

Hi @arghdos, thank you, much appreciated. Just hit us again on Crusher today.

[upsj]

Just tagging along, since we are interested in a clean roctx integration as well with ginkgo-project/ginkgo#1055. In the meantime, I added a Find module which does essentially the same thing (the ROCTRACER_PATH is set outside and could be deduced from ROCM_PATH instead)

[cameronrutherford]

+1 - would be nice to have this support for packages like ExaGO / Re::Solve as even if we have a dependency expressed in our package.py, we cannot consume the installed version without quite a hack

[pelesh]

+1 It would be good to have CMake.config for ROC Tracer so it can be linked to in the same way as most of the other ROCm packages.

[ppanchad-amd]

@ax3l Can you please check with the latest ROCm 6.2 to see if your issue is resolved? Thanks

[naromero77amd]

This is something that is needed for PyTorch as well and is not present in ROCm 6.2. The CMake module should include IMPORTED targets as well.

[jrmadsen]

Hi @ax3l @naromero77amd @pelesh @cameronrutherford @upsj just wanted to provide an update here. We have pretty much stopped development on roctracer, rocprofiler, and rocprofiler v2. We designed a new profiling library from scratch called rocprofiler-sdk and did a beta release in ROCm 6.2. It is dramatically improved over the previous iterations. The API is relatively stable at this point and we’ve done hackathons with several tools at this point (TAU, Caliper, HPCToolkit, Linaro Map, CrayPAT, PAPI, etc.) and I believe y’all will be as happy with our new direction as they were. Y’all will be please to hear that

find_package(rocprofiler-sdk)
find_package(rocprofiler-sdk-roctx)

are fully supported with namespaced imported interface CMake targets. We are distributing a new ROCTx for various reasons: the old ROCTx couldn’t be separated from roctracer without breaking existing codes, you couldn’t install the old ROCTx without installing roctracer, etc. so this new ROCTx is designed to be standalone (in addition to adding new functions such as roctxProfilerPause and roctxProfilerResume).

We encourage all of y’all to transition to using the new APIs sooner rather than later and if y’all need a consult on the transition, please let me know and we can set something up.

[naromero77amd]

Thanks @jrmadsen for the quick reply. I found a buglet and hope the it can be that it will not be too much work to address:
ROCm/rocprofiler-sdk#21

[pelesh]

Thanks, @jrmadsen! I created an issue at Re::Solve repository, and we will start transition.

[naromero77amd]

@jrmadsen Hoping you can clarify this questions or point us to the relevant documentation.

Is librocprofiler-sdk-roctx.so compatible with libroctracer64.so?

As an initial step in the transition, I only included rocprofiler-sdk-roctx package and the associated imported target:

rocprofiler-sdk-roctx::rocprofiler-sdk-roctx

To my surprise, it seems to automagically work in PyTorch.

I am still seeing libroctracer64.so linked in -- perhaps because I didn't explicitly add the rocprofiler-sdk package (nor the imported target rocprofiler-sdk::rocprofiler-sdk).

[jrmadsen]

@jrmadsen Hoping you can clarify this questions or point us to the relevant documentation.

I’ve been asking our docs team to take care of this for a while. There aren’t docs posted yet unfortunately.

Is librocprofiler-sdk-roctx.so compatible with libroctracer64.so?

No. The ROCTx implementation differs and roctracer will attempt to use the old method to talk to ROCTx, which isn’t supported in the new ROCTx. Unless it got backported without my knowledge, which I’d be very surprised to hear, but I also really only concern myself with the new library and leave other people to manage our legacy tool libraries.

As an initial step in the transition, I only included rocprofiler-sdk-roctx package and the associated imported target:

rocprofiler-sdk-roctx::rocprofiler-sdk-roctx

To my surprise, it seems to automagically work in PyTorch.

I’m surprised myself if I am understanding you correctly. You just did a find_package and didn’t add it to the target_link_libraries(…) of anything?

If you mean you did the find_package and used the interface target, I am not surprised. It was probably able to find the <roctracer/roctx.h> from the include path from HIP or the roctracer target and the symbols were resolved by the rocprofiler-sdk-roctx library when linking.

I am still seeing libroctracer64.so linked in -- perhaps because I didn't explicitly add the rocprofiler-sdk package (nor the imported target rocprofiler-sdk::rocprofiler-sdk).

Yeah, you are likely linking to roctracer somewhere. Just FYI, supporting roctracer and rocprofiler-sdk in the same build is not recommended. We’ve successfully used the new rocprofv3 (built on rocprofiler-sdk) on PyTorch applications which contain roctracer, but I highly suspect that roctracer is going to have problems with anything HSA related when a rocprofiler-sdk tool is present. In other words, we haven’t seen any problems with rocprofiler-sdk when roctracer is present but we haven’t delved into how well roctracer works when rocprofiler-sdk is present.

[jrmadsen]

Well, for clarity, the rocprofiler-sdk docs are here: https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/

I was referring to the rocprofiler-sdk-roctx docs being presently missing.

[upsj]

So to clarify, when just using roctxRangePush/Pop, the only thing I would need to change to move from roctracer to rocprofiler-sdk-roctx is the target I'm linking against and the include path?

[jrmadsen]

So to clarify, when just using roctxRangePush/Pop, the only thing I would need to change to move from roctracer to rocprofiler-sdk-roctx is the target I'm linking against and the include path?

Yes, the function signatures of roctxRangePush/Pop in rocprofiler-sdk-roctx are identical to the ones provided by the roctracer roctx.

#include <roctracer/roctx.h> --> #include <rocprofiler-sdk-roctx/roctx.h>

and

libroctx64.so --> librocprofiler-sdk-roctx.so