[HIP][MI100] Virtual Memory Management API on MI100?

[crozhon]

Does MI100 support the virtual memory management API? (functions hipMemAddressReserve, hipMemMap). I've been following this test case (https://github.com/ROCm-Developer-Tools/hip-tests/blob/094b9af382585a079d13c735fd170e69bea735b6/catch/unit/memory/hipMemVmm.cc), and virtual memory is always unsupported on all of our devices.

I've attached a small test program that queries for support and it always returns 0/false on rocm-5.4.2.

hipDeviceAttributeVirtualMemoryManagementSupported : 0

Is there something wrong with my setup or are these APIs just not supported on these devices?

#include <cstdio>

#include <hip/hip_runtime_api.h>

int main() {
    int vmm = 0;
    hipError_t result = hipDeviceGetAttribute(&vmm, hipDeviceAttributeVirtualMemoryManagementSupported, 0);
    if (result != hipSuccess) {
        return -1;
    }

    std::printf("hipDeviceAttributeVirtualMemoryManagementSupported : %d\n", vmm);
    return 0;
}

[b-sumner]

Which OS are you running? For Ubuntu 20.04, you need the HWE stack. I'm not sure about other distros.

[crozhon]

Yup Ubuntu 20.04. Thanks, that makes sense - I'll try switching the kernel.

[crozhon]

Switched to the HWE stack and reinstalled, but it's still coming back as not supported. Here's my kernel and package versions:

$dkms status
amdgpu, 5.18.13-1528701.20.04, 5.15.0-58-generic, x86_64: installed

hip-dev/focal,now 5.4.22803.50402-104~20.04 amd64 [installed,automatic]
rocm-dev/focal,now 5.4.2.50402-104~20.04 amd64 [installed]

[emankov]

I do not see here something HIP RT-related:

  case hipDeviceAttributeVirtualMemoryManagementSupported:
    *pi = static_cast<int>(g_devices[device]->devices()[0]->info().virtualMemoryManagement_);
    break;

Looks like HW or HSA RT specifics.

@gargrahul, could you please have a look too?

[jowens]

Hey @gargrahul and @emankov, can we check back on this? Possible for us to get more info on your roadmap here / when it might be available? Email is fine, thanks. cc: @crozhon

[emankov]

@gargrahul, @mangupta could you please let us know to whom we should address this ticket? We suspect HSA RT/HW.

Thank you in advance!

[mangupta]

@crozhon : The ROCm driver does not support Virtual Memory Management and hence HIP runtime reports hipDeviceAttributeVirtualMemoryManagementSupported as 0. The support is being added in a future ROCm release. Please keep an eye out on the ROCm release notes to track this.

[ppanchad-amd]

@crozhon We have an internal ticket to add support in future ROCm release. Will provide ROCm release info when available. Thanks!