swiftsimio/visualisation/projection_backends/gpu.py , numba/cuda version issues

[Will-McD]

Using:
Swiftsimio 7.0.0
Numba 0.58.0

from numba import jit
from swiftsimio import mask as sw_mask

There is an instant error from numba/cuda and swiftsimio/visualisation/projection_backends/gpu

The final lines of the error messages are:
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuDevicePrimaryCtxRetain results in CUDA_ERROR_OUT_OF_MEMORY raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuDevicePrimaryCtxRetain results in CUDA_ERROR_OUT_OF_MEMORY

[MatthieuSchaller]

Which machine is this on?

[MatthieuSchaller]

@Will-McD ?

[Will-McD]

Which machine is this on?

I was able to produce this on Cosma8a and 8b, have not tried on any of the other cosma machines. But I have not had this error show up on any of the local machines

[JBorrow]

Are you sure this is a swiftsimio problem and not a numba problem?

[Will-McD]

Are you sure this is a swiftsimio problem and not a numba problem?

I think the issue is within swiftsimio. When I replace swiftsimio with astropy or my own "swiftsimio-like" visualisation functions that are essentially just the functions from swiftsimio without the dependancies on all the packages that swiftsimio has, but keep numba I have had no problems.

[JBorrow]

Can you make appropriate changes to optional_packages.py and submit it as a PR? I would really appreciate it. Looks like we need to guard against some other CUDA errors.

# Numba/CUDA
try:
    from numba.cuda.cudadrv.error import CudaSupportError

    try:
        import numba.cuda.cudadrv.driver as drv
        from numba import cuda
        from numba.cuda import jit as cuda_jit

        try:
            CUDA_AVAILABLE = cuda.is_available()
        except AttributeError:
            # Backwards compatibility with older versions
            # Check for the driver

            d = drv.Driver()
            d.initialize()

            CUDA_AVAILABLE = True

    except CudaSupportError:
        CUDA_AVAILABLE = False

except (ImportError, ModuleNotFoundError):
    # Mock the CudaSupportError so that we can raise it in cases
    # where we don't have numba installed.

    class CudaSupportError(Exception):
        def __init__(self, message):
            self.message = message

    CUDA_AVAILABLE = False

[MatthieuSchaller]

@Will-McD still an issue?

[andreagebek]

I received exactly the same error message on the login8 cosma node at multiple occasions yesterday and today, by simply running:
import swiftsimio
The issue is so far not persistent, and seems to only occasionally pop up. Also, on the login7 cosma node this issue did not occur so far.
swiftsimio version: 9.0.2
numba version: 0.60.0

[JBorrow]

export NUMBA_DISABLE_CUDA=1