Disable cudaDeviceReset() calls in pmcx

[lkeegan]

• If MCX_DISABLE_CUDA_DEVICE_RESET is defined then cudaDeviceReset() calls are not made in mcx_core.cu
• cudaDeviceReset() destroys all resources on the GPU associated with the current process
• for pmcx this can leave other python GPU libraries like pytorch in a broken state
• resolves Pytorch seems to be broken after calling pmcx.run() #187

[fangq]

Thanks a lot. the cudaDeviceReset() call was previously added for mcxlab (then inherited to pmcx) to solve exceptions thrown from the GPU kernel - when the GPU host/kernel code throws a CUDA error, we found that a new session can not be launched as the GPU is being occupied by the previous run. cudaDeviceReset was a quick fix to this, but I see it won't play nice with pytorch where persistent GPU memory is used.

I don't know if the GPU lockup issue will happen after this patch, we can do more tests to see.

[lkeegan]

So in the event of a CUDA error, you had to call cudaDeviceReset() before the process terminated to be able to use the GPU again in a new process?

If so, and this issue remains, I guess the equivalent of this for pmcx would be to call cudaDeviceReset() just before the Python process terminates.

I think one way to do this could be that pmcx registers a function to run atexit, which calls e.g. _pmcx.reset_device(). This would then be called (at least in most cases) when the Python process terminates.