Upgrades KernelAbstractions with pinned GPUCompiler

[tomchor]

Same as #2899 but pinning GPUCompiler to version 0.16.4.

This pinning made tests pass in #2865

[vchuravy]

@tomchor Which version of GPUCompiler are you trying to pin to? It's not listed in Project.toml and so the pin is not effective.

[tomchor]

@tomchor Which version of GPUCompiler are you trying to pin to? It's not listed in Project.toml and so the pin is not effective.

@vchuravy Maybe pinning wasn't the right word to use. I'm doing ]add GPUCompiler@0.16.4 and then ]rm GPUCompiler, which ensures that Manifest.toml retains version 0.16.4 even though GPUCompiler doesn't appear in Project.toml.

That's probably not the best-practices way to do things, but afaik it works. That was the only way I could get tests passing in #2865. The issues I was seeing there were very similar to the issues that appeared in #2782 and to some degree in #2899, so I suspect keeping GPUCompiler.jl will help make those tests pass (or least narrow down what the issues are).

The reason tests didn't pass here here is because there's an error related to the always_inline=true flag, which (based on your comment here, hasn't made it into the main channel.

[glwagner]

best-practices way to do things, but afaik it works

it only works within the Oceananigans environment, not for users of Oceananigans

[vchuravy]

How sure are you about the precise version? Is GPUCompiler@0.16.5 the breaking release for you?

[tomchor]

best-practices way to do things, but afaik it works

it only works within the Oceananigans environment, not for users of Oceananigans

True. Although I wasn't able to reproduce the failing test results in any of the several GPUs I tried (they were either Tesla V100s or Quadro GP100). Every single time I tried to run GPU tests locally, they passed. Is it possible that the tests are running on a GPU that simply isn't supported anymore by one of the packages?

[tomchor]

How sure are you about the precise version? Is GPUCompiler@0.16.5 the breaking release for you?

I haven't tried to find the precise version, but GPUCompiler@0.16.7 already creates the errors, so the breaking release is either 0.16.7 or 0.16.6.

[vchuravy]

The most suspicious PR is JuliaGPU/GPUCompiler.jl#359 by yours truly,

[glwagner]

True. Although I wasn't able to reproduce the failing test results in any of the several GPUs I tried (they were either Tesla V100s or Quadro GP100). Every single time I tried to run GPU tests locally, they passed. Is it possible that the tests are running on a GPU that simply isn't supported anymore by one of the packages?

Here's GPU + driver info

glwagner@sverdrup:~$ nvidia-smi -q

==============NVSMI LOG==============

Timestamp                                 : Sun Feb  5 17:58:59 2023
Driver Version                            : 465.27
CUDA Version                              : 11.3

Attached GPUs                             : 1
GPU 00000000:82:00.0
    Product Name                          : NVIDIA Quadro P6000
    Product Brand                         : Quadro