Fix gpu column integrals #1380

charleskawczynski · 2023-07-17T17:09:09Z

Closes #1376.

src/Operators/integrals.jl

1381: Add CUDA field tests r=charleskawczynski a=charleskawczynski This PR adds CUDA field tests (and applies some fixes + adds some notes). We should probably do this before #1380 so that we only need to slightly modify the test suite. This was never actually tested. Here is a snapshot before #1321, which shows that `field.jl` was only on buildkite, and didn't have a gpu job: https://github.com/CliMA/ClimaCore.jl/tree/f38795b0769a6134afbba96ac1e66318c976ccd0 Co-authored-by: Charles Kawczynski <kawczynski.charles@gmail.com>

1381: Add CUDA field tests r=charleskawczynski a=charleskawczynski This PR adds CUDA field tests (and applies some fixes + adds some notes). We should probably do this before #1380 so that we only need to slightly modify the test suite. This was never actually tested. Here is a snapshot (https://github.com/CliMA/ClimaCore.jl/tree/f38795b0769a6134afbba96ac1e66318c976ccd0) before #1321, which shows that `field.jl` was only on buildkite, and didn't have a gpu job Co-authored-by: Charles Kawczynski <kawczynski.charles@gmail.com>

charleskawczynski · 2023-07-18T18:03:25Z

Now, locally, I'm getting:

Definite column integrals bycolumn: Error During Test at dev\ClimaCore.jl\test\Fields\field.jl:693
  Got exception outside of a @test
  InvalidIRError: compiling MethodInstance for ClimaCore.Operators.column_integral_definite_kernel!(::ClimaCore.Fields.Field{IJFH{Float64, 4, SubArray{Float64, 4, CUDA.CuDeviceArray{Float64, 5, 1}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}}, true}}, ClimaCore.Spaces.SpectralElementSpace2D{Nothing, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float64}, IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float64}, Float64, SMatrix{3, 3, Float64, 9}}, 4, SubArray{Float64, 4, CUDA.CuDeviceArray{Float64, 5, 1}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}}, true}}, IJFH{Float64, 4, CUDA.CuDeviceArray{Float64, 4, 1}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float64, ClimaCore.Geometry.UVVector{Float64}}, 4, CUDA.CuDeviceArray{Float64, 3, 1}}, NamedTuple{(), Tuple{}}}}, ::ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{Float64, 4, SubArray{Float64, 5, CUDA.CuDeviceArray{Float64, 5, 1}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}, Base.Slice{Base.OneTo{Int64}}}, false}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Spaces.CellCenter, ClimaCore.Spaces.SpectralElementSpace2D{Nothing, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float64}, IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float64}, Float64, SMatrix{2, 2, Float64, 4}}, 4, CUDA.CuDeviceArray{Float64, 4, 1}}, IJFH{Float64, 4, CUDA.CuDeviceArray{Float64, 4, 1}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float64, ClimaCore.Geometry.UVVector{Float64}}, 4, CUDA.CuDeviceArray{Float64, 3, 1}}, NamedTuple{(), Tuple{}}}, ClimaCore.Topologies.DeviceIntervalTopology{NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, ClimaCore.Spaces.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float64}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float64}, Float64, SMatrix{3, 3, Float64, 9}}, 4, CUDA.CuDeviceArray{Float64, 5, 1}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float64}, Float64, SMatrix{3, 3, Float64, 9}}, 4, CUDA.CuDeviceArray{Float64, 5, 1}}}}) resulted in invalid LLVM IR
  Reason: unsupported call through a literal pointer (call to memcmp)
  Stacktrace:
   [1] _memcmp
     @ .\strings\string.jl:113
   [2] ==
     @ .\strings\string.jl:125
   [3] multiple call sites
     @ unknown:0
  Reason: unsupported dynamic function invocation (call to ClimaComms.MPICommsContext)
  Stacktrace:
   [1] MPICommsContext
     @ C:\Users\kawcz\.julia\packages\ClimaComms\jYrr9\src\mpi.jl:16
   [2] context
     @ C:\Users\kawcz\.julia\packages\ClimaComms\jYrr9\src\context.jl:28
   [3] context
     @ dev\ClimaCore.jl\src\Fields\Fields.jl:43
   [4] column
     @ dev\ClimaCore.jl\src\Spaces\spectralelement.jl:655
   [5] column
     @ dev\ClimaCore.jl\src\Fields\Fields.jl:161
   [6] column_integral_definite_kernel!
     @ dev\ClimaCore.jl\src\Operators\integrals.jl:40
  Reason: unsupported call through a literal pointer (call to ijl_alloc_array_1d)
  Stacktrace:
   [1] Array

charleskawczynski · 2023-07-18T18:04:20Z

So, I suspect

function _column_integral_definite!(
    col∫field::Fields.PointField,
    field::Fields.ColumnField,
)
    @inbounds col∫field[] = _column_integral_definite(field)
    return nothing
end

function _column_integral_definite(field::Fields.ColumnField)
    field_data = Fields.field_values(field)
    Δz_data = Spaces.Δz_data(axes(field))
    Nv = Spaces.nlevels(axes(field))
    ∫field = zero(eltype(field))
    @inbounds for j in 1:Nv
        ∫field += field_data[j] * Δz_data[j]
    end
    return ∫field
end

is not cuda-capable

simonbyrne · 2023-07-18T19:44:49Z

the error appears to be the call to column, specifically this line:

ClimaCore.jl/src/Spaces/spectralelement.jl

Line 655 in 14ae299

context = ClimaComms.context(space)

I guess we didn't test the thomas algorithm on the GPU with MPI

charleskawczynski · 2023-07-18T20:39:56Z

the error appears to be the call to column, specifically this line:

ClimaCore.jl/src/Spaces/spectralelement.jl

Line 655 in 14ae299

context = ClimaComms.context(space)

I guess we didn't test the thomas algorithm on the GPU with MPI

Bummer, ok. What's the fix here? I don't immediately see what is wrong with a call to ClimaComms.context. It looks like it just leads to a getproperty call

charleskawczynski · 2023-07-19T22:03:44Z

bors try

bors · 2023-07-19T22:23:39Z

try

Build failed:

buildkite/climacore-ci

charleskawczynski · 2023-07-19T23:01:27Z

bors try

bors · 2023-07-19T23:20:32Z

try

Build failed:

buildkite/climacore-ci

charleskawczynski · 2023-07-19T23:56:58Z

@simonbyrne, should we just use if !(space isa PointSpace) around allreduce!(context, ...) in the reduction operators? (sum, maximum etc.)?

simonbyrne · 2023-07-20T16:01:54Z

bors try

bors · 2023-07-20T16:37:03Z

try

Build failed:

buildkite/climacore-ci

simonbyrne · 2023-07-20T17:49:22Z

bors try

bors · 2023-07-20T18:11:13Z

try

Build failed:

buildkite/climacore-ci

simonbyrne · 2023-07-20T22:33:30Z

bors try

bors · 2023-07-20T22:57:51Z

try

Build failed:

buildkite/climacore-ci

simonbyrne · 2023-07-20T23:45:55Z

bors try

bors · 2023-07-21T00:14:40Z

try

Build failed:

buildkite/climacore-ci

simonbyrne · 2023-07-21T04:17:15Z

I think this is ready to go.

simonbyrne · 2023-07-21T04:17:20Z

bors try

bors · 2023-07-21T04:59:29Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

simonbyrne · 2023-07-21T18:05:12Z

Okay, it looks like it works on ClimaAtmos:
https://buildkite.com/clima/climaatmos-ci/builds/11371#0189797b-ed7f-4290-80fb-06f7529bc1da
so i'll squash and merge.

simonbyrne · 2023-07-21T18:06:58Z

bors r+

bors · 2023-07-21T18:52:40Z

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

simonbyrne reviewed Jul 17, 2023

View reviewed changes

src/Operators/integrals.jl Outdated Show resolved Hide resolved

charleskawczynski mentioned this pull request Jul 17, 2023

Add CUDA field tests #1381

Merged

charleskawczynski force-pushed the ck/1376 branch from 874b139 to 692309c Compare July 17, 2023 20:44

charleskawczynski mentioned this pull request Jul 18, 2023

Fix some allocations and gpu moist example CliMA/ClimaAtmos.jl#1882

Closed

charleskawczynski force-pushed the ck/1376 branch from 692309c to 14ae299 Compare July 18, 2023 18:00

bors bot added a commit that referenced this pull request Jul 19, 2023

Try #1380:

d70640b

bors bot added a commit that referenced this pull request Jul 19, 2023

Try #1380:

1a44bee

bors bot added a commit that referenced this pull request Jul 20, 2023

Try #1380:

af038ab

bors bot added a commit that referenced this pull request Jul 20, 2023

Try #1380:

4b8ae9f

bors bot added a commit that referenced this pull request Jul 20, 2023

Try #1380:

eeb31b4

simonbyrne force-pushed the ck/1376 branch from 0dd6265 to dff77b0 Compare July 20, 2023 23:45

bors bot added a commit that referenced this pull request Jul 20, 2023

Try #1380:

3517017

simonbyrne marked this pull request as ready for review July 21, 2023 04:17

simonbyrne approved these changes Jul 21, 2023

View reviewed changes

bors bot added a commit that referenced this pull request Jul 21, 2023

Try #1380:

392c864

simonbyrne mentioned this pull request Jul 21, 2023

update ClimaCore CliMA/ClimaAtmos.jl#1902

Closed

1 task

charleskawczynski and others added 2 commits July 21, 2023 11:06

Fix gpu column integrals

6ff2dd2

disable nsys

586a841

simonbyrne force-pushed the ck/1376 branch from bd87cbc to 586a841 Compare July 21, 2023 18:06

bump patch

f94b146

bors bot merged commit 7120a4f into main Jul 21, 2023
5 of 6 checks passed

bors bot deleted the ck/1376 branch July 21, 2023 18:52

simonbyrne mentioned this pull request Oct 10, 2023

Define indefinite column integrals for arbitrary functions #1492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix gpu column integrals #1380

Fix gpu column integrals #1380

charleskawczynski commented Jul 17, 2023 •

edited

Loading

charleskawczynski commented Jul 18, 2023

charleskawczynski commented Jul 18, 2023

simonbyrne commented Jul 18, 2023 •

edited

Loading

charleskawczynski commented Jul 18, 2023

charleskawczynski commented Jul 19, 2023

bors bot commented Jul 19, 2023

charleskawczynski commented Jul 19, 2023

bors bot commented Jul 19, 2023

charleskawczynski commented Jul 19, 2023

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

simonbyrne commented Jul 20, 2023

bors bot commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

bors bot commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

bors bot commented Jul 21, 2023

Fix gpu column integrals #1380

Fix gpu column integrals #1380

Conversation

charleskawczynski commented Jul 17, 2023 • edited Loading

charleskawczynski commented Jul 18, 2023

charleskawczynski commented Jul 18, 2023

simonbyrne commented Jul 18, 2023 • edited Loading

charleskawczynski commented Jul 18, 2023

charleskawczynski commented Jul 19, 2023

bors bot commented Jul 19, 2023

try

charleskawczynski commented Jul 19, 2023

bors bot commented Jul 19, 2023

try

charleskawczynski commented Jul 19, 2023

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

try

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

try

simonbyrne commented Jul 20, 2023

bors bot commented Jul 20, 2023

try

simonbyrne commented Jul 20, 2023

bors bot commented Jul 21, 2023

try

simonbyrne commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

bors bot commented Jul 21, 2023

try

simonbyrne commented Jul 21, 2023

simonbyrne commented Jul 21, 2023

bors bot commented Jul 21, 2023

charleskawczynski commented Jul 17, 2023 •

edited

Loading

simonbyrne commented Jul 18, 2023 •

edited

Loading