How to pause/resume main chare while workers are working?

[fclingen]

After some helpful pointing to the ldbRedDemo.f90 example for my last question, I still have some questions about how to add a Charm++ reduction to the legacy Fortran code I'm trying to refactor.

What I have, is 3,000 lines of code with some REALLY expensive computations on a number of multidimensional arrays in the middle of it, and a lot of serial code bracketing the expensive operations that I don't want distributed to multiple workers

The legacy code I have is all one big file. a semi-pseudo-code is given below:

PROGRAM MAINPROGRAM
      COMPLEX A(NY,NZ,NX), B(NY,NZ,NX)! Every array in legacy code is modified in-place
      CALL LOAD_INIT_CONDITIONS_FROM_FILE (A,B,C,D) ! pull data from disk and do pre-processing to initialize dataset
      CALL PROCESS_A(A) ! A will be replaced with the processed version of A after calling
      CALL PROCESS_B(B) ! B will be replaced with the processed version of A after calling
      CALL WRITE_RESULTS(A,B)
END PROGRAM

SUBROUTINE PROCESS_A(A)
COMPLEX A (NY,NZ,NX)  ! Note the index order--none of the "layers" in the code are actually in contiguous memory since Fortran is column-major
REAL U(NZ,NX),V(NZ,NX),W(NZ,NX) !there are actually 10-50 of these arrays in the real code
REAL L(NX,NX), M(NZ,NX), N(NZ,NX)
REAL INPUT_PARAMS(NZ,NX)
INTEGER I

DO I=1,NY  ! Every NY layer DOES NOT depend on results from other layers.  In practice NY is roughly  50-ish
      CALL SETUP_UVW(I, A,U,V,W) ! Computations actually happen on U, V, and W, and the results of those computations get inserted into A one (NZ,NX) layer at a time
      
      CALL FFT_THINGS(U) !  These subroutines are where the legacy code spends 80% of its processing time
      CALL FFT_THINGS(V)  !  Each call to PROCESS is mutually independent, though the call to AGGREGATE needs results from all the work
      CALL FFT_THINGS(W) ! This is what I'm wanting to put into a chare array

! THIS IS WHERE I WOULD NEED TO SUSPEND A SUB-CHARE AND WAIT FOR RESULTS

      CALL DO_SOME_ARITHMETIC(U,V,W,L,M,N) ! This function is cheap, but needs results from all of U,V,W to run

     CALL MORE_FFT_THINGS(L)
     CALL MORE_FFT_THINGS(M)
     CALL MORE_FFT_THINGS(N)

     CALL AGGREGATE_UVWLMN(I,A,U,V,W,L,M,N)  ! The I-th (NZ,NX) layer will be modified here
END DO

! THIS IS WHERE I WOULD NEED TO SUSPEND THE MAIN CHARE TO WAIT FOR RESULTS

     CALL WRITE_TO_AN_OUTPUT_FILE(A)
END SUBROUTINE

IDEALLY, I want every call to FFT_THINGS or MORE_FFT_THINGS to happpen in its own chare, nested withing every NY layer happening within it's own chare (so I could have up to 300-ish calls to FFT_THINGS happening concurrently). In my mind, refactoring this code means refactoring the PROGRAM into a subroutine, and calling that subroutine within a main chare. After that, all the refactoring is going to happen in PROCESS_A and PROCESS_B--for both, it means Invoking entry methods on chars that are responsible for inidividual layers, which in turn invoke chares responsible for doing the invidiual FFTs.

As far as I've gotten, however, is just trying to get a main chare that will invoke the FFT functions and properly get back results using the Fortran90 bindings. What has me stuck is figuring out exactly how to get results out of chares called from the main chare, for a number of reasons:

1. I don't know how to get main chare to asynchronously launch chares, suspend while the chares work, then resume where it left off once they have returned results. I think I have enough of a grok on Charm++ I understand how it should work in principle with CkCallbacks, but I don't know how those are implemented in f90. The only example I've seen in Fortran callbacks is with reductions (Big Showstopper).
2. Speaking of reductions, I don't see any way to pass back one piece of the larger array back at a time using reductions. Again, the mechanism in Charm++ seems to exist in CkReduction::concat, but that reduction type is not one of the "simple" reduction types available in f90charm. It appears if I want to do that type of reduction I'd have to do some janky stuff with a SUM-type reduction on an array that's zeroed out everywhere but in the area where the workers did their work (janky, but not a show-stopper)
3. There's going to be A LOT of copying data to re-shape it (because processing is happening on strided data) once I actually get to the point of parallelizing the loop (still kinda janky, but not so bad)

[rbuch]

I'm not sure I fully understand your question, but hopefully this helps a bit. So long as the main chare has no pending messages for it, it will "wait" in an idle state until it receives something. The canonical way to do the sort of synchronization you're talking about is through reductions (perhaps multiple reductions if you need to synchronize over chare arrays of multiple types). AtSync isn't truly a barrier, the call doesn't block and ResumeFromSync is called immediately if there is no "receiver" for AtSync ("receivers" are basically services that will be called when every participating element has called AtSync). The semantics of AtSync are that any object that calls AtSync will later get its ResumeFromSync called by the runtime, your chare won't resume from where it called AtSync.

[fclingen]

I updated my OP to maybe make it more clear what I'm trying to accomplish.

There is a LOT of computation happening within each chare (in principal), but I don't think concat is a reduction type available? When I look here:
https://github.com/UIUC-PPL/charm/blob/main/src/ck-core/charmmod.f90
I don't see a parameter definition for a concat-type callback. I'd assumed that meant it hadn't been implemented fro f90?

[sanjaykale]

You are right. The C++ version does support concat. https://charm.readthedocs.io/en/latest/charm++/manual.html?highlight=concat#built-in-reduction-types but based on what you show, fortran version does not. But this might be an omission easy to fix. OTOH, if the data each chare is contributing is large, it may be ok to send it in separate direct message to the root. At least as a stopgap solution.

[fclingen]

That still leaves me uncertain if how to actually synchronize passing the message to the root. I'm back to wondering how to implement a CkCallbackReserveThread in Fortran

[fclingen]

OK, after playing with it some it seems I can do a concat-type reduction--I modified the ldbRedDemo to make it happen. It's just not represented in the parameters listed in charmmod.f90. However, if you put '68' in for the reduction-type argument in _contribute() the system knows what to do with it.

[fclingen]

I think I'm getting somewhere. It took a bit of creativity, but I got something workable for my application.

Instead of having multiple chare classes, I have a single 1D chare array with a a threaded "main" entry method that runs the code I am overhauling, as well as entry methods that do the parallelizable bits. I run the main entry method on a single chare, and when I want to parallelize computation I invoke the parallelizable methods from my main chare and immediately call CkWaitQD to let them finish their computations and message passing. When they're done CkWaitQD will resume my serial code where it left off.

[stwhite91]

Yes, this sounds like a good approach. We can work on adding real support for concat and other types of reductions to the Fortran interface too.

[fclingen]

While I haven't tested it fully (as in, not doing any instrumentation or anything, just running with what works), it seems like it'd be pretty easy, just add a CHARM_CONCAT=68 parameter to charmmod.f90

In order to use CkWaitQD, I had add include "ckwaitdqd" and add an extern "C" void ckwaitqd_(void){CkWaitQD()} to my .C file so Fortran could see it. That would probably be better in the code that generates the '.decl.h' file.