ruser_mesh_data values on restart

[andreaantoni]

I have a time-dependent black hole interior to my spherical inner boundary (i.e. not on the grid) that is accreting mass through the inner boundary. The black hole contributes to the gravity felt by the gas that is on the grid. When I restart the simulation, I need to initialize the mass of the BH based on what it was at the time the restart file was generated.

Reading the Wiki, it seems like I should be able to store the BH mass in a ruser_mesh_data variable and that value will be written to the restart file and then available upon restart. Is that correct?

I am finding that after restart, the ruser_mesh_data variable is equal to 0 (it should be, e.g., 60) because the ruser_mesh_data arrays are allocated in InitUserMeshData, which is called again upon restart.

Am I misunderstanding what is supposed to happen to these data upon restart?

Thanks!
Andrea

[tomidakn]

While the user-defined mesh/meshblock data are stored in a restart file and restored in the restarting process,

only the data on the master node (rank 0) is written in the restarting file, and it is users' responsibility to maintain consistency between nodes,

as written on the Wiki. So please make sure that you synchronize the data stored in ruser_mesh_data. You can do so, for example, using MPI_Allreduce in Mesh::UserWorkInLoop.

[leeyp2009]

Could you share how you resolve this problem? I find that ruser_mesh_data is still the initial value when I update ruser_mesh_data in Mesh: UserWorkInLoop and synchronize the data with MPI_Allreduce.

Upon restart, I suppose that I need to load the stored data from ruser_mesh_data in Mesh::InitUserMeshData, but I find it is still the initial value (e.g., zero) when I first initialize ruser_mesh_data before restart.
The res_flag is not seen Mesh::InitUserMeshData, so I manually set a variable to determine when to read the stored data from ruser_mesh_data. Many thanks!

[andreaantoni]

I don't do anything special upon restart. Here is all I do with the ruser_mesh_data in my setup and I find that when I restart the simulation, ruser_mesh_data has the value that it was set to in the last call of Mesh::UserWorkInLoop before the prior simulation ended. Note that I am using this in a slightly older version of Athena++ (the rad_newtonian branch from before it was merged with master a ~month ago).

1. allocate and initialize at the start of the simulation (cycle=0; this isn't called again on restart):

void  Mesh::InitUserMeshData(ParameterInput *pin) {
  AllocateRealUserMeshDataField(1);
  ruser_mesh_data[0].NewAthenaArray(1);  
  ruser_mesh_data[0](0) = some_value;
}

2. While the simulation is running, I update the value on CPUs and all meshblocks to the same value:

void Mesh::UserWorkInLoop() {

 #ifdef MPI_PARALLEL
    Real tempdm;
    MPI_Allreduce(&dm_global, &tempdm, 1, MPI_ATHENA_REAL, MPI_SUM, MPI_COMM_WORLD);
    dm_global = tempdm;
#endif

Real &m_base_global = ruser_mesh_data[0](0);
m_base_global += dm_global;
dm_global=0;

return;
}

This doesn't seem very different from what you are doing, so maybe there is something subtle.
Have you tried doing a test on 1 MeshBlock and 1 CPU?

[leeyp2009]

Hi Andrea,
Many thanks for your quick response! I have tested with a similar approach as yours. But the Mesh::InitUserMeshData is actually called again for restart runs. So the ruser_mesh_data is re-set to the initial value upon restart (e.g., the "some_value" you mentioned above) in Mesh::UserWorkInLoop. So I am wondering how could you restore the m_base_global data before the restart? Thanks!

[leeyp2009]

The problem is resolved. There is indeed something subtle here. I forget to save the ruser_mesh_data in some temporary array before updating them. Thanks, Andrea.

[andreaantoni]

Glad it works for you now!

For others who may be wondering why the variables are correct on restart even though Mesh::InitUserMeshData is called on restart (which is where you would set the initial values at the very beginning of the simulation:

In src/mesh/mesh.cpp, the mesh generator for restarts, with function prototype Mesh::Mesh(ParameterInput *pin, IOWrapper& resfile, int mesh_test), does call InitUserMeshData(pin); but then it immediately overwrites the, e.g., ruser_mesh_data arrays with the values from the restart file, e.g. lines ~756-760:

    for (int n=0; n<nreal_user_mesh_data_; n++) {
      std::memcpy(ruser_mesh_data[n].data(), &(userdata[udoffset]),
                  ruser_mesh_data[n].GetSizeInBytes());
      udoffset += ruser_mesh_data[n].GetSizeInBytes();
    }
    delete [] userdata;
  }