Is there any tutorial about how to transfer conventional MPI to HPX_async_mpi?

[lj-cug]

I notice the HPX_async_mpi in HPX is experimental, so is there any example code teaching how to convert non-blocking MPI API into async_mpi? I want to compare the efficiency between them. Thanks
Li Jian

[hkaiser]

The async_mpi API follows closely the underlying MPI functions. For instance:

int status;
MPI_recv(buff, count, type, src, tag, comm, &status);

turns into:

hpx::mpi::experimental::executor exec(comm);
hpx::future<int> status = hpx::async(exec, MPI_recv, buff, count, type, src, tag);

where the future will hold the returned status (once the operation has finished executing). The same scheme can be applied to any MPI call that expects a communicator and a status as their last arguments.

Also, please note that you need to enable the integration of the async MPI functionalities with the HPX scheduler once before doing any async MPI calls:

hpx::mpi::experimental::enable_user_polling enable_polling;

You may want to look at this test to see everything in action: https://github.com/STEllAR-GROUP/hpx/blob/master/libs/core/async_mpi/tests/unit/mpi_ring_async_executor.cpp.

Also, please report back on the performance results you gathered. This will be interesting for others as well.

[lj-cug]

I have built an example to test the HPX_async_MPI functionalities comparing with non-blocking p2p communication for matrix multiplication. At present I just take async_mpi instead of MPI_Isend and MPI_Irecv. Before port into my CFD code, I have to study the usage of HPX_async_mpi. If I gather some information, I will put here.
My testing code repository: https://gitee.com/lijian-cug/P2P_HPX_async_MPI

[lj-cug]

Dear Sir:
I don't change the basic structure of my MPI parallelized code, in this style, can I use HPX async MPI to speedup my code in this case? For example, there's "MPI_Waitall(2, requests, MPI_STATUSES_IGNORE);" in my application, how to change this into HPX async_mpi API?
My MPI code structure is like the following:
if (world_rank == 0) {
// Some MPI_Isend() operations here. I can easily change these operations into HPX futures.
}
else {
MPI_Request requests[2];
MPI_Irecv(a_block, num_block_elements, MPI_DOUBLE, 0, a_tag, MPI_COMM_WORLD, &requests[0]);
MPI_Irecv(b_block, num_block_elements, MPI_DOUBLE, 0, b_tag, MPI_COMM_WORLD, &requests[1]);
MPI_Waitall(2, requests, MPI_STATUSES_IGNORE); // If no waitall() here, the results will be wrong.

// If I chaged above MPI_Irecv into HPX_Async_mpi API, and the results will be wrong.

}

[hkaiser]

Dear Sir: I don't change the basic structure of my MPI parallelized code, in this style, can I use HPX async MPI to speedup my code in this case? For example, there's "MPI_Waitall(2, requests, MPI_STATUSES_IGNORE);" in my application, how to change this into HPX async_mpi API? My MPI code structure is like the following: if (world_rank == 0) { // Some MPI_Isend() operations here. I can easily change these operations into HPX futures. } else { MPI_Request requests[2]; MPI_Irecv(a_block, num_block_elements, MPI_DOUBLE, 0, a_tag, MPI_COMM_WORLD, &requests[0]); MPI_Irecv(b_block, num_block_elements, MPI_DOUBLE, 0, b_tag, MPI_COMM_WORLD, &requests[1]); MPI_Waitall(2, requests, MPI_STATUSES_IGNORE); // If no waitall() here, the results will be wrong.

// If I chaged above MPI_Irecv into HPX_Async_mpi API, and the results will be wrong.

}

You perform the waiting throught the returned futures. They become ready once the corresponding operations have finished:

hpx::mpi::experimental::executor exec(MPI_COMM_WORLD);
auto result1 = hpx::async(exec, MPI_Irecv, a_block, num_block_elements, MPI_DOUBLE, 0, a_tag);
auto result2 = hpx::async(exec, MPI_Irecv, b_block, num_block_elements, MPI_DOUBLE, 0, b_tag);
hpx::wait_all(result1, result2);

[lj-cug]

Dear Sir:
Now I can run my MPI parallelized matrix-multiplication test code in HPX runtime, sure, HPX async_MPI can run and get right result like that using MPI APIs, but only can be used in relatively little matrix and HPX costed more time that MPI version. When I use big sized matrix, like 800x800, the error happened.
At present, I run my test code on my laptop using openMPI-3.3.
The error information is:
{stack-trace}: 18 frames:
......
0x563a4c09beb1 : ./p2p_hpx(+0xbeb1) [0x563a4c09beb1] in ./p2p_hpx
0x7ff867601b6b : hpx::runtime::run_helper(hpx::function<int (), false> const&, int&, bool) [0x70b] in /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so
0x7ff867606a2c : /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so(+0x1f6a2c) [0x7ff867606a2c] in /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so
0x7ff8675331ed : hpx::threads::coroutines::detail::coroutine_impl::operator()() [0xed] in /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so
0x7ff86753256f : /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so(+0x12256f) [0x7ff86753256f] in /home/lijian/HPX/hpx-1.8.1/install/lib/libhpx_core.so
{what}: Segmentation fault

[hkaiser]

I would have to see the code or a small reproducing example to be able to diagnose what's the issue.

[lj-cug]

Dear Sir:
Thank you very much for your help. Now I use tcmalloc, the problem was solved.
The other problem is that the test code developed using HPX was slower than the code in MPI non-blocking communication. Now I use my laptop with 8 cores to run the application. I guess the advantage of HPX async MPI on one node cannot perform. I should test the code on the cluster in the next step.
Regards,
Li Jian
from China