Adds distributed row gatherer #1589

MarcelKoch · 2024-04-04T10:49:19Z

This PR adds a distributed row gatherer. This operator essentially provides the communication required in our matrix apply.

Besides the normal apply (which is blocking), it also provides two asynchronous calls. One version has an additional workspace parameter which is used as send buffer. This version can be called multiple times without restrictions, if different workspaces are used for each call. The other version doesn't have a workspace parameter, and instead uses an internal buffer. As a consequence, this function can only be called a second time, if the request of the previous call has been waited on. Otherwise, this function will throw.

This is the second part of splitting up #1546.

It also introduces some intermediate changes, which could be extracted out beforehand:

~~a type-erased DenseCache~~
~~makingdetail::run easier to use~~ now part of Use index_map in distributed::matrix #1544

PR Stack:

MarcelKoch · 2024-04-22T08:29:49Z

One issue that I have is the constructor. It takes a collective_communicator and an index_map. The index_map already defines the communication pattern, so the collective_communicator has to match that.
One option might be to have a virtual function like

std::unique_ptr<collective_communicator> create_with_same_type(communicator, index_map);

If I can't come up with anything better, I guess I will use that.

pratikvn · 2024-04-26T09:50:57Z

Do we need to have the std::future setup for the release ? Can we remove that for now and just use a normal synchronous approach ? I think that is a significant change that maybe needs more thought and probably a separate PR.

pratikvn

Really nice work! LGTM!

core/distributed/row_gatherer.cpp

pratikvn · 2024-07-23T08:01:34Z

core/distributed/row_gatherer.cpp

+    int is_inactive;
+    MPI_Status status;
+    GKO_ASSERT_NO_MPI_ERRORS(
+        MPI_Request_get_status(req_listener_, &is_inactive, &status));


Can we maybe move this MPI function into mpi.hpp and create a wrapper for it ?

That doesn't really work here, since this function would be a member function of request, but I'm using a bare MPI_Request (and can't use request, because it will try to free the request in the destructor), so it would not be applicable.

pratikvn · 2024-07-23T08:08:09Z

include/ginkgo/core/distributed/row_gatherer.hpp

+
+    mutable array<char> send_workspace_;
+
+    mutable MPI_Request req_listener_{MPI_REQUEST_NULL};


This can be of type mpi::request ?

No, because the destructor of mpi::request will try to free the request. But req_listenser_ doesn't own any requests, so the program would crash.

pratikvn · 2024-07-23T09:10:12Z

core/distributed/row_gatherer.cpp

+template <typename LocalIndexType>
+void RowGatherer<LocalIndexType>::apply_impl(const LinOp* alpha, const LinOp* b,
+                                             const LinOp* beta, LinOp* x) const
+    GKO_NOT_IMPLEMENTED;


I think you can also implement the advanced apply by replacing b_local->row_gather(idxs, buffer) by b_local->row_gather(alpha, idxs, beta, buffer) ?

core/distributed/matrix.cpp

yhmtsai · 2024-10-07T13:26:19Z

core/multigrid/pgm.cpp

-                      send_sizes.data(), send_offsets.data(), type, recv_ptr,
-                      recv_sizes.data(), recv_offsets.data(), type);
+    coll_comm
+        ->i_all_to_all_v(use_host_buffer ? exec->get_master() : exec, send_ptr,


any difference between using all_to_all_v vs i_all_to_all_v? I assume all_to_all_v also update the interface

all_to_all_v is a blocking call, while i_all_to_all_v is non-blocking. Right now the collective_communicator only provides the non-blocking interface, since it is more general.

include/ginkgo/core/distributed/row_gatherer.hpp

yhmtsai · 2024-10-07T13:28:56Z

include/ginkgo/core/distributed/row_gatherer.hpp

+ * auto x = matrix::Dense<double>::create(...);
+ *
+ * auto future = rg->apply_async(b, x);
+ * // do some computation that doesn't modify b, or access x


I think it access x but it is unclear when it will be accessed before the wait

I guess this just meant to say that you can't expect any meaningful data when accessing x before the wait has completed.

I think I get it wrong.
Is the comment here to describe that user can do something safely after the call or the apply_async behavior?
My comment was based on that it is the behavior of the apply_async because apply_async definitely accesses x.
If it is for user action during async and wait, then it is correct.

core/distributed/row_gatherer.cpp

yhmtsai · 2024-10-23T08:37:11Z

core/distributed/row_gatherer.cpp

+                    workspace.set_executor(mpi_exec);
+                    if (send_size_in_bytes > workspace.get_size()) {
+                        workspace.resize_and_reset(sizeof(ValueType) *
+                                                   send_size[0] * send_size[1]);
+                    }


combining them to assign the workspace directly?

Combine how? Do you mean like

workspace = array<char>(mpi_exec, sizeof(ValueType) * send_size[0] * send_size[1]);

yhmtsai · 2024-10-23T08:41:27Z

core/distributed/row_gatherer.cpp

+                    req = coll_comm_->i_all_to_all_v(
+                        mpi_exec, send_ptr, type.get(), recv_ptr, type.get());


send_buffer might be on the host but the recv_ptr(x_local) might be on the device

I have a check above to ensure that the memory space of the recv buffer is accessible from the mpi executor. So if GPU aware MPI is used, it should work (even if send buffer is on the host and recv buffer in the device or vice versa). Otherwise an exception will be thrown.

core/test/mpi/distributed/row_gatherer.cpp

- only allocate if necessary - synchronize correct executor Co-authored-by: Pratik Nayak <pratik.nayak@kit.edu>

- split tests into core and backend part - fix formatting - fix openmpi pre 4.1.x macro Co-authored-by: Pratik Nayak <pratik.nayak4@gmail.com> Co-authored-by: Yu-Hsiang M. Tsai <yhmtsai@gmail.com>

MarcelKoch self-assigned this Apr 4, 2024

ginkgo-bot added reg:build This is related to the build system. reg:testing This is related to testing. mod:core This is related to the core module. type:matrix-format This is related to the Matrix formats labels Apr 4, 2024

MarcelKoch requested a review from pratikvn April 4, 2024 10:49

MarcelKoch force-pushed the distributed-row-gatherer branch from 6b4521b to ae60198 Compare April 4, 2024 11:00

MarcelKoch force-pushed the neighborhood-communicator branch from 6acf7c4 to 8aa6ab9 Compare April 4, 2024 11:00

MarcelKoch force-pushed the distributed-row-gatherer branch 2 times, most recently from 49557f1 to 4a79442 Compare April 5, 2024 08:18

MarcelKoch modified the milestone: Ginkgo 1.8.0 Apr 5, 2024

MarcelKoch force-pushed the neighborhood-communicator branch from 8aa6ab9 to 77398bd Compare April 17, 2024 16:28

MarcelKoch force-pushed the distributed-row-gatherer branch from 4a79442 to 172eb7d Compare April 17, 2024 16:28

MarcelKoch requested a review from upsj April 19, 2024 09:20

MarcelKoch mentioned this pull request Apr 19, 2024

Adds sparse communicator class #1546

Closed

7 tasks

MarcelKoch force-pushed the neighborhood-communicator branch from 77398bd to d278cad Compare April 19, 2024 14:39

MarcelKoch force-pushed the distributed-row-gatherer branch 2 times, most recently from 98fa10a to 79de4c3 Compare April 19, 2024 16:19

This was referenced Apr 22, 2024

Add segmented array type #1545

Merged

Distributed Index Map #1543

Merged

Adds Index map device kernels #1579

Merged

Use index_map in distributed::matrix #1544

Merged

MarcelKoch force-pushed the distributed-row-gatherer branch from 79de4c3 to b0e5c92 Compare April 22, 2024 11:11

MarcelKoch force-pushed the neighborhood-communicator branch from d278cad to d6112ef Compare April 22, 2024 11:11

MarcelKoch force-pushed the distributed-row-gatherer branch from b0e5c92 to 775854a Compare April 25, 2024 07:16

MarcelKoch force-pushed the neighborhood-communicator branch from d6112ef to 1582673 Compare April 25, 2024 07:16

MarcelKoch force-pushed the neighborhood-communicator branch from 8e3e932 to 40cd2c0 Compare July 18, 2024 16:39

MarcelKoch force-pushed the distributed-row-gatherer branch 2 times, most recently from 98dcc4f to 5e970e9 Compare July 18, 2024 17:00

MarcelKoch force-pushed the neighborhood-communicator branch from 40cd2c0 to fe864bb Compare July 18, 2024 17:00

MarcelKoch force-pushed the distributed-row-gatherer branch from 5e970e9 to 0a8e28c Compare July 19, 2024 08:18

MarcelKoch force-pushed the neighborhood-communicator branch from fe864bb to db7f6ed Compare July 19, 2024 08:18

pratikvn approved these changes Jul 23, 2024

View reviewed changes

MarcelKoch force-pushed the distributed-row-gatherer branch from 0a8e28c to bfc5233 Compare August 9, 2024 13:40

MarcelKoch force-pushed the neighborhood-communicator branch from db7f6ed to 0ad4ee8 Compare August 9, 2024 13:40

MarcelKoch force-pushed the distributed-row-gatherer branch from bfc5233 to 8697971 Compare August 16, 2024 15:21

MarcelKoch force-pushed the neighborhood-communicator branch from 0ad4ee8 to 1f49b91 Compare August 16, 2024 15:21

MarcelKoch requested review from upsj and removed request for upsj August 27, 2024 12:05

MarcelKoch force-pushed the distributed-row-gatherer branch from 8697971 to 341e781 Compare October 7, 2024 13:06

MarcelKoch force-pushed the neighborhood-communicator branch from 1f49b91 to 4db050c Compare October 7, 2024 13:06

pratikvn reviewed Oct 7, 2024

View reviewed changes

core/distributed/matrix.cpp Show resolved Hide resolved

yhmtsai requested changes Oct 23, 2024

View reviewed changes

MarcelKoch force-pushed the neighborhood-communicator branch from 4db050c to 1ebe59f Compare October 23, 2024 13:32

MarcelKoch and others added 10 commits October 23, 2024 15:32

[dist-rg] adds distributed row-gatherer

6140d4b

[dist-rg] handle copy to host buffer

81d8b3a

[dist-rg] use mpi request instead of future

359c739

[dist-rg] adds distributed row-gatherer tests

f3323b1

[dist-mat] use row-gatherer

4a19949

[pgm] use row-gatherer from matrix

cf03975

[core] enable array conversion between ints

76c7efa

[dist-rg] review update:

e564047

- only allocate if necessary - synchronize correct executor Co-authored-by: Pratik Nayak <pratik.nayak@kit.edu>

fixup! [dist-mat] use row-gatherer

8d3bdcb

[dist-rg] review updates:

f77cb6c

- split tests into core and backend part - fix formatting - fix openmpi pre 4.1.x macro Co-authored-by: Pratik Nayak <pratik.nayak4@gmail.com> Co-authored-by: Yu-Hsiang M. Tsai <yhmtsai@gmail.com>

MarcelKoch force-pushed the distributed-row-gatherer branch from b2025a8 to f77cb6c Compare October 23, 2024 14:17

MarcelKoch requested a review from yhmtsai October 24, 2024 10:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds distributed row gatherer #1589

Adds distributed row gatherer #1589

MarcelKoch commented Apr 4, 2024 •

edited

Loading

MarcelKoch commented Apr 22, 2024

pratikvn commented Apr 26, 2024

pratikvn left a comment

pratikvn Jul 23, 2024

MarcelKoch Oct 23, 2024

pratikvn Jul 23, 2024

MarcelKoch Oct 23, 2024

pratikvn Jul 23, 2024

yhmtsai Oct 7, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 7, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 23, 2024

yhmtsai Oct 23, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 23, 2024

MarcelKoch Oct 23, 2024


		mutable array<char> send_workspace_;

		mutable MPI_Request req_listener_{MPI_REQUEST_NULL};

		req = coll_comm_->i_all_to_all_v(
		mpi_exec, send_ptr, type.get(), recv_ptr, type.get());

Adds distributed row gatherer #1589

Are you sure you want to change the base?

Adds distributed row gatherer #1589

Conversation

MarcelKoch commented Apr 4, 2024 • edited Loading

MarcelKoch commented Apr 22, 2024

pratikvn commented Apr 26, 2024

pratikvn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcelKoch commented Apr 4, 2024 •

edited

Loading