GHEX APIs

Main concepts

• domain_descriptor: computational domain (potentially more than one per rank) with a coordinate system and already defined receive halos (only);

  • member types:
    • domain_id_type;
    • coordinate_type;
    • device;
  • member functions:
    • domain_id();
    • ranges();
    • receive_halos();

• data_descriptor: just a wrapper on a single data field;

  • member types:
    • coordinate_type;
    • data_type;
  • member functions:
    • access operator [coordinate_type coords];

Generic interfaces

• communication_pattern: built on top of each domain, provides information on which internal data range should be sent to which other domain;

  • member types:
    • domain_id_type;
    • coordinate_type;
    • domain_descriptor_type (i.e.: cartesian, octahedral, ... . Concrete implementation for cartesian and octahedral grids may be provided by the library);

• communication_buffer: complete information to build the send / receive buffer; actual send / receive buffers will be allocated by the communication object accordingly;

  • member types:
    • data_type;
    • device;
  • private member functions:
    • pack();
    • unpack();

• communication_object:

  • private member functions:
    • pack();
    • unpack();
  • public member functions:
    • exchange(communication_buffer... bs): performs the asynchronous send / receive, waiting for the results and calling pack() and unpack(); allocation is done here, following timesteps reusing the allocated memory;

Mock user code

domain_decomposition();

my_domain_descriptor d{...};
// ghex::pattern<my_domain_descriptor::domain_id_type, my_domain_descriptor::coordinate_type, my_domain_descriptor> p;
auto p = ghex::make_pattern(d);

// Performs an all_to_all communication in order to find
// which domain owns the needed receive indices
ghex::init(p); // (may be extended to ghex::init(p_1, ..., p_n);

ghex::communication_object co;

my_data_descriptor_1 f_1;
...
my_data_descriptor_n f_n; // n = # of fields to be exchanged

// ghex::communication_buffer b_1{f_1, p}, ..., b_n{f_n, p};
auto b_1 = ghex:make_communication_buffer(f_1, p);
...
auto b_n = ghex:make_communication_buffer(f_n, p);

co.exchange(b_1, ..., b_n);