Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modified dataman doc to address latest changes #1757

Merged
merged 3 commits into from
Sep 20, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 31 additions & 21 deletions docs/user_guide/source/engines/dataman.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
***************
DataMan for WAN
***************
******************************************
DataMan for Wide Area Network Data Staging
******************************************

The DataMan engine is designed for data transfers over the wide area network. To use this engine, you can either specify it in your xml config file, with tag ``<engine type=DataMan>`` or set it in your application code:

Expand All @@ -21,23 +21,33 @@ On the reader side you need to do instead:
.. note::
The DataMan engine currently does not support data staging within a cluster.

The DataMan engine does not accept any parameters. However, users are allowed to specify the following transport parameters:

1. **Library**: the underlying network / socket library used for data transfer.

2. **IPAddress**: the IP address of the host where the writer application runs.

3. **Port**: the port on the writer host that will be used for data transfers.

4. **Timeout**: the timeout in seconds to wait for every send / receive.

============= ================= ================================================
**Key** **Value Format** **Default** and Examples
============= ================= ================================================
Library string **ZMQ**
IPAddress string **127.0.0.1**, 22.195.18.29
Port integer **12306**, 22000, 33000
Timeout integer **5**, 10, 30
============= ================= ================================================
The DataMan engine takes the following parameters:

1. ``IPAddress``: No default value. The IP address of the host where the writer application runs.
This parameter is compulsory in wide area network data staging.

2. ``Port``: Default **50001**. The port number on the writer host that will be used for data transfers.

3. ``Timeout``: Default **5**. Timeout in seconds to wait for every send / receive operation.
Packages not sent or received within this time are considered lost.

4. ``AlwaysProvideLatestTimestep``: Default **TRUE**.
AlwaysProvideLatestTimestep is a boolean parameter that affects what
of the available timesteps will be provided to the reader engine. If
AlwaysProvideLatestTimestep is **TRUE**, then if there are multiple
timesteps available to the reader, older timesteps will be skipped and
the reader will see only the newest available upon BeginStep.
This value is interpreted by only by the DataMan Reader engine.
If AlwaysProvideLatestTimestep is **FALSE**, then the reader engine
will be provided with the oldest step that has not been processed.

=============================== ================== ================================================
**Key** **Value Format** **Default** and Examples
=============================== ================== ================================================
IPAddress string **N/A**, 22.195.18.29
Port integer **50001**, 22000, 33000
Timeout integer **5**, 10, 30
AlwaysProvideLatestTimestep boolean **TRUE**, false
=============================== ================== ================================================


8 changes: 4 additions & 4 deletions examples/hello/datamanReader/helloDataManReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
#include <thread>
#include <vector>

int rank, size;
int mpiRank, mpiSize;

template <class T>
void PrintData(std::vector<T> &data, size_t step)
{
std::cout << "Rank: " << rank << " Step: " << step << " [";
std::cout << "Rank: " << mpiRank << " Step: " << step << " [";
for (size_t i = 0; i < data.size(); ++i)
{
std::cout << data[i] << " ";
Expand All @@ -33,8 +33,8 @@ int main(int argc, char *argv[])
{
// initialize MPI
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);

// initialize adios2
adios2::ADIOS adios(MPI_COMM_WORLD, adios2::DebugON);
Expand Down
14 changes: 7 additions & 7 deletions examples/hello/datamanWriter/helloDataManWriter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ adios2::Dims shape;
adios2::Dims start;
adios2::Dims count;

int rank, size;
int mpiRank, mpiSize;

template <class T>
void PrintData(std::vector<T> &data, const size_t step)
{
std::cout << "Rank: " << rank << " Step: " << step << " [";
std::cout << "Rank: " << mpiRank << " Step: " << step << " [";
for (const auto i : data)
{
std::cout << i << " ";
Expand All @@ -43,7 +43,7 @@ std::vector<T> GenerateData(const size_t step)
std::vector<T> myVec(datasize);
for (size_t i = 0; i < datasize; ++i)
{
myVec[i] = i + rank * 10000 + step;
myVec[i] = i + mpiRank * 10000 + step;
}
return myVec;
}
Expand All @@ -52,13 +52,13 @@ int main(int argc, char *argv[])
{
// initialize MPI
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);

// initialize data dimensions
count = {Nx, Ny};
start = {rank * Nx, 0};
shape = {size * Nx, Ny};
start = {mpiRank * Nx, 0};
shape = {mpiSize * Nx, Ny};

// initialize adios2
adios2::ADIOS adios(MPI_COMM_WORLD, adios2::DebugON);
Expand Down