From d95377829f9564384dfe5b913f045656d095af07 Mon Sep 17 00:00:00 2001 From: NAThompson Date: Wed, 26 Jun 2019 15:08:47 -0400 Subject: [PATCH] Additional documentation improvements. --- docs/user_guide/source/installation/cpp_c.rst | 30 ++++--- .../source/installation/examples.rst | 12 +-- .../source/installation/fortran.rst | 14 ++-- .../source/installation/hpc_systems.rst | 30 ++++--- .../user_guide/source/installation/python.rst | 42 +++++----- .../source/introduction/adaptable_io.rst | 25 +++--- .../source/introduction/introduction.rst | 2 - .../source/introduction/nutshell.rst | 84 ++++++++----------- docs/user_guide/source/using/using.rst | 11 ++- 9 files changed, 116 insertions(+), 134 deletions(-) diff --git a/docs/user_guide/source/installation/cpp_c.rst b/docs/user_guide/source/installation/cpp_c.rst index 820e3320a6..39756798f4 100644 --- a/docs/user_guide/source/installation/cpp_c.rst +++ b/docs/user_guide/source/installation/cpp_c.rst @@ -2,26 +2,24 @@ Installing the ADIOS2 library and the C++ and C bindings ******************************************************** -ADIOS2 by default will build and install the C++11 ``libadios2`` library and the C++11, C bindings. +By default, ADIOS2 will build the C++11 ``libadios2`` library and the C and C++ bindings. -1. **Minimum requirements:** +1. **Minimum requirements:** - * C++ compiler supporting C++11 - * If MPI is enabled, a MPI C (not MPI C++ bindings) implementation must be installed. + * A C++11 compliant compiler + * An MPI C implementation on the syspath, or in a location identifiable by CMake. -2. **Linking** ``make install`` will generate the required header and libraries to link applications with ADIOS2 under the installation directory (from ``CMAKE_INSTALL_PREFIX``): +2. **Linking** ``make install`` will copy the required headers and libraries into the directory specified by ``CMAKE_INSTALL_PREFIX``: + + * Libraries: - * Libraries: - - ``lib/libadios2.*`` C++11 and C bindings - - * Headers: - + + * Headers: + - ``include/adios2.h`` C++11 ``namespace adios2`` - ``include/adios2_c.h`` C prefix ``adios2_`` - - * Config file: run this command to get installation info - - - ``bin/adios2-config`` - - + + * Config file: run this command to get installation info + + - ``bin/adios2-config`` diff --git a/docs/user_guide/source/installation/examples.rst b/docs/user_guide/source/installation/examples.rst index 7445da457f..10bbb29f69 100644 --- a/docs/user_guide/source/installation/examples.rst +++ b/docs/user_guide/source/installation/examples.rst @@ -2,14 +2,14 @@ Running Examples **************** -The best way to have a grasp for ADIOS2 usage is to follow the examples under ADIOS2/examples. They are self-describing so the user can modify these for their own applications. +ADIOS2 is best learned by `examples `_. -An incremental way to look at examples: +A few very basic examples are described below: ================================ ========================================================================================================================== - Directory Description + Directory Description ================================ ========================================================================================================================== -``ADIOS2/examples/hello`` very basic "hello world" style examples for Writing and Reading. -``ADIOS2/examples/heatTransfer`` Poisson 2D solver for transient temperature in Fourier's heat transfer. Outputs ``bp.dir`` using ADIOS2, or HDF5. -``ADIOS2/examples/basics`` covers different Variable usage cases classified by the nature of their dimensions. +``ADIOS2/examples/hello`` very basic "hello world"-style examples for reading and writing `.bp` files. +``ADIOS2/examples/heatTransfer`` 2D Poisson solver for transients in Fourier's modeo of heat transfer. Outputs ``bp.dir`` or HDF5. +``ADIOS2/examples/basics`` covers different ``Variable`` use cases classified by the dimension. ================================ ========================================================================================================================== diff --git a/docs/user_guide/source/installation/fortran.rst b/docs/user_guide/source/installation/fortran.rst index 1368e889d1..eca6da5a2c 100644 --- a/docs/user_guide/source/installation/fortran.rst +++ b/docs/user_guide/source/installation/fortran.rst @@ -2,20 +2,18 @@ Enabling the Fortran bindings ***************************** -To enable the Fortran bindings in ADIOS2 follow these guidelines: +1. **Minimum requirements:** -1. **Minimum requirements:** + * A Fortran 90 compliant compiler + * A Fortran MPI implementation - * Compiler supporting Fortran 90 or more recent standards - * If MPI is enabled a MPI Fortran implementation must be installed. - -2. **Linking the Fortran bindings:** ``make install`` will generate under the installation directory (from ``CMAKE_INSTALL_PREFIX``) the required library and modules to link ADIOS2 with Fortran applications: +2. **Linking the Fortran bindings:** ``make install`` will copy the required library and modules into the directory specified by ``CMAKE_INSTALL_PREFIX`` * Library (note that ``libadios2`` must also be linked) - ``lib/libadios2_f.*`` - ``lib/libadios2.*`` - - * Modules + + * Modules - ``include/adios2/fortran/*.mod`` 3. **Module adios2:** only module required to be used in an application ``use adios`` diff --git a/docs/user_guide/source/installation/hpc_systems.rst b/docs/user_guide/source/installation/hpc_systems.rst index 67616f9516..1f0934d203 100644 --- a/docs/user_guide/source/installation/hpc_systems.rst +++ b/docs/user_guide/source/installation/hpc_systems.rst @@ -1,25 +1,29 @@ -**************************************************** -Building on High Performance Computing, HPC, Systems -**************************************************** +.. _HPCBuild: -Please read this section before building ADIOS2 with CMake on a Supercomputer: +*********************** +Building on HPC Systems +*********************** -#. **Modules with minimum requirements:** Make sure the corresponding "module" dependencies are loaded and that minimum requirements are satisfied. Load the latest cmake module as many systems have older cmake versions as default. The same with your build environment, make sure compilers support C++11. This includes gcc >= 4.8.1, Intel >= 15, PGI >= 15, etc. +#. **Modules:** Make sure all "module" dependencies are loaded and that minimum requirements are satisfied. + Load the latest CMake module as many HPC systems default to an outdated version. + Build with a C++11-compliant compiler, such as gcc >= 4.8.1, Intel >= 15, and PGI >= 15. -#. **Optimization flags:** For actual performance tests and operations, it is recommended to set ``-DCMAKE_BUILD_TYPE=Release`` explicitly so CMake can turn on appropriate optimization flags. - -#. **Static/Dynamic build:** On most Cray systems, like `Titan `_, the default library link behavior is "static", thus CMake builds ADIOS2 statically (``libadios2.a``) by default. Read the system's documentation to enable dynamic compilation, usually by setting an environment variable such as ``CRAYPE_LINK_TYPE=dynamic``. Click `here `_ for a full configurable script example on OLCF systems. +#. **Static/Dynamic build:** On Cray systems such as `Titan `_, + the default behavior is static linkage, thus CMake builds ADIOS2 creates the static library ``libadios2.a`` by default. + Read the system documentation to enable dynamic compilation, usually by setting an environment variable such as ``CRAYPE_LINK_TYPE=dynamic``. + Click `here `_ for a fully configurable script example on OLCF systems. #. **Big Endian and 32-bit systems:** ADIOS2 hasn't been tested on big endian or 32-bit systems. Please be aware before attempting to run. -#. **System guidelines and policies:** Follow your system guidelines and policies for launching jobs, executing MPI communication, memory allocation, and I/O storage before attempting any type of unconventional usage of ADIOS2. - -#. **CMake minimum version:** The ADIOS2 build system requires a minimum CMake version of 3.6.0, however, if using IBM XL, Cray, or PGI compilers, 3.9.0 or newer is required. ADIOS2 is using CMake's language level abstraction to instruct the compiler to build with C++11 support. This feature was available in CMake all the way back to 3.3, but with the feature only supporting GCC, Clang, and MSVC compilers. Langiuage level abstraction support was added for Intel compilers in CMake 3.6.0 and for all other compilers supproted by CMake (including XL, Cray, and PGI) in 3.9.0. +#. **CMake minimum version:** The ADIOS2 build system requires a minimum CMake version of 3.6.0. However, IBM XL, Cray, and PGI compilers require version 3.9.0 or newer. -#. **PGI compilers and C++11 support:** Recent versions of the PGI compiler >= v15 support the C++11 language standard, however they rely on the set of C++ standard library headers supplied by GCC. On most systems, the default configuration at the time compiler is instaled is to use the headers supplied by the system gcc in /usr. On many systems (Titan at OLCF, for example) even though the PGI compiler supports C++11 the configured GCC and it's headers do not (4.3.x on Cray Linux Environment v5 systems like Titan). To configure the PGI compiler to use a newer GCC, you will need to create a configuration file in your home directory that overrides the PGI compiler's default configuration. On Titan, the following steps will re-configure the PGI compiler to use GCC 6.3.0 provided by a module: +#. **PGI compilers and C++11 support:** Version 15 of the PGI compiler is C++11 compliant. + However it relies on the C++ standard library headers supplied by the system version of GCC, which may or may support all the C++11 features used in ADIOS2. + On many systems (Titan at OLCF, for example) even though the PGI compiler supports C++11, the configured GCC and its headers do not (4.3.x on Cray Linux Environment, and v5 systems like Titan). + To configure the PGI compiler to use a newer GCC, you must create a configuration file in your home directory that overrides the PGI compiler's default configuration. + On Titan, the following steps will re-configure the PGI compiler to use GCC 6.3.0 provided by a module: .. code-block:: bash $ module load gcc/6.3.0 $ makelocalrc $(dirname $(which pgc++)) -gcc $(which gcc) -gpp $(which g++) -g77 $(which gfortran) -o -net 1>${HOME}/.mypgirc 2>/dev/null - diff --git a/docs/user_guide/source/installation/python.rst b/docs/user_guide/source/installation/python.rst index f80c9ffd85..19b69d0276 100644 --- a/docs/user_guide/source/installation/python.rst +++ b/docs/user_guide/source/installation/python.rst @@ -4,32 +4,32 @@ Enabling the Python bindings To enable the Python bindings in ADIOS2, based on `PyBind11 `_, make sure to follow these guidelines: -1. **Dynamic library support:** The Python bindings in ADIOS2 require that your platform supports shared libraries, which for Cray systems (and some BlueGene systems) can require additional steps. See the previous section on :ref:`Building on High Performance Computing, HPC, Systems`. Note that you can still build the Python bindings with a static libadios2.a library, as long as your platform supports it. - -2. **Minimum requirements:** +- **Minimum requirements:** * Python 2.7 and above. - * Corresponding NumPy for the Python version used for the build - * If MPI is enabled, a compatible version of mpi4py needs to be installed. - * If Python versions 2.7 and 3 are installed in the system, CMake will autodetect Python 3 requirements as the default option. A Python executable can be passed explicitly to CMake builds using ``-DPYTHON_EXECUTABLE=...`` + * `numpy` + * `mpi4py` + +- **Running:** If CMake enables Python compilation, an ``adios2.so`` library containing the Python module is generated in the build directory under ``lib/pythonX.X/site-packages/`` + + * make sure your ``PYTHONPATH`` environment variable contains the path to ``adios2.so``. + + * make sure the Python interpreter is compatible with the version used for compilation via ``python --version``. + + * Run the Python tests with ``ctest -R Python`` -3. **Running:** if CMake enables Python compilation, an ``adios2.so`` library containing the Python modules is generated in the build directory under ``lib/pythonX.X/site-packages/`` + * Run `helloBPWriter.py `_ and `helloBPTimeWriter.py `_ - * make sure your ``PYTHONPATH`` environment variable contains the path to the ``adios2.so`` module. - - * make sure the Python interpreter is compatible with the version used for compilation. - - * try running Python tests with: ``ctest -R Python`` - - * try running the `helloBPWriter.py `_ example under ADIOS2/examples/hello/bpWriter, and `helloBPTimeWriter.py `_ under ADIOS2/examples/hello/bpTimeWriter + * In Python 2.7: .. code-block:: bash - - Python 2.7: - $ mpirun -n 4 python helloBPWriter.py + + $ mpirun -n 4 python helloBPWriter.py $ python helloBPWriter.py - - Python 3: - $ mpirun -n 4 python3 helloBPWriter.py + + * In Python 3: + + .. code-block:: bash + + $ mpirun -n 4 python3 helloBPWriter.py $ python3 helloBPWriter.py - diff --git a/docs/user_guide/source/introduction/adaptable_io.rst b/docs/user_guide/source/introduction/adaptable_io.rst index 66dcbf9f66..dcf656728c 100644 --- a/docs/user_guide/source/introduction/adaptable_io.rst +++ b/docs/user_guide/source/introduction/adaptable_io.rst @@ -2,22 +2,23 @@ Adaptable IO beyond files in Scientific Data Lifecycles ******************************************************* -Exascale computing, Big Data, Internet of Things (IoT), Burst Buffers, High Bandwidth Memory (HBM), Remote Direct Memory Access (RDMA)...all these novel terms and technologies have one thing in common: Data Management at large scales has become more relevant when making informed decisions. The goal of ADIOS2 is to provide an **adaptable**, **scalable**, and **unified** framework to aid scientific applications in their data transfer needs going beyond file input/output (IO) storage. As shown in the figure, there are many data management challenges in the scientific data life cycle beyond file I/O that ADIOS2 aspires to address +Performant and usable tools for data management at scale are essential in an era where scientific breakthroughs are collaborative, multidisciplinary, and computational. +ADIOS2 is an *adaptable*, *scalable*, and *unified* framework to aid scientific applications when data transfer volumes exceed the capabilities of traditional file I/O. -.. image:: http://i65.tinypic.com/2h5k38w.png : alt: my-picture1 +.. image:: http://i65.tinypic.com/2h5k38w.png : alt: my-picture1 -In that regard, we are attempting to create a framework to provide: +ADIOS2 provides -* Custom application management of massive data sets from generation, analysis, movement, to short-term and long-term storage. - -* Self-describing data in binary-packed (BP) format for quick information extraction +* Custom application management of massive data sets, starting from generation, analysis, and movement, as well as short-term and long-term storage. -* Ability to separate and extract relevant information from large data sets for better decision making - -* Ability to make near real-time decisions based on in-transit or in-situ analytics +* Self-describing data in binary-packed (`.bp`) format for rapid metadata extraction -* Expand to other transport mechanisms with minimal overhead to the user: Wide-Area-Network (WAN), Remote Direct Memory Access (ibverbs, NVLink, etc.), Shared-Memory - -* Exploit new memory hierarchy and scalability in novel hardware architectures: HBM, Burst Buffers, many-cores, etc. +* An ability to separate and extract relevant information from large data sets + +* The capability to make real-time decisions based on in-transit or in-situ analytics + +* The ability to expand to other transport mechanisms such wide area networks, remote direct memory access, and shared memory, with minimal overhead + +* The ability to utilize the full capabilities of emergent hardware technologies, such as high-bandwidth memory and burst buffers diff --git a/docs/user_guide/source/introduction/introduction.rst b/docs/user_guide/source/introduction/introduction.rst index 478835d18e..b1c9b21c72 100644 --- a/docs/user_guide/source/introduction/introduction.rst +++ b/docs/user_guide/source/introduction/introduction.rst @@ -2,8 +2,6 @@ Introduction ############ -You can find ADIOS2 source code on `github`_ - .. _`github`: https://github.com/ornladios/ADIOS2 .. include:: nutshell.rst diff --git a/docs/user_guide/source/introduction/nutshell.rst b/docs/user_guide/source/introduction/nutshell.rst index 37d880aa9d..31456a282c 100644 --- a/docs/user_guide/source/introduction/nutshell.rst +++ b/docs/user_guide/source/introduction/nutshell.rst @@ -1,77 +1,61 @@ -******************** -ADIOS2 in a Nutshell -******************** - -**ADIOS2** is the latest implementation of the `ADaptable Input Output System, ADIOS `_. -This brand new architecture was designed to continue supporting the performance -legacy of ADIOS, and extend its current capabilities to address current and -future input/output (IO) challenges in the scientific data lifecycle through -effective research and development (R&D) activities. - -ADIOS2 infrastructure is developed as a multi-institutional collaboration -between: - - * `Oak Ridge National Laboratory `_ - * `Kitware Inc. `_ - * `Lawrence Berkeley National Laboratory `_ - * `Georgia Institute of Technology `_ +**ADIOS2** is the latest implementation of the `Adaptable Input Output System `_. +This brand new architecture continues the performance +legacy of ADIOS1, and extends its capabilities to address the extreme challenges of scientific data IO. + +The ADIOS2 infrastructure is developed as a multi-institutional collaboration +between + + * `Oak Ridge National Laboratory `_ + * `Kitware Inc. `_ + * `Lawrence Berkeley National Laboratory `_ + * `Georgia Institute of Technology `_ * `Rutgers University `_ -The key aspects of the ADIOS2 infrastructure include: - +The key aspects ADIOS2 are + #. **Modular architecture:** ADIOS2 takes advantage of the major features - added in C++11. The architecture offers a balanced combination of runtime - Object-Oriented and static Template components to achieve the right level of - granularity to reuse components and cover a broad range of IO applications. + of C++11. The architecture utilizes a balanced combination of runtime + polymorphism and template metaprogramming to expose intuitive abstractions for a broad range of IO applications. + - -#. **Community:** by creating a set of coding standards, GitHub collaboration - workflows and the proper level of documentation: Doxygen API, code comments, - git issues and commits, ADIOS2 eases the collaboration aspects for community - engaging. +#. **Community:** By maintaining coding standards, collaborative + workflows, and understandable documentation, ADIOS2 lowers the barriers to entry for scientists to meaningfully interact with the code. - -#. **Sustainability:** automatic nightly build, testing and continuous - integration to ensure the quality of the contributions in our software. Please report any issue on github: https://github.com/ornladios/ADIOS2/issues +#. **Sustainability:** Continuous integration and unit testing ensure that ADIOS2 evolves responsibly. + Bug reports are triaged and fixed in a timely manner and can be reported on `Github `_. -#. **Language Support:** ADIOS2 is written in full taking advantage of the - C++11 major standard update. In addition, it started providing bindings for - seamless support in Python, C, Fortran and Matlab. +#. **Language Support:** In addition to the native C++, bindings for Python, C, Fortran and Matlab are provided. -#. **Commitment:** ADIOS2 is committed to the community of researchers, users - and developers to release a new version every 6 months. -**ADIOS2 is funded by the Department of Energy, as part of the Exascale Computing Program.** +#. **Commitment:** ADIOS2 is committed to the research community, releasing a new version every 6 months. + +*ADIOS2 is funded by the Department of Energy as part of the* `Exascale Computing Program `_. ************************ What ADIOS2 is and isn't ************************ -**ADIOS2 is:**: +**ADIOS2 is:** -1. **A Unified I/O Performance Framework**: using the same abstraction API ADIOS2 would Transport and Transform groups of self-describing data variables and attributes across different media (file, wide-area-network, in-memory staging, etc.) with performance an ease of use as the main goals. +- **A Unified I/O Performance Framework**: using the same abstraction API ADIOS2 would Transport and Transform groups of self-describing data variables and attributes across different media (file, wide-area-network, in-memory staging, etc.) with performance an ease of use as the main goals. -2. **MPI-based**: out-of-box MPI-based, non-MPI is optional at build time. +- **MPI-based**: out-of-box MPI-based, non-MPI is optional at build time. -3. **Group-based**: adios2 favors a deferred/prefetch/grouped variables transport mode by default. Sync mode, one variable at a time, is the special case. +- **Group-based**: ADIOS2 favors a deferred/prefetch/grouped variables transport mode by default. Sync mode, one variable at a time, is the special case. -4. **Step-based**: to resemble actual production of data in "steps" of variable groups, for either streaming or random-access (file) media +- **Step-based**: to resemble actual production of data in "steps" of variable groups, for either streaming or random-access (file) media -5. **Free and open-source** +- **Free and open-source**: ADIOS2 is permissibly licensed under the OSI-approved Apache 2 license. **ADIOS2 is not**: -1. **A File-only I/O library** - -2. **MPI-only** - -3. **A Database** +- **A file-only I/O library**: Multiple transport protocols are supported. -4. **A Data Hierarchy Model**: they build on top of the adios2 library according to the application +- **MPI-only** -5. **A Memory Manager Library**: we don't own or manage the application's memory +- **A Hierarchical Model**: Data hierarchies can be built on top of the ADIOS2 according to the application, but ADIOS2 sits a layer of abstraction beneath this. -6. **Difficult** +- **A Memory Manager Library**: we don't own or manage the application's memory diff --git a/docs/user_guide/source/using/using.rst b/docs/user_guide/source/using/using.rst index a5c9619cad..8d93635264 100644 --- a/docs/user_guide/source/using/using.rst +++ b/docs/user_guide/source/using/using.rst @@ -6,7 +6,7 @@ Using ADIOS in your own projects From CMake ********** -ADIOS exports a CMake package configuration file that allows it's targets to be directly imported into another CMake project via CMake's ``find_package`` command: +ADIOS exports a CMake package configuration file that allows its targets to be directly imported into another CMake project via the ``find_package`` command: .. code-block:: cmake @@ -16,16 +16,16 @@ ADIOS exports a CMake package configuration file that allows it's targets to be find_package(MPI REQUIRED) find_package(ADIOS2 REQUIRED) #... - add_library(my_library src1.cxx src2.cxx...) - target_link_libraries(my_library PRIVATE adios2::adios2 MPI::MPI_C ...) + add_library(my_library src1.cxx src2.cxx) + target_link_libraries(my_library PRIVATE adios2::adios2 MPI::MPI_C) -When configuring your project you can then set the ``ADIOS2_ROOT`` or ``ADIOS2_DIR`` environment variables to the install prefix for adios. You can also point it to an ADIOS build directory, which may be easier when using development versions of ADIOS than you may not want installed system-wide +When configuring your project you can then set the ``ADIOS2_ROOT`` or ``ADIOS2_DIR`` environment variables to the install prefix of ADIOS2. **************************** From non-CMake build systems **************************** -If you're not using CMake then you can manually get the necessary compile and link flags for your project using the ``adios2-config`` tool from the installation folder: +If you're not using CMake then you can manually get the necessary compile and link flags for your project using ``adios2-config``: .. code-block:: bash @@ -35,4 +35,3 @@ If you're not using CMake then you can manually get the necessary compile and li $ /path/to/install-prefix/bin/adios2-config --cxxlibs ADIOS2_DIR: /path/to/install-prefix -Wl,-rpath,/path/to/install-prefix/lib:/opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib /path/to/install-prefix/lib/libadios2.so.2.3.1 -pthread -Wl,-rpath -Wl,/opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib -Wl,--enable-new-dtags -pthread /opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib/libmpi.so -Wl,-rpath-link,/path/to/install-prefix/lib -