add documentation for creating/using plugins

docs/user_guide/source/engines/engines.rst

@@ -24,5 +24,4 @@ Parameters are passed at:
 .. include:: dataman.rst
 .. include:: inline.rst
 .. include:: null.rst
-.. include:: plugin.rst
 .. include:: virtual_engines.rst

docs/user_guide/source/index.rst

@@ -26,6 +26,7 @@ Funded by the `Exascale Computing Project (ECP) <https://www.exascaleproject.org
    components/components
    engines/virtual_engines
    engines/engines
+   plugins/plugins
    api_full/api_full
    api_high/api_high
 

docs/user_guide/source/engines/plugin.rst → docs/user_guide/source/plugins/engine.rst

@@ -1,9 +1,4 @@
-*************
-Plugin Engine
-*************
-
-The ``Plugin`` engine enables the ability to load an engine located in a separate library.
-Your plugin class needs to inherit from the ``PluginEngineInterface`` class in the ``adios2/engine/plugin/PluginEngineInterface.h`` header.
+Your engine plugin class needs to inherit from the ``PluginEngineInterface`` class in the ``adios2/engine/plugin/PluginEngineInterface.h`` header.
 Depending on the type of engine you want to implement, you'll need to override a number of methods that are inherited from the ``adios2::core::Engine`` class.
 These are briefly described in the following table.
 More detailed documentation can be found in ``adios2/core/Engine.h``.
@@ -33,69 +28,26 @@ An example write engine is ``ExampleWritePlugin.h``, while an example read engin
 The writer is a simple file writing engine that creates a directory (called ``ExamplePlugin`` by default) and writes variable information to vars.txt and actual data to data.txt.
 The reader example reads the files output by the writer example.
 
-In addition to implementing the methods above, you'll need to implement ``EngineCreate()`` and ``EngineDestroy`` functions so the Plugin Engine can create/destroy the engine object.
+In addition to implementing the methods above, you'll need to implement ``EngineCreate()`` and ``EngineDestroy`` functions so ADIOS can create/destroy the engine object.
 Because of C++ name mangling, you'll need to use ``extern "C"``.
 Looking at ``ExampleWritePlugin.h``, this looks like:
 
 .. code-block:: c++
 
     extern "C" {
 
-    adios2::core::engine::ExampleWritePlugin *
+    adios2::plugin::ExampleWritePlugin *
     EngineCreate(adios2::core::IO &io, const std::string &name,
                  const adios2::Mode mode, adios2::helper::Comm comm)
     {
-        return new adios2::core::engine::ExampleWritePlugin(io, name, mode,
+        return new adios2::plugin::ExampleWritePlugin(io, name, mode,
                                                             comm.Duplicate());
     }
 
-    void EngineDestroy(adios2::core::engine::ExampleWritePlugin * obj)
+    void EngineDestroy(adios2::plugin::ExampleWritePlugin * obj)
     {
         delete obj;
     }
 
     }
 
-To build your plugin, your CMake should look something like:
-
-.. code-block:: cmake
-
-    find_package(ADIOS2 REQUIRED)
-    set(BUILD_SHARED_LIBS ON)
-    add_library(PluginEngineWrite
-      ExampleWritePlugin.cpp
-    )
-    target_link_libraries(PluginEngineWrite adios2::cxx11 adios2::core)
-
-When using the Plugin Engine, ADIOS will check for your plugin at the path specified in the ``ADIOS2_PLUGIN_PATH`` environment variable.
-If ``ADIOS2_PLUGIN_PATH`` is not set, and a path is not specified in the settings for the Plugin Engine (see below steps for using a plugin in your application), then the usual ``dlopen`` search is performed (see `dlopen man page <https://man7.org/linux/man-pages/man3/dlopen.3.html>`_).
-
-The following steps show how to use your engine plugin in your application.
-``examplePluginEngine_write.cpp`` and ``examplePluginEngine_read.cpp`` are an example of how to use the engine plugins described above.
-The key steps to use your plugin are:
-
-1. Set engine to ``Plugin``. i.e.:
-
-.. code-block:: c++
-
-   io.SetEngine("Plugin");
-
-2. Set ``PluginName`` (optional) and ``PluginLibrary`` (required) parameters.
-   If you don't set ``PluginName``, the Plugin Engine will give your plugin a default name of ``UserPlugin``.
-   In the write example, this looks like
-
-.. code-block:: c++
-
-   io.SetParameters({{"PluginName", "WritePlugin"}});
-   io.SetParameters({{"PluginLibrary", "PluginEngineWrite"}});
-   // also possible to use the path here instead of setting ADIOS2_PLUGIN_PATH
-   // io.SetParameters({{"PluginLibrary", "/path/to/libPluginEngineWrite.so"}})
-
-.. note::
-    You don't need to add the ``lib`` prefix or the shared library ending (e.g., ``.so``, ``.dll``, etc.).
-    ADIOS will add these when searching for your plugin library.
-    If you do at the prefix/suffix, ADIOS should still be able to find your plugin.
-
-
-At this point you can open the engine and use it as you would any other ADIOS engine.
-You also shouldn't need to make any changes to your CMake files for your application.

docs/user_guide/source/plugins/operator.rst

@@ -0,0 +1,34 @@
+Your operator plugin class needs to inherit from the ``PluginOperatorInterface`` class in the ``adios2/operator/plugin/PluginOperatorInterface.h`` header.
+There's three methods that you'll need to override from the ``adios2::core::Operator`` class, which are described below.
+
+========================= ===========================================================
+ **Method**                **Description**
+========================= ===========================================================
+``Operate()``              Performs the operation, e.g., compress data
+``InverseOperate()``       Performs the inverse operation, e.g., decompress data
+``IsDataTypeValid()``      Checks that a given data type can be processed
+========================= ===========================================================
+
+An example showing how to implement an operator plugin can be found at ``examples/plugins/operator/EncryptionOperator.h``.
+This operator uses `libsodium <https://doc.libsodium.org/>`_ for encrypting and decrypting data.
+
+In addition to implementing the methods above, you'll need to implement ``OperatorCreate()`` and ``OperatorDestroy`` functions so ADIOS can create/destroy the operator object.
+Because of C++ name mangling, you'll need to use ``extern "C"``.
+Looking at ``EncryptionOperator``, this looks like:
+
+.. code-block:: c++
+
+    extern "C" {
+
+    adios2::plugin::EncryptionOperator *
+    OperatorCreate(const adios2::Params &parameters)
+    {
+        return new adios2::plugin::EncryptionOperator(parameters);
+    }
+
+    void OperatorDestroy(adios2::plugin::EncryptionOperator * obj)
+    {
+        delete obj;
+    }
+
+    }

docs/user_guide/source/plugins/plugins.rst

@@ -0,0 +1,154 @@
+###########################
+Engine and Operator Plugins
+###########################
+
+ADIOS now has the ability for users to load their own engines and operators through the plugin interface.
+The basic steps for doing this are:
+
+1. Write your plugin class, which needs to inherit from the appropriate ``Plugin*Interface`` class.
+2. Build as a shared library and add the path to your shared library to the ``ADIOS2_PLUGIN_PATH`` environment variable.
+3. Start using your plugin in your application.
+
+*************************
+Writing Your Plugin Class
+*************************
+
+
+Engine Plugin
+-------------
+
+.. include:: engine.rst
+
+Operator Plugin
+---------------
+
+.. include:: operator.rst
+
+********************
+Build Shared Library
+********************
+
+To build your plugin, your CMake should look something like the following (using the plugin engine example described above):
+
+.. code-block:: cmake
+
+    find_package(ADIOS2 REQUIRED)
+    set(BUILD_SHARED_LIBS ON)
+    add_library(PluginEngineWrite
+      ExampleWritePlugin.cpp
+    )
+    target_link_libraries(PluginEngineWrite adios2::cxx11 adios2::core)
+
+When using the Plugin Engine, ADIOS will check for your plugin at the path specified in the ``ADIOS2_PLUGIN_PATH`` environment variable.
+If ``ADIOS2_PLUGIN_PATH`` is not set, and a path is not specified when loading your plugin (see below steps for using a plugin in your application), then the usual ``dlopen`` search is performed (see the `dlopen man page <https://man7.org/linux/man-pages/man3/dlopen.3.html>`_).
+
+.. note::
+    The ``ADIOS2_PLUGIN_PATH`` environment variable can contain multiple paths, which must be separated with a ``:``.
+
+***********************************
+Using Your Plugin in an Application
+***********************************
+
+For both types of plugins, loading is the same and there are two ways to do it.
+
+The first option is programmatically. Within your application, you need to call ``ADIOS::LoadPlugin()``
+
+.. code-block:: c++
+
+    adios2::ADIOS adios;
+    adios.LoadPlugin("WritePlugin", "PluginEngineWrite");
+
+Where "WritePlugin" is the name that ADIOS will use to keep track of the plugin, and "PluginEngineWrite" is the shared library name.
+
+The second option is using an ADIOS XML config file. If you'd like to load your plugins through an XML config file, the following shows an example XML:
+
+.. code-block:: xml
+
+    <adios-config>
+        <plugins>
+            <plugin name="WritePlugin" library="PluginEngineWrite"/>
+            <plugin name="ReadPlugin" library="PluginEngineRead"/>
+            <plugin name="OperatorPlugin" library="EncryptionOperator"/>
+            <parameter key="verbose" value="5"/>
+        </plugins>
+        ...
+    </adios-config>
+
+The parameter ``verbose`` is optional.
+Setting this turns on output related to finding and loading libraries, so it can be useful in debugging to ensure that your library is being loaded.
+
+.. note::
+    For either approach, you don't need to add the ``lib`` prefix or the shared library ending (e.g., ``.so``, ``.dll``, etc.).
+    ADIOS will add these when searching for your plugin library.
+    If you do add the prefix/suffix, ADIOS will still be able to find your plugin.
+    It's also possible to put the full path to the shared library here, instead of using ``ADIOS2_PLUGIN_PATH``.
+
+
+Engine Plugins
+--------------
+
+The examples ``examples/plugins/engine/examplePluginEngine_write.cpp`` and ``examples/plugins/engine/examplePluginEngine_read.cpp`` are an example of how to use the engine plugins described above.
+
+Set engine to ``Plugin`` and the name you gave the plugin. i.e.:
+
+.. code-block:: c++
+
+    io.SetEngine("Plugin", "WritePlugin");
+
+If your engine plugin requires any parameters, you can add them as you would for a built-in engine:
+
+.. code-block:: c++
+
+    adios2::Params params;
+    params["PluginKey0"] = "value0";
+    ...
+    io.SetParameters(params);
+
+At this point you can open the engine and use it as you would any other ADIOS engine.
+You also shouldn't need to make any changes to your CMake files for your application.
+
+If you use XML config files, you can set up your engine plugins like this:
+
+.. code-block:: xml
+
+    <io name="writer">
+        <engine type="plugin" plugin="WritePlugin">
+        <!-- any parameters needed for your plugin can be added here in the parameter tag -->
+        </engine>
+    </io>
+
+    <io name="reader">
+        <engine type="plugin" plugin="ReadPlugin">
+        <!-- any parameters needed for your plugin can be added here in the parameter tag -->
+        </engine>
+    </io>
+
+
+Operator Plugins
+----------------
+
+The examples ``examples/plugins/operator/examplePluginOperator_write.cpp`` and ``examples/plugins/engine/examplePluginOperator_read.cpp`` show an example of how to use the ``EncryptionOperator`` plugin described above.
+Using an operator plugin is the same as using a built-in operator, except you need to make sure you provide the ``PluginName`` parameter.
+For instance, using the ``EncryptionOperator`` as an example:
+
+.. code-block:: c++
+
+    adios2::Params params;
+    params["PluginName"] = "OperatorPlugin";
+    params["SecretKeyFile"] = "/path/to/secret-key";
+    var.AddOperation("plugin", params);
+
+When you create your own operator, any options the user needs to pass would be added to the ``adios2::Params`` object (instead of the "SecretKeyFile" param, which is specific to the ``EncryptionOperator``).
+
+If you use XML config files, here's how to set up your operator plugin for a variable:
+
+.. code-block:: xml
+
+    <io name="writer">
+         <variable name="data">
+            <operation type="plugin">
+                <parameter key="PluginName" value="OperatorPlugin"/>
+                <!-- any other parameters needed for your plugin can be added here in a parameter tag -->
+            </operation>
+        </variable>
+    </io>