availableChunks open logic (HDF5: Early Chunk Read)

docs/source/details/mpi.rst

@@ -29,6 +29,7 @@ Functionality                Behavior           Description
 ``::makeConstant`` [3]_      *backend-specific* declare, write
 ``::storeChunk`` [1]_        independent        write
 ``::loadChunk``              independent        read
+``::availableChunks`` [4]_   collective         read, immediate result
 ============================ ================== ===========================
 
 .. [1] Individual backends, i.e. :ref:`parallel HDF5 <backends-hdf5>`, will only support independent operations if the default, non-collective (aka independent) behavior is kept.
@@ -42,6 +43,7 @@ Functionality                Behavior           Description
        Note that openPMD represents constant record components with attributes, thus inheriting this for ``::makeConstant``.
 
 .. [4] We usually open iterations delayed on first access. This first access is usually the ``flush()`` call after a ``storeChunk``/``loadChunk`` operation. If the first access is non-collective, an explicit, collective ``Iteration::open()`` can be used to have the files already open.
+Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.
 
 .. tip::
 

docs/source/usage/streaming.rst

@@ -25,6 +25,7 @@ The reading end of the streaming API is activated through use of ``Series::readI
 The returned object of type ``ReadIterations`` can be used in a C++11 range-based for loop to iterate over objects of type ``IndexedIteration``.
 This class extends the ``Iteration`` class with a field ``IndexedIteration::iterationIndex``, denoting this iteration's index.
 
+Iterations are implicitly opened by the Streaming API and ``Iteration::open()`` needs not be called explicitly.
 Users are encouraged to explicitly ``.close()`` the iteration after reading from it.
 Closing the iteration will flush all pending operations on that iteration.
 If an iteration is not closed until the beginning of the next iteration, it will be closed automatically.
@@ -42,6 +43,7 @@ The reading end of the streaming API is activated through use of ``Series.read_i
 The returned object of type ``ReadIterations`` can be used in a Python range-based for loop to iterate over objects of type ``IndexedIteration``.
 This class extends the ``Iteration`` class with a field ``IndexedIteration.iteration_index``, denoting this iteration's index.
 
+Iterations are implicitly opened by the Streaming API and ``Iteration.open()`` needs not be called explicitly.
 Users are encouraged to explicitly ``.close()`` the iteration after reading from it.
 Closing the iteration will flush all pending operations on that iteration.
 If an iteration is not closed until the beginning of the next iteration, it will be closed automatically.

include/openPMD/Iteration.hpp

@@ -118,6 +118,9 @@ class Iteration : public LegacyAttributable
      * operation is flush-ed. In parallel contexts where it is know that such a
      * first access needs to be run non-collectively, one can explicitly open
      * an iteration through this collective call.
+     * Also necessary when using defer_iteration_parsing.
+     * The Streaming API (i.e. Series::readIterations()) will call this method
+     * implicitly and users need not call it.
      *
      * @return Reference to iteration.
      */

include/openPMD/Series.hpp

@@ -385,6 +385,25 @@ class SeriesImpl : public AttributableImpl
     void readGorVBased( bool init = true );
     void readBase();
     std::string iterationFilename( uint64_t i );
+
+    enum class IterationOpened : bool
+    {
+        HasBeenOpened,
+        RemainsClosed
+    };
+    /*
+     * For use by flushFileBased, flushGorVBased
+     * Open an iteration, but only if necessary.
+     * Only open if the iteration is dirty and if it is not in deferred
+     * parse state.
+     */
+    IterationOpened openIterationIfDirty( uint64_t index, Iteration iteration );
+    /*
+     * Open an iteration. Ensures that the iteration's m_closed status
+     * is set properly and that any files pertaining to the iteration
+     * is opened.
+     * Does not create files when called in CREATE mode.
+     */
     void openIteration( uint64_t index, Iteration iteration );
 
     /**

include/openPMD/backend/Attributable.hpp

@@ -47,9 +47,10 @@ namespace traits
 } // traits
 class AbstractFilePosition;
 class AttributableImpl;
+class Iteration;
 namespace internal
 {
-class SeriesInternal;
+    class SeriesInternal;
 }
 
 class no_such_attribute_error : public std::runtime_error
@@ -235,6 +236,17 @@ class AttributableImpl
     internal::SeriesInternal const & retrieveSeries() const;
     internal::SeriesInternal & retrieveSeries();
 
+    /** Returns the corresponding Iteration
+     *
+     * Return the openPMD::iteration that this Attributable is contained in.
+     * This walks up the linked parents until it finds the Iteration object.
+     * Throws an error otherwise, e.g., for Series objects.
+     * @{
+     */
+    Iteration const & containingIteration() const;
+    Iteration & containingIteration();
+    /** @} */
+
     void seriesFlush( FlushLevel );
 
     void flushAttributes();

src/Iteration.cpp

@@ -156,13 +156,8 @@ Iteration::open()
     internal::SeriesInternal * s = &retrieveSeries();
     // figure out my iteration number
     auto begin = s->indexOf( *this );
-    auto end = begin;
-    ++end;
-    // set dirty, so Series::flush will open the file
-    this->dirty() = true;
-    s->flush_impl( begin, end, FlushLevel::UserFlush );
-    this->dirty() = false;
-
+    s->openIteration( begin->first, *this );
+    IOHandler()->flush();
     return *this;
 }