Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mapping between ADIOS steps and openPMD iterations #949

Merged
merged 8 commits into from
Oct 25, 2022
Merged

Mapping between ADIOS steps and openPMD iterations #949

merged 8 commits into from
Oct 25, 2022

Commits on Oct 18, 2022

  1. Backend additions 

    1) New streaming status: RANDOM_ACCESS, for non-streaming situations
2) Variable attributes, to be written only if the backend has support
   for steps
    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    29e492b View commit details
    Browse the repository at this point in the history

  2. Writing changes: Write current step(s) to snapshot attribute 

    Only set snapshot attribute if Iteration is not yet written

For v-based iteration encoding, the snapshot attribute is already being
set before this PR. Just add a comment there.

Also add missing <cstdint> includes

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
    @franzpoeschel @ax3l
    franzpoeschel and ax3l committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    dee6768 View commit details
    Browse the repository at this point in the history

  3. Reading changes: Use snapshot attribute 

    This means that the snapshot attribute, if present, is used for
accessing iterations inside `series.readIterations()`. Fallback to the
old behavior (linear progression through iterations) if the attribute is
not found.

Variable-b. encoding: Allow several (equivalent) iterations per step

This means that a single step can be marked by /data/snapshot to
represent iterations 0,10,20,30 at the same time.
The underlying data is the same, but the API will treat it as 4 times a
different iteration with equivalent content.

Avoid const_cast by introducing a parsing state and use that when
re-parsing.

Skip repeated iterations that occur in Append mode
Before the explicit iteration-step mapping, these were not seen by
reading procedures at all. Now they are, so we skip the second instance.

Better error message when calling readIterations() too late

This commit includes some refactoring

1. Remove recursion of operator++(), this leads to constant memory usage
   rather than filling the stack at some point
2. Extract subroutines from operator++()
3. Steal some refactoring that solved some bugs on topic-read-leniently, so it stands
to reason that we should apply it here already
    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    695e82f View commit details
    Browse the repository at this point in the history

  4. Testing 

    In the tests, don't try to read the series with listSeries after already
having fully drained it

Combined test: append mode and weird iteration order

Deactivate troublesome Schema 2021 Append test
    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    81cfec2 View commit details
    Browse the repository at this point in the history

  5. Add -wd1011 flag to Icc workflow

    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    11cd90d View commit details
    Browse the repository at this point in the history

  6. Fix priority of JSON/envvar config for ADIOS2 schema

    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    b70af3e View commit details
    Browse the repository at this point in the history

  7. Preview support for Linear read mode without snapshot attribute 

    Currently only available for BP5 engine, will be generalized into Linear
read mode in openPMD#1291.
If the backend does not support the snapshot attribute, then iterate in
ascending order, skipping duplicate and non-linear iteration indices.
Not possible if the Series is parsed ahead of time.
    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    8dcf776 View commit details
    Browse the repository at this point in the history

  8. Test edge cases of snapshot attribute

    @franzpoeschel
    franzpoeschel committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    e491869 View commit details
    Browse the repository at this point in the history