Why python code in a standard ?

[mccoys]

It is a little confusing to me to see that some python code is included in a standard. IMO, a standard should define rules and conventions, but not an implementation.

As I understand the current python files, they seem to be some kind of template for future codes that wish to implement a translator from PICMI to their code. If this is true, then shouldn't it be an external tool, rather than a part of the standard ? See for instance the structre of OpenPMD

[jlvay]

We started with something very similar to the structure of OpenPMD, with a markdown file describing the standard. During the development of the standard, it became established that there would be at least a Python implementation of it. People then worked directly on the Python implementation and the markdown was trailing. One possibility could have been that a parser generates the Python implementation from the markdown or the other way around but nobody volunteered (had the time) to write such parser. So we opted to remove the markdown and have the standard essentially described in the Sphynx documentation, generated automatically from the Python PICMI implementation.
Any suggestion of improvement is welcome.

[RemiLehe]

Having worked on both openPMD and PICMI, I would like to add that the case of PICMI is a bit different:

• in openPMD, the standard describes how the HDF5 file is layed out ; it does not say anything about APIs, or anything language-specific.

• For PICMI, the standard is the Python API. So the standard is much closer to actual code, and we quickly realized that the Python implementation (with the associated docstrings) directly duplicated the information in the markdown file. As mentioned by @jlvay the standard is now automatically generated from the Python docstrings, using Sphinx, and can be found here: https://picmi-standard.github.io/standard/standard.html

@mccoys Does this make sense?

[mccoys]

I understand your points (although I need to think a bit more about it). I guess my question is now about how this python module would come into play into an existing code. The page about this question is still not ready it seems. For instance, imagine a code that uses python natively: why would it need that piece of python code ?

[mccoys]

Let me add also the following point. @RemiLehe you say that the standard is the python API. However, it is clearly stated that the API is not mandatory to use, which contradicts your point. A standard establishes conventions that are mandatory or optional, but if the whole standard is optional, then I don't see the goal. In the end, I think I don't understand how people are supposed to use it.

[jlvay]

For codes that do not have a Python interface, there would be a layer to translate the Python PICMI file to the native input format of the given code.
For codes with a Python interface, they could either translate the PICMI Python file to a Python file that is understood by the code or add PICMI classes that follow the PICMI standard and call the native functions under the hood. This is what we do with Warp for example. The Python PICMI classes that are provided in the PICMI repo give the base classes to be used by individual codes for their PICMI-aware implementation.

[jlvay]

The API is not mandatory to use in the sense that we cannot enforce it but if people want their input script to be portable between PICMI-aware code, then they have to follow the standard for that to work.

[mccoys]

I think the reason why I have a hard time understanding your points is because the philosophy of PICMI appears very different from that of Smilei.

In PICMI, the python code is really the main code itself: you start by creating objects like species, lasers, etc. then you add them to your simulation, and the simulation has a step method that basically tells it to run. I would say this is not really an input file per se, but the actual process of the simulation. The PIC code is just a library beneath.

In Smilei, the python code is a bunch of parameters that are read by a pure C++ implementation using python's C-API.

PICMI somewhat imposes to the user a deep knowledge of how python works, and this is specifically what we tried to avoid in Smilei. But anyway, this is a choice that you guys probably made in the beginning. I still maintain that those aspects make PICMI not a real standard, because it almost imposes dependency to the python module, and it does not only defines names and conventions, but also forces the code to handle a timeline controlled by python. For instance, what if the user calls sim.steps(1000) several times? How many codes can handle that? PICMI really looks like a high-level manager for PIC libraries, not a standard.

In the end, I fear that very few PIC codes will join this "standard" if that is the strategy.

[jlvay]

Actually, the PICMI Python interface has two main modes:
a) very similar to Smilei, the Python PICMI file defines all the parameters that are then converted to a file read by a code that does not used a Python interfaced (written in C, C++, Fortran, …),
b) Python is used as a frontend to the code and the PICMI script controls the simulation directly.

Of course, most codes do not use Python as a frontend and would use mode a), which is very similar to what Smilei does.

[ax3l]

Just a comment on nomenclature: an interface (API) can also be standardized/a standard, see e.g. the MPI standard - a message passing interface standard.

In PIConGPU, our input is currently included C++ header files and bash files for command line options, adjusted by the user. We experiment with PICMI by means of pre-processor and Python macro templatization, adding a input file API generator as in case a).