Software needed to be developed.

[DavidSagan]

Here is my idea for what software will be needed with the lattice standard. The boxes in the figure are:

• Simulation program: A simulation program.
• Interface layer: Code to interface between the structures defined by the program and the lattice structure defined by the Translator. The program maintainer(s) must create the interface layer since the code will be program specific. However, structure translation should (hopefully) be relatively straight forward.
• Translator: Package supplied with the lattice standard. The Translator has code to put lattice information into a "basic form". This includes expanding beam lines, evaluating expressions, etc.
• Reader / Writer: Package to read/write files of a standard format (JSON, YAML, etc). Packages to do this are widely available.
• Lattice File: A lattice file in a standard format such as JSON, YAML, etc.

Note: The Translator could be interfaced to multiple Reader/Writers so that multiple languages can be handled at once. The structure that the Translator passes to the Interface layer will be language independent so that the interface layer does not
have to deal with what language is being used directly (other than telling the Translator what language to use).

[DavidSagan]

[ax3l]

That structure makes sense to me.

For the reader / writer level, I would suggest we use something that can autocratically serialize/deserialize and validate dictionaries/lists/structs, e.g., pydantic:
https://github.com/ax3l/pydantic-mad (playground/demo project)

The translator on top of the reader / writer ([de]serializer) is probably one package.

[ax3l]

@ChristopherMayes commented:

We use Pydantic extensively in LUME project. For example, Genesis4 lattice elements: https://github.com/slaclab/lume-genesis/blob/master/genesis/version4/input/_lattice.py

[ax3l]

XSuite uses a package called Xdeps underneath their lattice parsing: https://github.com/xsuite/Xdeps

[rdemaria]

XSuite uses a package called Xdeps underneath their lattice parsing: https://github.com/xsuite/Xdeps

Xdeps implements several functionalities. The main is one to allow controlling element attributes by changing variables, inspecting and manipulating these relations and code generation. It is completely general and can control data in any valid Python container without modifications. Xdeps contains code to translate MAD-X expressions into xdeps framework and other MAD-X syntax parsing functions. As well, optimizers and tables used in xtrack and other components of xsuite.

[DavidSagan]

Needed will be translators for C, C++, Fortran, Python, and Julia. Maintenance is an issue and a possible headache. One thought is to write a C based package to handle the bookkeeping and then interface that to the other languages.

[ax3l]

An alternative is to expose C bindings and write in another language (e.g., C++).

But my feeling is we should first write a quick version in a scripting language like Python, especially since we want to exploit again the capabilities of pydantic (as done in LUME, xopt, etc.).

A quick implementation in any other language could be: use the scripts to prepare and validate files, then in the language of choice just read the format that is supported (e.g., there are many YAML/JSON/TOML readers available to maintainers). Not every language needs a validator for instance, that part can be done in a single, scripted implementation (as we do in openPMD-validator).

[ldeniau]

Nice to see that you converge to my proposal made during the first meeting (30.11.2024) :-)

I fully agree that a tool/library should be provided with the standard, written in C (at least exposed through a C API), in order to spread its use.

The parts I was proposing:

• Database / Object model: allow to store and retrieve information efficiently (sequences, elements, etc), evaluate expressions, handle hierarchy of classes (a-la-MAD-X).
• Parsers / Translators: read and write various formats to feed the Database.
• Validator: ensure consistency of attributes in elements, elements in sequences, sequence constructions, compliance with the standard, etc...
  External tools using this library would only need a C API to use components of these parts and build their own representation. If the database / object model is fast enough for queries, it may save the tool from building its own representation.

[DavidSagan]

There must have been some miscommunication here. This scheme was my thought all along and basically mirrors what was done for Accelerator Markup Language (AML).

[rdemaria]

The strategy we adopted in Xsuite was to have a basic object model in C (relatively simple with limited dynamic features, movable in memory, https://github.com/xsuite/xobjects) and leave the expressions for the host language/framework (in our case Python) to seamlessly meld with the goodies of the host ecosystem.

[DavidSagan]

This works well when using Python or Julia but unfortunately does not work well for programs based on C, C++, Fortran.