CalcJobs executing python code on a different python interpreter

[addman2]

Current situation

AiiDA contains two main tools how to execute a calculation. One is calcfunction and the other one is CalcJob. The first mentioned is good for small operation written in python while the other one more suited for large scale calcutions provided by external codes, also having interesting features such as ExitCodes, help string, automatic input validations, dedicated work directory, etc ... .

Problem

There are python packages such as PySCF [1], that can execute large-scale calculation, however, they does not work in a classical way, where one produces a an inputfiles that are crunched by some binaries on HPC machine producing some output files. Whereas in PySCF you have a set of python functions which you can orchestrate as you want.

So,

imagine you want to make a AiiDA plugin with pre-ochestrated calculation workflows. I can use calcfunction but I will loose all the nifty infrastructure of CalcJob and it will run on my workstation. And it is hard to use CalcJob because it assumes you are execution an external code.

Currently, I have three projects that are struggling because of this issue

Proposed solution

Following disussion on aiida-users mailing list there are a few suggestions

One obvious solution is to have all python functions within the library as a calcfuntion, and then wrap them into WorkChainNode. But still they will be executed on your workstation and you will not have dedicated work directory. Also It would require to have each and every inputs/outputs as DataNodes.

Another interesting solution is to use aiida dynamical workflows [2]. Solving most of the issues, but you have to publish your workflows not as a plugin but as AiiDA data nodes.

A solution proposed by me is following. Imagine you have a special CalcJob that runs a python script on a different python interpreter in a dedicated work directory on a HPC machine (or any machine). You have basically benefits from both worlds. I made a small sketch [3] to illustrate my approach (please, look at aiida_python/tests/classme.py). One creates a new CalcJob class that inherates CalcJobPython. You create inputs, outputs, ExitCodes a it would be a normal CalcJob but when it is executed you it will run a special method of this class called run_python. Code in this case would be some python interpreter on some machine.

I can imagine this could be a nice part of aiida-core, but if the developer would not think the same I will make a full plugin out of aiida-python sketch.

If you have any other solution please tell me.

[1] https://github.com/pyscf/pyscf
[2] https://github.com/microsoft/aiida-dynamic-workflows
[3] https://github.com/addman2/aiida_python

[ltalirz]

Hi @addman2, thanks for your sharing your thoughts!
I've migrated the issue to the discussion section since this a somewhat larger topic.

You are certainly correct that it would be great to improve support for python-based simulation codes (it's one of the many topics on the roadmap, which also mentions the somewhat related pyOS and mincePy projects by @muhrin).

Solving most of the issues, but you have to publish your workflows not as a plugin but as AiiDA data nodes.

Can you elaborate a bit on that?
It is true that aiida-dynamic-workflows stores the python code in AiiDA data nodes, but I don't think this would stop you from publishing your pyscf aiida plugin as a python package.

It is true that this is somewhat novel territory for AiiDA, so ideas are welcome, and others may want to chime in as well.

P.S. I currently don't have time but I'll have a look at your package when I find it.

[addman2]

Dear @ltalirz,

thank you for your answer. I work with AiiDA for a couple of months and I don't think I am an expert. I write my own plugins that are related to my project and my work, but never have done a big project so far. This might made me a little bit conservative about the workflows I do in AiiDA. That's why I was trying to avoid using aiida-dynamic-workflows. But you are right, aiida-dynamic-workflows can do the job as well. I have some deadlines by which I have to finish some plugins, I think I will polish a little bit the aiida-python plugin a try them both, figuring out which one of them suits me better.

aiida-python plugin is only a sketch, but by implementing ExitCodes and rethinking the way how data are serialized between the interpreters is ready to use.

Thank again

Otto from SISSA Trieste