Simulation manager tools for tracking simulation runs, generated data, interfacing with clusters/HPCs

[sanjayankur31]

Hi folks,

What tools are you using to manage simulations when running parameter sweeps and so on, especially when interfacing with clusters that use torque/slurm and so on? Ideally, the tool should allow tracking of metadata related to each run, parameters, send the run to the cluster queue, monitor it, fetch the data and link it to the simulation run.

I know Sumatra does this, but I don't think it's up to date to run on new Pythons. What are you all using?

https://github.com/open-research/sumatra

[ChristophMetzner]

Hi Ankur, hi everyone,
in our group at the TU a PhD student developed pypet (https://github.com/SmokinCaterpillar/pypet) for parameter explorations. However, that was already before I came to Berlin 6 years ago and Robert, who developed it has long left the lab and it is not really maintained anymore. I personally have not used it but some of my students have.

[sanjayankur31]

I did have a look at pypet, it looks very good! But yeh, even though it does seem to work with current Python versions, it needs a bit of an update---it uses scoop, for example, that hasn't been maintained in a few years. Also, it seems to link to sumatra for the simulation management bit and sumatra, unfortunately, is not currently maintained. :(

I ran into a few others too. I'll post them as individual comments below.

[sanjayankur31]

https://github.com/CSNG-MFF/mozaik , and based on it: Arkheia: http://arkheia.org/about

[sanjayankur31]

https://github.com/task123/sim_db -> unmaintained

[edd26]

Hi Ankur, I have a solution that fits part of what you need. I used it for machine learning and thought about trying it for computational neuroscience, but I still need to find time to do it- it is called Hydra. As said, it doesn’t cover everything you need, but the advantage is that it is more general, so a larger community actively maintains it. You can use it for parameter sweeps and runs’ metadata tracking. What is left out is sending it to a cluster, monitoring, and fetching data, but maybe some other software could cover this. https://hydra.cc/ Best regards, Emil Dmitruk From: Ankur Sinha ***@***.***> Date: Thursday, 14 November 2024 at 09:43 To: OCNS/SoftwareWG ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [OCNS/SoftwareWG] Simulation manager tools for tracking simulation runs, generated data, interfacing with clusters/HPCs (Discussion #148) https://github.com/task123/sim_db<https://github.com/task123/sim_db> -> unmaintained — Reply to this email directly, view it on GitHub<#148 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEF6CHQHEVUUVEUMFDFOXCL2ARWFPAVCNFSM6AAAAABRUL5HCKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRVGIYTENY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sanjayankur31]

Thanks, that looks very interesting!

[sdiazpier]

We are developing the tool L2L which allows for large parameter space explorations on local devices and HPC systems. In its origins it used scoop and pypet. Although we still keep large inspiration from pypet we have moved forward with other strategies to orchestrate job deployment. We are about to make a new release of the tool in about 2 weeks with a new way to do task deployment which is more efficient and suitable for HPC systems where having many small files produced by the parameter space explorations is a problem. Here is our repository: https://github.com/Meta-optimization/L2L
At the JSC we also have JUBE, which we use for benchmarking and can be used to do brute force parameter explorations on HPC systems: http://apps.fz-juelich.de/jsc/jube/docu/index.html

[sanjayankur31]

Thanks @sdiazpier : I've been meaning to look at L2L for a while. Wasn't aware of JUBE---I'll check that out.

[sanjayankur31]

There's also snakemake. It's more for data analysis pipelines though, so something to be to analyse simulation generated data perhaps:

https://snakemake.readthedocs.io/en/stable/