Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for more executors #96

Open
Nick-Eagles opened this issue Aug 7, 2023 · 0 comments
Open

Add support for more executors #96

Nick-Eagles opened this issue Aug 7, 2023 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@Nick-Eagles
Copy link
Member

One specific request was AWS, but there are a huge number of other executors people could request in the future.

I think the real goal here is to find an elegant way to balance:

  • keeping the executor modular conceptually (executor-related configuration and execution details should be separate from those regarding hardware resources or unrelated details)
  • providing executor-specific details like this that provide sensible default configuration, thus reducing set-up difficulty for users

Ideally we'd have a way to, at runtime or install time, specify to SPEAQeasy an arbitrary executor supported by Nextflow, and have things mostly work out of the box. Then for the most popular and in-demand executors (say SLURM, SGE, JHPCE, and AWS with or without docker/singularity containers), we could spend additional effort maintaining executor-specific configuration (like this or this).

I imagine that such a system would work by having a sort of "template configuration" with core specifications supported by all executors (i.e. cpus, memory, and probably others), and at install time add executor-specific details that at minimum would be executor.name. This system would probably break backwards compatibility though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

2 participants