Integration with a cloud HPC meta-scheduler(CloudyCluster)

[cbmckni]

Hey Henrik,

I work as a developer for CloudyCluster(http://gcp.cloudycluster.com/). Our product creates a scalable HPC cluster on AWS or GCP, where users can submit Slurm or PBS jobs. These jobs have regular #SBATCH and #PBS headers and are no different from a standard Slurm/PBS script. Our meta-scheduler(CCQ) reads the scripts, deploys worker nodes to run the jobs, then tears them down once the jobs have finished. Instead of sbatch or qsub, you simply use ccqsub with any of the supported flags.Some of our customers use Futures with their traditional HPC clusters, and we want that functionality to be available to all of our users. I looked into integrating CCQ as a backend for a batchtools_custom plan and template file. Can you please offer advice on how to integrate a custom scheduler(that still uses Torque and Slurm headers) with Futures? I would like to determine the best path forward and required modifications to both products(CloudyCluster and this repo) before I proceed. I appreciate anything you or other members of the community have to offer.

~Cole McKnight

[cbmckni]

Hey @HenrikBengtsson did you see this? I am hoping to get back to this soon. I am thinking that a batchtools_custom backend will work with some minor modifications. Do you see the Future API being of any use in this case? Also, could you please link to an example of a batchtools_custom implementation?

[HenrikBengtsson]

Hi, sorry, this one slipped. It sounds like you should be able to use a custom batchtools template. It's been a while since I had to do customizations like that myself, but I think it's easist to first get it to work with vanilla batchtools code, i.e. something that involves it's makeRegister() and batchMap(). When that is work, the same template should work with future.batchtools.

[cbmckni]

Ok! Thanks for the advice, I will look into those functions.