Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock files should be pipeline ID specific in some way -- this will enable pipelines to flag when old run locks are present. #194

Open
mmcdermott opened this issue Aug 30, 2024 · 1 comment
Labels
Computational Performance Issues relating to efficient computational performance of MEDS_transforms pipelines Pipeline Configuration and Stage Management Issues relating to proper definition and usability of different stages in a pipeline priority:medium A medium priority issue.

Comments

@mmcdermott
Copy link
Owner

Tagging @Oufattole as this will help MEDS-Tab be more usable too.

@mmcdermott
Copy link
Owner Author

This will be a bit tricky as it depends on hydra nonsense. Basically we need all workers to be able to have the same "pipeline ID" as assigned via the hydra config, meaning when you run a job with multirun, we need a variable to interpolate once before the sweeper goes, then the sweeper to go and not reset that variable. This can be a little finnicky with hydra interpolation rules. But I think some resolvers work like that -- in particular maybe the hydra specific ones do, like now. This would also preclude someone from manually launching multiple workers, which I am fine with, as I can't imagine a situation where that would be better than using hydra launchers.

@mmcdermott mmcdermott added Computational Performance Issues relating to efficient computational performance of MEDS_transforms pipelines priority:medium A medium priority issue. Pipeline Configuration and Stage Management Issues relating to proper definition and usability of different stages in a pipeline labels Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Computational Performance Issues relating to efficient computational performance of MEDS_transforms pipelines Pipeline Configuration and Stage Management Issues relating to proper definition and usability of different stages in a pipeline priority:medium A medium priority issue.
Projects
None yet
Development

No branches or pull requests

1 participant