-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Race conditions with config.yml
#346
Comments
Hi thank you for using AGAT and thank you for your feedback. The first solution is what we had before. It was not doable anymore because AGAT contains 72 scripts and the options has to be handle for all of them, so any tiny modification was a nightmare to scatter over all the scripts. The second solution can be setup but means modifying all the script to add a --config parameter, and modifying the mechanism to create the config file to be named as wished. In my sense a proper pipeline should run each step in a dedicated folder (as it is handle by e.g. Nextflow) to avoid any of these inconvenience. Would't be better to modify your pipeline to avoid to encounter that type of cases? Using a workflow manager system (WMS) as Nextflow would avoid to encounter that type of problems. |
I understand the concern, but i find that solution pretty opaque and it would be a disappointment if this can't be fixed in agat. An honest question (from someone who has not coded with perl in over a decade): can't one "simply" do something like the following in each of the scripts?
In english, have a function that extends a local hash of this script's args with a centrally-defined set of parser args, and another that generates a config from the global set of defaults in the yaml and the CLI args? It's a pretty common pattern in at least python & C/C++, so I assume this is possible even if my perl code above is a bit off. |
^^ Thank you for your suggestions. |
Thanks for AGAT, i've found it very useful. I've just updated to v1.0.0 to fix a bug I hit in an earlier version, and am having trouble getting my existing pipeline working given the new config.yml mechanism.
Describe the bug
V1's configuration mechanism sufferers from race conditions and overwrites other tool's
config.yml
.General (please complete the following information):
To Reproduce
Run the following two blocks in parallel with e.g.
snakemake
One
agat config
will overwrite the other, and often both commands with get either--tabix
or--no-tabix
at random, depending on FS timing. It will also occasionally work as intended, and I'd imagine there are other strange race condition effects that could be observed.Expected behavior
In order of preference:
agat_convert_sp_gxf2gxf.pl --tabix --gff ...
and so one does not need to doagat config --expose
at allagat-tabix.yml
andagat-notabix.yml
and pass e.g.agat_convert_sp_gxf2gxf.pl --config agat-tabix.yml
The text was updated successfully, but these errors were encountered: