-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
comments about rule 7: "Mount dataset at run time" #102
Comments
The data itself can be "the same" from an archive - that still doesn't make it feasible to store a very large dataset in the container. |
@vsoch It might not be most efficient or easy to maintain, but it is best for reproducibility. Of course there are other requirements beside reproducibility, such let's say "maintainability", so the rule should be moved there. Actually a lot of rules lead to conflicts: the more reproducible something is, the more expensive the maintenance might become (alone storing everything is a burden). But if you have an archive of the data anyway, why not adding it as layer in OCI and be done? |
It doesn’t sound like you’ve worked with the large datasets that I have in the past, where it would not be reasonable or feasible to add to a container. |
@vsoch Yes, I did not work with large data, my biggest RAID is only 0.1PB, so I couldn't even store that (I know LHC produces 2 GB/sec and has some hundred PBs stored), but I work on reproducible systems (reproducible in "continuum" sense from previous item, unfortunately :)). |
No small datasets are appropriate to add to the container, given it's de-identified.
The rule targets large datasets time and time again, so perhaps we didn't make this clear enough because we added a context that it's for review/testing, which it doesn't need to be. Small datasets are fine to include if it's reasonably small. |
@vsoch yes, I see. Please remember I don't want to critise any work but I like to give input for possible future improvements. |
The rule is saying:
It's reasonably stated and although the focus is on large datasets to tell the user about bind mounts, small datasets inside the container (with other small files) is also in scope. This feels like nit picking to me, and losing sight of the audience that the writing was intended for. |
@vsoch hehe yes it might be |
Because it is possible to put large data in a container, and people do try (and registries vary based on the size / number of layers to accept) so it needs to be explicitly stated. A container is not a storage vehicle for large data - it's for small data and/or software and analysis scripts. We have better means (data archives, object storage) more appropriate for that. |
comments about rule 7: "Mount dataset at run time"
To be reproducible, there is nothing that can be different at different runtime, all must be the same. Thus it does not matter when to include data. Of course, for practical use on different data or maintenance this might be a good habit.
The text was updated successfully, but these errors were encountered: