Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cpackget] --concurrent-downloads (default 5) corrupts downloaded *.pdsc files #197

Closed
thorstendb-ARM opened this issue Aug 9, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@thorstendb-ARM
Copy link
Collaborator

thorstendb-ARM commented Aug 9, 2023

Describe the bug
When running "update-index -a" (where -a option skips already installed files, if an update is necessary, this is done in the next step) some files are not written into .Web folder. It seems they get downloaded to .Download, but the file move to .Web fails.

To Reproduce
Steps to reproduce the behavior:

  1. Delete some (random) files from .Web folder
  2. Run the cpackget with the following args: "update-index -a"
  3. See some of the files are not copied

Expected behavior
All files that do exist in index.pidx but not in .Web folder should be added

Environment

  • OS: windows 10

Additional context
Adding the option "-C 0" (disable concurrency) seems to solve the issue for "update-index -a", which means this can be related to concurrency in file operations

@thorstendb-ARM thorstendb-ARM self-assigned this Aug 9, 2023
@thorstendb-ARM thorstendb-ARM added the bug Something isn't working label Aug 9, 2023
@jkrech jkrech changed the title [cpackget] concurrent actions seem to corrupt each other [cpackget] --concurrent-downloads (default 5) corrupts downloaded *.pdsc files Aug 22, 2023
thorstendb-ARM added a commit that referenced this issue Aug 29, 2023
…dsc files #197

fixed:
- concurrent download scheduler
- added tests on downloaded files
thorstendb-ARM added a commit that referenced this issue Sep 1, 2023
…dsc files #197 (#200)

fixed:
- concurrent download scheduler
- added tests on downloaded files
@thorstendb-ARM
Copy link
Collaborator Author

thorstendb-ARM commented Sep 1, 2023

Wrote a new concurrent scheduler using semaphore. In the original version, one blocking download was done every n (concurrent) +1 downloads, but at least all files were spawned as thread (which resulted in much more threads than the concurrent number). There was no wait to finish spawned threads at the end of the loop, which caused the cut off or missing files.
The semaphore mechanism also allows that finished threads are immediately filled up with new threads instead of waiting for a bunch of threads to be finished before spawning the next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant