Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improving the download and upload functionality #24

Merged
merged 14 commits into from
Apr 2, 2024
Merged

Improving the download and upload functionality #24

merged 14 commits into from
Apr 2, 2024

Conversation

PeterDing
Copy link
Owner

  1. Updates to dependencies like ruff and adding python-dateutil library.
  2. Adding test cases and formatting code.
  3. Fixing issues related to termination when playing shared files.
  4. Refactoring the download functionality to use an Executor class, which is a ThreadPoolExecutor for concurrent downloads or a single thread executor.
  5. Adding an update_download_url method to various API classes (AliPCSApi, AliOpenPCSApi, and AliPCSApiMix) for updating download URLs.
  6. Handling expired shared authentication tokens and getting download link information from AliPCS.meta or using the PcsNode.DownloadUrl API.
  7. Deprecating the PcsFile.update_download_url method and adding a PcsDownloadUrl.download_url property.
  8. Using the dateutil library to parse ISO 8601 date formats.
  9. Refactoring the RangeRequestIO class to add a read_iter method.

@PeterDing
Add read_iter method for RangeRequestIO
267f226
@PeterDing
Use dateutil library to parse iso 8601 date format
428b1e2
@PeterDing
Change 0.0 to 0
6a2cc8e
@PeterDing
Deprecate PcsFile.update_download_url method; Add
b5988a8 
`PcsDownloadUrl.download_url`
@PeterDing
* Update shared_auth when it is expired
cd3cb23 
* Get download link info from `AliPCS.meta` firstly.
  When the link is not included at the info, using `PcsNode.DownloadUrl`
  api

* Fix variable error
@PeterDing
Add update_download_url method for AliPCSApi, AliOpenPCSApi and…
822b070 
… `AliPCSApiMix`
@PeterDing
Add Executor class
dbc6a34 
Executor is a ThreadPoolExecutor when max_workers > 1,
else a single thread executor.
@PeterDing
Disentangle MeDownloader from ThreadPoolExecutor
363e1b8
@PeterDing
* Use Executor to handle concurrent download task
5c5ee12 
* Catch the download request error at `download_file`
function

* Use iteration to walk directory ignoring use recursion
@PeterDing
Not retry in upload_file, let uplevel funtion to handle it
89d3432
@PeterDing
Fix the termination when play shared files
4341901
@PeterDing
Format code
15467fe
@PeterDing
Add test case
4faa5ef
@PeterDing
Update ruff and add python-dateutil
30ad8f5
@PeterDing PeterDing merged commit 2ba550f into master Apr 2, 2024
20 checks passed
@PeterDing PeterDing deleted the fix/disentangle branch April 3, 2024 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@PeterDing