Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(fs#S3#{list,rmtree}): fix support of more than 1000 files #6103

Merged
merged 3 commits into from
Jan 20, 2022
Merged

fix(fs#S3#{list,rmtree}): fix support of more than 1000 files #6103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
51f7a2f
feat(fs): speedup s3 delete tree 8-10/s => 100-120/s
fbeauchamp Jan 20, 2022
2c85c40
Update CHANGELOG.unreleased.md
julien-f Jan 20, 2022
da03c7f
Merge branch 'master' into fix_s3_isTruncated
julien-f Jan 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions @xen-orchestra/fs/src/s3.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ export default class S3Handler extends RemoteHandlerAbstract {
Delimiter: '/', // will only return path until delimiters
})

if (result.isTruncated) {
if (result.IsTruncated) {
const error = new Error('more than 1000 objects, unsupported in this implementation')
error.dir = dir
throw error
Expand Down Expand Up @@ -267,7 +267,7 @@ export default class S3Handler extends RemoteHandlerAbstract {
Prefix: this._dir + path + '/',
ContinuationToken: NextContinuationToken,
})
NextContinuationToken = result.isTruncated ? result.NextContinuationToken : undefined
NextContinuationToken = result.IsTruncated ? result.NextContinuationToken : undefined
for (const { Key } of result.Contents) {
// _unlink will add the prefix, but Key contains everything
// also we don't need to check if we delete a directory, since the list only return files
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.unreleased.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
- [Backup] Detect and clear orphan merge states, fix `ENOENT` errors (PR [#6087](https://github.com/vatesfr/xen-orchestra/pull/6087))
- [Backup] Ensure merges are also executed after backup on S3, maintaining the size of the VHD chain under control [Forum#45743](https://xcp-ng.org/forum/post/45743) (PR [#6095](https://github.com/vatesfr/xen-orchestra/pull/6095))
- [Backup] Delete backups immediately instead of waiting for the next backup (PR [#6081](https://github.com/vatesfr/xen-orchestra/pull/6081))
- [Backup] Delete S3 backups completely, even if there are more than 1000 files (PR [#6103](https://github.com/vatesfr/xen-orchestra/pull/6103))

### Packages to release

Expand Down