Fix making too many create directory call #2828
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dphulkar-msft
requested review from
gapra-msft,
adreed-msft,
vibhansa-msft,
seanmcc-msft,
souravgupta-msft and
tanyasethi-msft
as code owners
|
@gapra-msft @adreed-msft |
dphulkar-msft
changed the title
vibhansa-msft
approved these changes
Oct 14, 2024
gapra-msft
reviewed
Oct 14, 2024
adreed-msft
requested changes
Oct 15, 2024
There was a problem hiding this comment.
A couple concerns I thought of...
- This only affects folders we're directly tracking to transfer properties. What about those that we're not?
- The mention of the use of a Trie instead of a hashmap on Vikas' comment
- What about when we're not trying to persist directories at all? e.g. blob->file. We need to carefully track the existing directory structure for a valid implementation of this.
…com/Azure/azure-storage-azcopy into dphulkar/fixCreatingTooManyDirCalls
…com/Azure/azure-storage-azcopy into dphulkar/fixCreatingTooManyDirCalls
gapra-msft
reviewed
Oct 22, 2024
…into dphulkar/fixCreatingTooManyDirCalls
gapra-msft
reviewed
Oct 23, 2024
gapra-msft
reviewed
Oct 23, 2024
adreed-msft
requested changes
Oct 23, 2024
adreed-msft
approved these changes
Oct 23, 2024
gapra-msft
added a commit
that referenced
this pull request
Nov 12, 2024
This reverts commit b462994.
gapra-msft
added a commit
that referenced
this pull request
Nov 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue:
In the folderCreationTracker.go file, we addressed an issue where multiple createDirectory REST calls were being sent for the same folder during the copy operation, particularly with deeply nested directory structures. This was caused by the StopTracking() function prematurely removing folder entries from the contents map, leading to redundant createDirectory requests.
Fix:
To resolve this issue while maintaining memory optimization (which was the original reason for removing entries), we have replaced the contents map with a trie data structure. The trie allows us to efficiently track folder creation and ensures that redundant requests are avoided, while also optimizing memory usage.