Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NC | UploadPart | finish_upload | stat(part-{num}) should not check ctime due to concurrency part uploads of the same part. #8017

Merged
merged 1 commit into from
May 7, 2024

Conversation

romayalon
Copy link
Contributor

Explain the changes

  1. Concurrent/retries uploadPart of the same part number caused InternalError with "cancelled due to ctime change" message. This error originated from stat(part-{num}), the last row of the stat code is check ctime change, which was changed during the retry of the same part.

Issues: Fixed #xxx / Gap #xxx

  1. Fixed observed in noobaa log as "cancelled due to ctime change" actual upload didn't fail  #7833

Testing Instructions:

  1. Couldn't reproduce it without adding sleep(10); in our native code, so adding manual testing instructions -
Pre Tests - 
1. update fs_napi.cpp FileStat function - add sleep(10); before the ctime check.
2. build the changes - node-gyp build

Tab 1 - 
1. Start NooBaa NSFS endpoint - 
sudo node src/cmd/nsfs.js --debug=5

Tab 2 -
2. Create Account - 
sudo node src/cmd/manage_nsfs.js account add --name account1 --uid=0 --gid=0 --new_buckets_path=/private/tmp/

3. Create s3 alias with account1 credentials, pointing to NooBaa endpoint -
alias s3api='AWS_ACCESS_KEY_ID=x AWS_SECRET_ACCESS_KEY=y aws --endpoint https://127.0.0.1:6443 --no-verify-ssl s3api'

4. Create a bucket using s3api- 
s3api create-bucket --bucket=buck1

5. Create MPU - 
s3api create-multipart-upload --bucket=buck1 --key=obj1

Tab 3 - 
6. Create s3 alias with account1 credentials, pointing to NooBaa endpoint -
alias s3api='AWS_ACCESS_KEY_ID=x AWS_SECRET_ACCESS_KEY=y aws --endpoint https://127.0.0.1:6443 --no-verify-ssl s3api'

7. create a file with some content - 
echo "blablbla" > bla

8. Upload the same part from Tab 2 and Tab 3 with a latency of less than 10 seconds - 
Tab 2 -
s3api upload-part --bucket buck1 --key obj1 --part-number 1 --upload-id {upload id specified at step 7} --body bla
Tab 3 (after 2 seconds) - 
s3api upload-part --bucket buck1 --key obj1 --part-number 1 --upload-id {upload id specified at step 7} --body bla
  • Doc added/updated
  • Tests added

src/sdk/namespace_fs.js Outdated Show resolved Hide resolved
Signed-off-by: Romy <35330373+romayalon@users.noreply.github.com>
@romayalon romayalon force-pushed the romy-put-part-disable-ctime branch from 11e1980 to 490755a Compare May 7, 2024 08:52
@romayalon romayalon merged commit 029c07a into noobaa:master May 7, 2024
10 checks passed
@romayalon romayalon mentioned this pull request May 7, 2024
2 tasks
@guymguym guymguym added this to the 5.15.3 milestone May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

observed in noobaa log as "cancelled due to ctime change" actual upload didn't fail
2 participants