Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/archive: Use map of long names avoiding max path limits #8449

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kaovilai
Copy link
Member

@kaovilai kaovilai commented Nov 24, 2024

Signed-off-by: Tiger Kaovilai tkaovila@redhat.com

Thank you for contributing to Velero!

Please add a summary of your change

pkg/archive: Use map of long names avoiding max path limits

Test image: ghcr.io/kaovilai/velero:maxpathlimits-a2699e765 for amd64, arm64

Does your change fix a particular issue?

Fixes #8434

Please indicate you've done the following:

Sorry, something went wrong.

Verified

This commit was signed with the committer’s verified signature.
kaovilai Tiger Kaovilai
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
@kaovilai kaovilai marked this pull request as ready for review November 24, 2024 07:38
Copy link

codecov bot commented Nov 24, 2024

Codecov Report

Attention: Patch coverage is 86.04651% with 6 lines in your changes missing coverage. Please review.

Project coverage is 59.00%. Comparing base (9f0026d) to head (a2699e7).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
pkg/archive/extractor.go 70.58% 5 Missing ⚠️
pkg/restore/prioritize_group_version.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8449      +/-   ##
==========================================
+ Coverage   58.98%   59.00%   +0.01%     
==========================================
  Files         368      368              
  Lines       39000    39017      +17     
==========================================
+ Hits        23004    23021      +17     
  Misses      14532    14532              
  Partials     1464     1464              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:


// If the target is longer than maxPathLength, we'll use the sha256 of the header name as the filename.
// https://github.com/vmware-tanzu/velero/issues/8434
if len(target) > maxPathLength {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See here and here, header.Name contains more than one fs separators, which results in the generation of more than one sub dirs after calling e.fs.MkdirAll.
If we replace it with a hash string, the separators will be eliminated, as a result, the followed logics, like Parse will be affected.

Is the restore still functioning with this PR? Or anything I misunderstood?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will double check that restore Parse is working and report back.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hash string is saved in a map, and separators are stored. When parser encounters a sha256 in a map, it gets back the original

I believe per unit test in pkg/archive/parser_test.go show that it is functioning correctly on long dir name and long file name.

We're only generating hash for name of dir or name of file, not a path containing separators.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At here, we change the target to a hash string of header.Name:
target = filepath.Join(dir, shortSha256Name)

Per my understanding header.Name contains file system separators. So the separators are missing in target

Then at here, we create dirs according to target:
err := e.fs.MkdirAll(target, header.FileInfo().Mode())

As a result, the separators in the original header.Name are eliminated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is low priority let's bump this one out. Perhaps I can scope the hashing to if header.Name has no separators

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See

// GetVersionedItemFilePath returns an item's file path once extracted from a Velero backup archive, with version included.
, I think header.Name has separators in most cases.

@kaovilai kaovilai marked this pull request as draft February 14, 2025 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Backups stuck Deleting because of "filename too long"
4 participants