Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use GNU tar on macOS for OCIContainer interactions #1434

Closed
wants to merge 2 commits into from

Conversation

mayeut
Copy link
Member

@mayeut mayeut commented Mar 18, 2023

macOS uses BSD tar as the default tar command.
This can result in hard to debug issues when building linux wheels on macOS because of incompatibilities between GNU tar (used inside containers) & BSD tar (used on host).
This commit tries to use GNU tar on the host macOS and warns if it is not found.

macOS uses BSD tar as the default `tar` command.
This can result in hard to debug issues when building linux wheels on macOS because of incompatibilities between GNU tar (used inside containers) & BSD tar (used on host).
This commit tries to use GNU tar on the host macOS and warns if it is not found.
@mayeut
Copy link
Member Author

mayeut commented Mar 18, 2023

For a bit of context,
I already had seen some tar: Ignoring unknown extended header keyword XXXXXXXXX'` in the past but I didn't recall it to be an issue at the time (everything was building fine in the end).
I found the issue while working on patchelf-pypi when tests started to fail when running locally on my laptop (most likely after a macOS update). This time around there were no tar warnings printed out and thus tracking down the root cause took a bit of time.

Copy link
Contributor

@joerick joerick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good to me. What were the symptoms of the tar incompatibility, as you saw? I'm just wondering if we could set some flags on BSD tar to force it to be compatible? Or even use pax instead, which is also preinstalled on macos.

@mayeut
Copy link
Member Author

mayeut commented Mar 19, 2023

GNU tar was extracting some of the files produced by BSD tar as "sparse" (I don't know much about this):

/project/src/patchelf-upstream/tests/no-rpath-prebuild/GNUSparseFile.0/no-rpath-pie-powerpc

vs

/project/src/patchelf-upstream/tests/no-rpath-prebuild/no-rpath-pie-powerpc

Using --format pax does not change the outcome.
Using --format ustar allows to build/test patchelf-pypi but there are some : Pathname too long warnings printed out along the way.

@mayeut mayeut marked this pull request as draft March 19, 2023 14:05
@mayeut
Copy link
Member Author

mayeut commented Mar 19, 2023

I converted to draft because I need to run some more checks (the image is alpine in my case so: does it only impact musllinux & shall something be done in there ?).

@mayeut
Copy link
Member Author

mayeut commented Mar 19, 2023

So in fact it's not an issue between BSD tar & GNU tar that I'm seeing but with BusyBox tar used in musllinux images.
Using GNU tar inside musllinux images, I'm getting a successful build with BSD tar on host side with a bunch of tar: Ignoring unknown extended header keyword warnings printed out.

It seems there's also a --format gnutar that seems to work in all cases (but is that limited to certain versions of macOS) ? maybe that's the way to go here ?

I'll probably add GNU tar to musllinux images in any cases.

@joerick
Copy link
Contributor

joerick commented Mar 19, 2023

Gotcha, thanks for the info.

Using --format pax does not change the outcome.

Ah, my apologies, I was not clear, I meant to use the pax tool, which can compress/decompress tar archives (and I believe was created in response to differing tar implementations).

(Please take these comments only as ideas, I'm happy with the current change as-is.)

@joerick
Copy link
Contributor

joerick commented Mar 19, 2023

Oops, our replies overlapped there.

So in fact it's not an issue between BSD tar & GNU tar that I'm seeing but with BusyBox tar used in musllinux images. Using GNU tar inside musllinux images, I'm getting a successful build with BSD tar on host side with a bunch of tar: Ignoring unknown extended header keyword warnings printed out.

Interesting! I suppose we still don't know what that 'unknown extended header keyword' is, it might not be relevant/critical, though it's kinda a smell.

It seems there's also a --format gnutar that seems to work in all cases (but is that limited to certain versions of macOS) ? maybe that's the way to go here ?

I don't have that option on my machine, running macOS 13.2.1...

$ /usr/bin/tar --help       
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
  -c Create  -r Add/Replace  -t List  -u Update  -x Extract
Common Options:
  -b #  Use # 512-byte records per I/O block
  -f <filename>  Location of archive
  -v    Verbose
  -w    Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
  <file>, <dir>  add these items to archive
  -z, -j, -J, --lzma  Compress archive with gzip/bzip2/xz/lzma
  --format {ustar|pax|cpio|shar}  Select archive format
  --exclude <pattern>  Skip files that match pattern
  -C <dir>  Change to <dir> before processing remaining files
  @<archive>  Add entries from <archive> to output
List: tar -t [options] [<patterns>]
  <patterns>  If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
  <patterns>  If specified, extract only entries that match
  -k    Keep (don't overwrite) existing files
  -m    Don't restore modification times
  -O    Write entries to stdout, don't restore to disk
  -p    Restore permissions (including ACLs, owner, file flags)
bsdtar 3.5.3 - libarchive 3.5.3 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.8 

@mayeut
Copy link
Member Author

mayeut commented Mar 19, 2023

I don't have that option on my machine, running macOS 13.2.1...

It's not listed in --help, if you do man tar, it redirects you to man libarchive-formats

it might not be relevant/critical, though it's kinda a smell.

If/When GNU tar is added to musllinux, agreed. Otherwise, that still leave to debug issues on musllinux (where I didn't see those warnings).

@mayeut
Copy link
Member Author

mayeut commented Mar 25, 2023

With GNU tar added in musllinux images, I'm closing this PR.

@mayeut mayeut closed this Mar 25, 2023
mayeut added a commit to mayeut/patchelf-pypi that referenced this pull request Apr 2, 2023
This allows to build Linux wheels locally on macOS.
c.f. pypa/cibuildwheel#1434
@mayeut mayeut deleted the macos-gnu-tar branch October 19, 2024 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants