-
-
Notifications
You must be signed in to change notification settings - Fork 259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Installation of DSGE.jl broken on Julia v1.5 for Windows #1943
Comments
Or we remove that assert completely... |
Or that... but then the following line unpacked = joinpath(dir, dirs[1]) assumes the relevant path to be the first element of BTW, I just rebuilt the sysimage using my fix above. Installation of DSGE.jl now works for me. |
Yeah! That's true... So we need to be a bit careful with that.. |
And this is why I wrote Tar.jl... which unfortunately doesn't help us with 1.5. Do you have a link to the archive that it's trying to install so that I can take a look at it and see what's going on? Seems like it may be a TAR archive with large files or some other features that 7z doesn't know what to do with. It might be possible to rewrite the archive so that it's less confusing to 7z or to configure the program differently. |
Do you have an idea where I might find this TAR archive? I'm not precisely sure where to start looking for this file (fyi I only have access to MacOS and Linux, so it's something specific to Windows, I won't be able to find that link). On the subject of large files, there are some large JLD2 files used for testing (50-60MB). These files are git tracked. If that may be an issue, then I can try reducing the size of those JLD2 files. |
The URL is https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 but insert the correct organization, package name and tree hash. |
This tarball has some paths that are too long for the standard TAR format to handle, so it's being created with POSIX extended headers. The 7z tool doesn't understand these headers so it extracts them as if they were files, which are in the root of the tarball instead of where they're supposed to be. Note that file sizes aren't the issue here, it's just the file names. Names like
The last option could be done by hand and would allow 7z to extract something a bit better—like a truncated version of these files—but it would still not be correct. There's simply no way to get 7z to extract this file tree correctly. Note, however, that the max total path length on Windows is 260 bytes, and since the total path of one of these files ends up being Bottom line: don't put paths this long in your artifacts or packages. |
Ah, ancient Windows restrictions! 😖However, at least for Windows 10, Version 1607 and above there is another solution: Enabling long paths! For whatever reasons, long paths have to be opted in, see here. Opting in is done by setting the DWORD After that, I could install DSGE.jl without using my dirty hack above. I'm not sure why Microsoft chose to make this opt-in, because enabling long paths doesn't seem to have any real downside, see e.g. here:
Still, a more informative error message might be useful. Package maintainers who exclusively develop on Linux might not even be aware of the 260 bytes limitation and therefore might not follow Stefan's sound advice :
And a lot of Windows users will simply give up (maybe even on Julia) as soon as they get AssertionError: length(dirs) == 1 Maybe something like:
(Here I implicitly assume that this error will only be triggered in this particular situation, which is probably too optimistic.) |
The Windows limit isn't the main issue here though—it's not even getting that far. The main problem is that the 7z extraction tool doesn't know how to handle POSIX extended TAR features, which are necessary to put these files into a tarball. It's not even getting to the point where Windows can have a problem with file lengths because the 7z extracts the files with long names incorrectly into the root of the folder. I agree that the error message is not great, but the message is misleading. The real solution going forward is to use the Pkg protocol to get better formed tarballs and Tar.jl on the client for extraction. Tarballs sent via Pkg protocol don't have the package name as a leading directory like this, so there's no need for the assertion or figuring out which item to look for the tree under. Tar.jl knows how to extract tarballs with POSIX extended features, so it will correctly extract these. Which will lead to the point where it can hit the Windows path length limit, which should hopefully have a clear enough error message. In short, aside from improving the error message here, I don't think there's a whole lot to do for 1.5. The most pragmatic immediate solution is that the DGSE package should be modified to not include such long file names. |
I am working on fixing the problem for DSGE.jl. The problematic files appear to just be git tracked output produced during tests. In principle, the tests should still be able to run if I delete the output files after the tests finish, so these files will no longer need to be git tracked. Thanks for all the assistance! |
Note that if someone runs the tests on Windows, they may end up hitting the path limit. Not sure if that's a concern. |
Thanks for the heads up; I'll leave a note on the main page about which tests would be problematic to Windows users. |
As mentioned here installation on Windows starting from Julia v1.1 seems to be broken right now. I can confirm that this is still an issue on Julia v1.5 for Windows. When trying to install DSGE.jl in a clean environment I get:
The function
install_archive()
inOperations.jl
checks whether the temporary folder, which includes the extracted package, contains exactly one path - excluding a possible spurious file calledpax_global_header
, which seemingly 7z might create on Windows. Starting on line 559 ofOperations.jl
:However, for DSGE.jl the contents of the temporary folder that is created during the installation of DSGE.jl look like this:
Thus, filtering
pax_global_header
doesn't help because it isn't even there. Instead there is a bunch of other.data
and.paxheader
files, which are not filtered out such that@assert length(dirs) == 1
errors.A quick fix would simply replace
with
(However, the fix is not really future proof, because installation of a package will break whenever 7z writes other spurious files which don't match the filter patterns above...). I should also mention that I never had this issue with any other package in Julia (and I installed quite a lot...).
Version info:
The text was updated successfully, but these errors were encountered: