Headers not being extracted into include folder properly #2584
Comments
|
I can confirm that manually finishing the extraction process by removing the suffixes fixes things. |
|
For now I am working around the issue by manually fixing the cache (copying files with |
|
I'm experiencing the same issue. Many build attempts fail because of missing node header files. I see those files with "DELETE..." appended. This is on a GitHub Action runner using "windows-latest". |
|
I was able to reproduce this locally with a small project. The package mmmagic uses node-gyp during install. Windows 11 (latest) This results in a corrupted node-gyp\Cache\16.13.2\include\node directory (bunch of DELETE labeled files). I tested several times by deleting the node_modules and node-gyp\Cache folder and re-running the This does not appear to happen in MacOS. And it did not happen for many months when we were using Node 14. |
|
I've found that doing the following seems more reliable (will report back if it fails in the future) |
|
I'm experiencing the same issue. I think the problem is likely that we are attempting to |
|
Reporting back, I've experienced no issues after using my work around above. Using |
see nodejs/node-gyp#2584 (comment) for more information
see nodejs/node-gyp#2584 (comment) for more information
|
Still repro's as of node-gyp 9.0.0 |
|
Please create a GitHub Action that reproduces this behavior. |
|
@cclauss how about this? https://github.com/MRayermannMSFT/node-gyp/actions/runs/3475120714/jobs/5809002444 - 16.x,3.6, windows-latest Things to note:
|
|
I decided to go down the route of seeing if this was a bug in I then modified the script to download & extract the tarball in the same way that I then modified the script again to do many download & extracts in parallel, but this time all to the same location. My script now reproduces the issue. You can find the script here: https://gist.github.com/MRayermannMSFT/b008c565eff8eaa4d180408002cf61e5 To change whether or not the download & extracts target the same location change So I guess node-gyp is probably doing parallel downloads of the headers tarball when it shouldn't be. Or at least that would be my hypothesis based off what I am seeing. |
|
Spent some time trying to dig into silly logs to verify my thinking but I think that npm logging is making that difficult. I can see what looks like multiple native module re-builds happening in parallel: But then below all that I only have the npm error logs for one of the failed installs, even though it says both finished with an exit code of 1. I gues npm is just choosing to show one of the failure logs? |
|
FYI. |
|
I went somewhat deep on this issue and came out the other side with a PR to fix it, so sharing my findings along the way here. The others in these comments correctly identified the repro case, which is parallel builds of dependencies all attempting to populate I'm a little surprised that multiple parallel builds extracting the tarball into I looked into the errors originating from I've got a PR (will open later, GitHub is having incidents around GitHub Actions at the moment) which fixes the incorrect handling of errors so that an error won't leave the cache in a corrupted state, and fixes this issue by extracting the tarball to a temp directory and then copying the files over to the final destination. In practice this is more robust since it doesn't involve unlinking the target file path, and multiple parallel installs can copy to the same file without causing any major problems - just the occasional |
|
Tip jar for @dsanders11?! |
|
GitHub actions back to working well enough to get the PR up, #2846.
I appreciate the sentiment. 🙂 Just happy to fix such a persistent bug. |
Verbose output (from npm or node-gyp):
It seems like node-gyp is not properly extracting the header files. Sometimes it works, sometimes it doesn't. Here's an example of what my cache include folder looks like when it does not work:
You can see there's a bunch of files with a
DELETE.some-thingsuffix, including two for the file that node-gyp ultimately failed because of.When it does work, there are still a few
DELETE.some-thingfiles, but only a few, none with two of them, and none that don't have a normal copy of the file.The text was updated successfully, but these errors were encountered: