-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[nodefs] Don't try to resolve absolute symlinks outside of the VFS #21805
Conversation
src/library_fs.js
Outdated
@@ -904,7 +904,8 @@ FS.staticInit();` + | |||
if (!link.node_ops.readlink) { | |||
throw new FS.ErrnoError({{{ cDefs.EINVAL }}}); | |||
} | |||
return PATH_FS.resolve(FS.getPath(link.parent), link.node_ops.readlink(link)); | |||
var parent = PATH_FS.resolve(FS.getPath(link.parent), link.node_ops.readlink(link)) === path ? link.parent.parent : link.parent; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you construct a test case that shows why this is needed?
Perhaps modify test/unistd/links.c
.
Also can you explain in more detail why this is needed? I'm strugglying the understanding why sometimes we resolve relative to one path and sometimes relative to another
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll try my best to explain my use case. Based on the scheme I gave above, with the current behavior, I can't access the directory store
outside of the directory public
where the symlink
is. The symlink
is "blocked" in the public
directory.
As it is my first emscripten
contribution, and as I use emscripten from within another tool, I don't know if I can reproduce this in test/unistd/links.c
, but if you have some steps or suggestions, I would be glad to make things work.
This suggestion is based on this issue
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the above scheme what is value of the symlink in public/store
? I.e. where is it pointing? i.e. Is it pointing to ../store
? i.e. what call did you use to create the link?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the context, WordPress/wordpress-playground#1283 looks like it has more info
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used this script :
test/script.php
<?php
symlink( __DIR__.'/store', __DIR__.'/public/store' );
and ran the script in the test
folder : php script.php
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see.. and presumably both of those are absolute paths?
That means that target of the symbol is absolute (/full/path/to/public/store
) and not relative (../public/store
)? Is that what you want? Does the problem go away if you use a relative symlink?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right! It works perfectly with relative paths! But unfortunately, I'm dealing with absolute paths in my current situation. A situation where I am not the one who created the symlinks. Do you think it's still worth pursuing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is worth fixing, I'm was just trying to figure out the set of circumstances in which this occurs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I should try different use cases then. My suggestion probably works if the symlink path leads to a file in parent directories. But what if the symlink path leads to children directories within a parent directory. I should investigate.
@sbc100 I ended up removing the suggestion I made. Because it didn't work for other use cases. But I dug into code to find out where it could be problematic and tried to correct these lines with multiple use cases in mind. Here are the use cases I based my PR on :
My use cases are using emscripten directly within |
Do you know which of these 6 cases fail currently with emscripten and which succeed? |
@sbc100 without the modifications I made in this PR, the three first succeed and the three last fail, because these 3 last use absolute paths. But with the modifications I added with my last commit, all of the uses cases are working. |
@sbc100 I reduced the code modification to one simple line in - path = nodePath.relative(nodePath.resolve(node.mount.opts.root), path);
+ path = nodePath.relative( PATH.isAbs(path) ? NODEFS.realPath(node.parent) : nodePath.resolve(node.mount.opts.root), path ); This will check if the Here is an example of this behavior with a relative path based on the structure I gave above [ Assuming this is in a
If So the line above will be :
If This is the current behavior :
This is not working. Because it then concatenates So the modification above will bring this :
This transforms the absolute path in a relative path that we know works completely fine. I believe this is correct, but for some reason, these tests are failing :
Feel free to share any insights if any. |
@sbc100 Hi ! I merged the 'main' branch into the PR and every test passed. What do you think I should do next ? |
src/library_nodefs.js
Outdated
@@ -235,7 +235,7 @@ addToLibrary({ | |||
var path = NODEFS.realPath(node); | |||
try { | |||
path = fs.readlinkSync(path); | |||
path = nodePath.relative(nodePath.resolve(node.mount.opts.root), path); | |||
path = nodePath.relative( PATH.isAbs(path) ? NODEFS.realPath(node.parent) : nodePath.resolve(node.mount.opts.root), path ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should even be calling nodePath.relative
at all? What happens if you just remove this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Without this line, the resolve
method which works correctly with relative paths will return wrong paths when using absolute paths. The idea with this line is to return a relative path in any case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But is that how readlink is supposed to work? If the link as absolute then shouldn't readlink return an absolute path?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sbc100 You're correct. Removing this line resolved the issues I was experiencing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like issue #3222 was the reason for this line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.. from my reading of #3222 and the fix that landed with the test_unistd_symlink_on_nodefs test, i don't think it is correct. See https://github.com/emscripten-core/emscripten/pull/21805#issuecomment-2094926239
below
Can you take the use cases you created above and add them to |
@kleisauke and @sbc100 I pushed the code without that line and indeed the The line
Inserts an absolute path in symlink. Therefore it couldn't be recognized except if you transform that absolute path in a relative path based on the mounted root in file
The - fs.symlinkSync(fs.realpathSync('./new-directory'), './symlink');
+ fs.symlinkSync('./new-directory', './symlink'); Obviously. So the absolute path issue still remains. Except if I replace : - path = nodePath.relative(nodePath.resolve(node.mount.opts.root), path);
+ path = nodePath.relative( PATH.isAbs(path) ? NODEFS.realPath(node.parent) : nodePath.resolve(node.mount.opts.root), path ); this |
Having looked at |
@sbc100 Ok then, I'll modify the test |
@sbc100 I suggested a new test to verify a inside VFS tree and outside VFS tree symlinks. One will succeed opening the file and the other will fail. Besides these new tests, you'll have the relative and absolute paths tests asked earlier. Anyways, only one test failed in
I suppose this is probably not linked with this PR. |
test/unistd/links.c
Outdated
mkdir("working", 0777); | ||
chdir("working"); | ||
symlink("../test/../there!", "link"); | ||
int fd = open("file", O_RDWR); | ||
fd = open("file", O_RDWR); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks unrelated.. can this be reverted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sbc100 My bad. I'm on it.
test/unistd/symlink_on_nodefs.c
Outdated
FS.mount(NODEFS, { root: './inside-symlink' }, 'direct-inside-link'); | ||
|
||
FS.mkdir('direct-outside-link'); | ||
FS.mount(NODEFS, { root: './outside-symlink' }, 'direct-outside-link'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for looking into this! I also put up a change to simplify this test a little: #21890
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will copy your changes.
test/unistd/links.c
Outdated
@@ -31,10 +31,11 @@ void setup() { | |||
FS.mkdir('folder'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think its still worth extending this test to include some absolute symlinks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know what I can do here that is not already done in the test/unistd/symlink_on_nodefs.c
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to see some tests or absolute symlinks added to test/unistd/links.c
.
test/unistd/links.c
it where we run as many of the symlink tests as possible. symlink_on_nodefs
I think should be reserved to tests that we can't run outside of nodefs (i.e. tests that involve using non-trivial mounting setups that require nodefs to pass).
One reason that we should use links.c
where possible is that its actually possible to run links.c
on native desktop systems to confirm they have the same behaviour as emscripten (i.e. to check emscripten is doing the right thing).
Perhaps symlink_on_nodefs.c
should be renamed to something better.. but I can't think of anything. Basically its for tests that can't run outside of nodefs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sbc100 Ok, I understand. I should probably duplicate the tests :
test_relative_path_symlinks();
test_absolute_path_symlinks();
from symlink_on_nodefs.c
based on the symlink system from links.c
in setup()
function.
Would you like me to refactor the code readability in links.c
as well like you indicated below ? :
int main() {
setup();
test_foo_bar();
test_baz();
return 0;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so yes, but maybe wait for #21890 to land to avoid too many conflicts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually I wonder if we even need symlink_on_nodefs.c
at all anymore? Since it won't really be testing anything will it? Other than "symlinks to outside the VFS don't work" which is basically the same as "symlinks to non-existant files don't work"? Or is it will worth keeping it around do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My understanding is that this use case and this test could probably be an indicator that this is not possible and that it shouldn't be possible. So we could imagine that the only test that the symlink_on_nodefs.c
could have are :
test_inside_symlink();
test_outside_symlink();
test_mount_link();
The others will be moved to links.c
?
What I find mostly interesting here is the proof that this has been decided to work like this. With this use :
fs.symlinkSync(fs.realpathSync('directory'), 'outside-symlink');
test/unistd/symlink_on_nodefs.c
Outdated
printf("buffer is %s\n", buffer); | ||
fclose(fd); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you put each of these test "block" into their own function with a specific name? e.g.
int main() {
setup();
test_foo_bar();
test_baz();
return 0;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated it.
test/unistd/symlink_on_nodefs.c
Outdated
FS.mkdir('direct-link'); | ||
FS.mount(NODEFS, { root: './symlink' }, 'direct-link'); | ||
FS.mkdir('direct-inside-link'); | ||
FS.mount(NODEFS, { root: './inside-symlink' }, 'direct-inside-link'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is "direct" here short for directory "directory" if so maybe call "dir" so as not to be confused with that "direct" adjective?
Better still since these are mount points how about calling them "mount-inside-symlink" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I modified it.
@sbc100 I made the corrections you asked, maybe did I miss something. Don't hesitate to give me other insights to improve the code readability. |
@sbc100 I merged and pushed the changes you requested and kept the
What do you think about this ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking like a really awesome change now! Thanks for working on this.
test/unistd/links.c
Outdated
} | ||
} | ||
|
||
void test_absolute_path_symlinks() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you put the opening curly braces on the same line as the function decl?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, on it.
// mounted | ||
fd = fopen("/direct-link/test", "r"); | ||
void test_outside_symlink() | ||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a comment here.
// outside-symlink is link to an absolute path which is not part of the emscripten VFS
// and so we should be able to open it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On it.
fd = fopen("/direct-link/test", "r"); | ||
void test_outside_symlink() | ||
{ | ||
FILE* fd = fopen("/working/outside-symlink/test", "r"); | ||
assert(fd == NULL); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
assert(errno == ENOENT)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On it.
test/unistd/links.c
Outdated
@@ -31,25 +31,66 @@ void setup() { | |||
rtn = write(fd, "test", 5); | |||
assert(rtn == 5); | |||
close(fd); | |||
rtn = mkdir("folder", 0777); | |||
rtn = mkdir("directory", 0777); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make we can use makedir(name)
and makefile(name, content)
helpers to make this setup more readable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(also makelink so that we can add the assert(rtn == 0)
for those symlink calls)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I replace every mkdir
function for a makedir
one, write
function for a makefile
and a makelink
for any symlink
function ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that is what I was thinking.
Then it would read something like:
makedir("test"):
makefile("test/file", "content");
makelink("file1", "file2");
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you also add an entry to to the ChangeLog, mentioning that this essentially reverts a previous change in behaviour?
@sbc100 Is this changelog ok for you ? :
|
Can you mention that this is nodejs behaviour? |
How about:
|
@sbc100 I added the changes you requested and added the |
ChangeLog.md
Outdated
@@ -524,7 +529,7 @@ See docs/process.md for more on how version tagging works. | |||
(#18861) | |||
- The `emscripten_proxy_async_with_callback` API was replaced with a simpler | |||
`emscripten_proxy_callback` API that takes a second callback to be called if | |||
the worker thread dies before completing the proxied work. | |||
the worker thread dies before completing the proxied work. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you revert all these unralated changes to the ChangeLog where newlines are removed?
BTW, the way I setup my editor is to (1) show trailing whitespace and (2) have a shortcut to strip it all, but not to have it do this automatically.. otherwise you end up with changes like this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the insights and sorry for the inconvenience.
I indeed had these three lines in my vscode/settings.json
file enabled :
"files.insertFinalNewline": true,
"files.trimFinalNewlines": true,
"files.trimTrailingWhitespace": true,
I should be more careful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.. just two ultra minor nits.
test/unistd/links.c
Outdated
int fd = open("file", O_RDWR | O_CREAT, 0777); | ||
} | ||
|
||
void makefile(char *file, char *content) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use const char *
for both args here and args below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On it!
@sbc100 Minor changes pushed. This was a long conversation 🙂. Thank you for your time! |
emscripten-core#21805 created a new entry, 3.1.61 (with release data 5/8), while we haven't released 3.1.60. Putting the entry back into 3.1.60.
#21805 created a new entry, 3.1.61 (with release data 5/8), while we haven't released 3.1.60. Putting the entry back into 3.1.60.
This PR reverts the behaviour that was added in #3277 and suggested in #3222.